Education: Low powered comparisons are futile

futile (fyootl) adjective: Incapable of producing any useful result; pointless.
Synonyms: vain – useless – unavailing – fruitless – idle – nugatory

(recallThe instrument the clinical trialist uses to detect, observe and measure hypothesized treatment effects is the clinical trial)

An astonomer wouldn’t search for the existence of a hypothesized binary star in the distant Andromeda galaxy with binoculars…

A pathologist wouldn’t examine biopsied tissue for the presence of tumor markers with a magnifying glass…

An elephant hunter wouldn’t set out on safari with only a pistol…

Yet curiously, underpowered methods are persistent and widespread in published psychiatric research. It is common to see either no sample size justification or a veiled apology for the small sample size followed by a battery of significance tests and the inevitable reports of “no significant differences…”

A more subtle form is when the trial has adequate power for the primary objective but not for subsequent secondary, subgroup and safety analyses however the investigators act as if having adequate power for the primary objective warrants any further comparisons they wish to make.

Results reporting “no significant difference” from underpowered comparisons resist meaningful interpretation because there exist two possibilities:
1) that there is no underlying difference or
2) there is a difference but the study was too small to detect it i.e. false negative (type II) error.
and there is no way to tell which is the correct explanation.

Underpowered research has been described by Lemuel Moyé as looking for something in the basement… without bothering to switch the light on.

Underpowered research can be exploited in order to make unwarranted claims of “safety” , “equivalence” and “non-inferiority”.

Example:

“Lions are safe.”
“Lions are as safe as llamas.”
“Llamas are as dangerous as lions!” 

Underpowered clinical trials are usually doomed to fail before they even begin; they signal to the reader the investigators do not understand the basic elements of clinical trial design.

Suggested reading:
The continuing unethical conduct of underpowered clinical trials.
Muda (無駄)
What is statistical power?
The RCT safety fallacy