Study design & execution:
A 6 week, randomized, double blinded, parallel design. Elements of the trial design are clearly detailed (as per the CONSORT statement) in Methods section of the text.
The study’s primary outcome variable remains consistent from the registry to the top-line result:
(IRCT): The primary outcome measure is the Teacher and Parent ADHD Rating Scale-IV.
(Methods section): Symptoms were rated using the Parent and Teacher ADHD-RS-IV at baseline and weeks 3 and 6.
(Abtract: Results): No significant difference was found between the two groups on the Parent and Teacher ADHD-RS-IV scores…
Completion was high with 19/22 patients completing the trial in both treatment groups.
Analysis & Reporting:
The analysis gets off to an inauspicious start by committing the baseline data testing howler. A sensible statistical analysis was undertaken for the primary outcome, the text reports the treatment effect and its 95% confidence interval.
An additional post-hoc analysis is reported:
…we carried out a post hoc calculation of sample size for detection of observed mean differences between the two groups…
A sample size of 176 in each group was calculated to detect an average difference in effect size of 0.3 between the two groups.
This calculation is mysterious. This seems to be confirmation of the ready, FIRE, aim methodology whereby the text reports how big the sample size ought to have been. The other classic symptom is the small sample size listed as a limitation:
Relatively small sample size and use of ADHD-RS-IV as the sole outcome measure are other limitations of our study.
A post-hoc analysis is reported:
Results of post hoc tests showed significant improvement of total and subscale scores in each post-baseline visit compared with previous visits in both groups…
We feel this analysis is misguided, the difference between the groups (the treatment effect) should be the only comparison undertaken.
In our opinion the Discussion section leads with an unwarranted conclusion:
…we showed that both bupropion and methylphenidate could safely and effectively improve the ADHD symptoms.
This was not the question the trial aimed to answer (“To compare the safety and efficacy of bupropion with methylphenidate…”). This claim is only warranted with the inclusion of a placebo arm.
It is 17 years since Altman and Bland published Absence of evidence is not evidence of absence. All trialists and journals should be aware of this basic fact:
The effect of time x treatment interaction was not signiﬁcant, showing that the behavior of both treatments was similar across time.
This is not correct; The study is almost certainly underpowered for the detection of interactions and so does not offer adequate evidence “showing” the treatments were “similar across time”.
Although we showed equal efﬁcacy of Bupropion and methylphenidate…
A small, underpowered trial which reports no significant difference has not established “equal efficacy”. The question remains to be answered.
Similar effects were observed for the time, time x treatment interaction, and the treatment group in inattention and hyperactivity sub-scales of the Parent ADHD-RS-IV (data not shown).
These results must be supported by evidence – “data not shown” is not acceptable.
The text also falls for the RCT safety fallacy:
Frequency of other side effects did not differ significantly between the two groups.
22 patients per group is probably inadequate for establishing a reliable safety profile.
Design a study to ensure you have a large enough sample size to detect a clinically meaningful difference.
Do not conflate “no significant difference” in a small study with therapeutic equivalence.
Power is indeed irrelevant in interpreting completed studies
The baseline significance testing howler
LOCF – the earth is not flat
Low powered comparisons are futile
Absence of evidence is not evidence of absence
The RCT safety fallacy
Questions & comments for authors:
Care needs to be taken when reporting results small trials. We feel the Discussion section has overstated claims for the effectiveness of bupropion.
The authors are guaranteed this space for replies/rejoinder
Questions & comments for the journal:
It is 17 years since Altman and Bland published Absence of evidence is not evidence of absence (here is an easy to understand example, p-0.8669). All modern, scientific journals should be aware of this basic fact. Publishing underpowered results as evidence for therapeutic equivalence is an obvious and easy mistake to detect. All efforts should be made in future not to allow this to happen.
The journal is guaranteed this space for replies/rejoinder
Questions & comments for readers:
Comments are welcome in the Discussion section below.
Click the link directly below for detailed CONSORT based appraisal.