Ranked #1 during May 2012: A Randomized Controlled Trial of CBT Therapy for Adults with ADHD with and without Medication.
Study design & execution:
A subset of data from a previous study is taken to compare stimulants with placebo in adult patients with ADHD all of whom receive cognitive based therapy (CBT). The study objective is spelt out as follows:
The objective of the study reported here is to determine whether CBT psychotherapy in combination with medication is superior to CBT psychotherapy and placebo in mitigating core symptoms of ADHD and functioning
The multi-centre, five month study appears to be well designed with random allocation to treatment and blinding. Completion rates of 62% (stimulant) and 77% (placebo) are reported.
Analysis & Reporting:
The analysis gets off to an inauspicious start by committing the baseline data testing howler. The primary analysis finds no significant difference between the stimulant and placebo treatment groups. The investigation then changes tack by pooling both treatment groups to analyse the data as a single arm CBT study.
Analyses of both groups as a whole for efficacy of CBT will be done, if it was found that there are no between group differences.
This additional analysis is ill-founded – the “C” in RCT stands for Controlled. All patients in this study received CBT treatment thus without a control arm any claims regarding the “efficacy” of CBT are completely unwarranted from this study design. In order to gain insights into the efficacy of CBT the study design must ensure one group is treated with CBT while simultaneously withholding CBT treatment from the other group.
Unfortunately this unwarranted analysis usurps the original primary objective and becomes the top line result:
(Abstract) Conclusion: This study replicates previous work demonstrating that CBT is an effective treatment for ADHD in adults.
Conclusion: This study supports previous findings that adults with ADHD are responsive to CBT intervention.
From a statisticians’s perspective the above conclusion is inadmissible, this result cannot be considered in the evidence base for CBT as a treatment for ADHD in adults.
Consider the article title:
A Randomized Controlled Trial of CBT Therapy for Adults with ADHD with and without Medication
This is potentially confusing; it is not a randomised trial of CBT therapy (CBT wasn’t randomly allocated, all patients received it), nor is it a controlled trial of CBT (all patients received CBT). As a result this trial may be automatically indexed inappropriately in various electronic databases.
To help ensure that a study is appropriately indexed and easily identified, authors should use the word “randomised” in the title to indicate that the participants were randomly assigned to their comparison groups. (CONSORT item 1A)
Avoid co-primary objectives. Choose either ADHDRS-IV scale or SDS as the primary efficacy variable, relegate the other to a supportive secondary efficacy variable. The ranked objectives provide a ‘structure’ to the investigation and shouldn’t be re-arranged when reporting the results.
Fit a mixed effects model to the longitudinal data, report the observed treatment effect (i.e. the difference in means between the 2 groups at Week 20) and its 95% confidence interval. Report the treatment*time interaction term from the longitudinal model. Repeat the longitudinal analysis with alternative strategies for handling the missing data. Compare results in order to get a feel for the sensitivity of the analysis to the different ways of handling missing data.
The following links cover relevant statistical issues in more detail:
the baseline data significance testing howler
confidence intervals instead of p-values
the treatment effect
re-shuffling the objectives
low powered comparisons are futile
beware the one-armed bandit
LOCF – the earth is not flat
Questions & comments for authors:
We would caution the authors against performing significance testing of baseline data and, more importantly, in situations where there is little power to make valid comparisons. Above all we strongly caution that investigations of “efficacy” cannot be undertaken using a single arm study design.
Authors are guaranteed this space for replies/rejoinder
Questions & comments for the journal:
In our opinion the top line result reported in this article is inadmissible.
In the pre-publication section reviewers comments include:
The methodology is generally very sound, and the use of statistics seems appropriate…
No, the manuscript does not need to be seen by a statistician. x3
This highlights an interesting issue whereby reviewers decide whether or not a statistician is required for peer review.
Bias can occur not only in the analysis of a trial but also, in a more subtle form, in its design. It would seem the reviewers were prepared to accept a conclusion “…demonstrating that CBT is an effective treatment for ADHD” from a study which did not have a control arm necessary to demonstrate this effect. It also escaped the reviewers attention that the top-line result did not match the stated primary objective.
This is disappointing given BMC psychiatry has a useful “Guide for BMC Psychiatry reviewers” including the following from the “Points to consider” section:
2 Are the data sound and well controlled? If you feel that inappropriate controls have been used please say so, indicating the reasons for your concerns, and suggesting alternative controls where appropriate. If you feel that further experimental/clinical evidence is required to substantiate the results, please provide details. http://www.biomedcentral.com/bmcpsychiatry/about/reviewers
Two simple policies may be useful: Do not consider results from uncontrolled (one armed) studies for publication (i.e. stick to the above guidelines) and always always ensure the top-line result matches the stated primary objective. These policies would not require a statistician to uphold.
The journal is guaranteed this space for replies/rejoinder
Questions & comments for readers:
Comments are welcome in the Discussion section below.
Click the link directly below for detailed CONSORT based appraisal.