Education: The machine gun fallacy


salient (sālyənt/ -lēənt/) adjective: 1 Arresting, striking, most noticeable or important, 2 Prominent, conspicuous, 3 (of an angle) Pointing outward

scattershot (skatərˌSHät/) adjective: 1 Denoting something that is broad but random and haphazard in its range

A good trial is salient. Science grapples with large questions best by breaking them down into a finite number of small, related and very sharply defined questions. While it is natural to want to extract (and report) as much information as possible from your trial a competent trial is designed to definitively answer one particular question as its primary objective. Other findings can and should be reported but as secondary or exploratory findings but these must be handled in a different manner.

In psychiatry there are many important hypotheses of interest and yet relatively few resources available to investigate these:

It is common to see investigators try to test as many hypotheses as possible with their study. This is easily identified by co-primary objectives, a large number of secondary objectives, exploratory and subgroup results, post-hoc analyses, unspecified analyses, absence of a pre-specified analysis plan and a proliferation of p-values cluttering the Results section and figures. Unfortunately rather than advance knowledge with the best of intentions this approach actually hinders it.

As you test more hypotheses the chances of a false positive discovery increase dramatically. Rather than answering one question reliably (thus controlling the rate of false positive error) the machine-gun analysis technique answers a large number of questions unreliably. All results are under a cloud of false positive error, readers don’t know which to trust.

It takes discipline but a salient, trenchant analysis takes careful aim at one specific  pre-specified hypothesis.

Beware of shapeless, out of focus, scattershot clinical trial results. Beware of articles where the primary objective slips from sight and is usurped by other results. By having a predefined set of trial objectives a kind of structure is imposed on the on the entire investigation. This calls for discipline, the rigid structure can chafe but it is essential to ensure the results are trustworthy.

Beware the machine-gun fallacy.

Suggested reading:
Disciplined analyses in clinical trials: the dark heart of the matter (try and read everything by Lemuel A Moyé)

Clinical research analyses must balance the desire to `learn all that is learnable’ from the database with the observation that sample-based data commonly lead to conclusions that are perfectly correct for the sample, but wholly incorrect for the population from which the data were based.

Investigators who defend exploratory analyses as reliable, misuse important tools that have taken over three hundred years to develop.