Findings of study should prompt debate

EDITOR,—P Gill and colleagues' adaptation to a general practice setting1 of a study originally designed to assess interventions in an acute hospital medical firm2 encouraged me to apply their methodology to acute admissions (n=50) over four weeks in the paediatric department of a district general hospital. My finding that, by Gill and colleagues' criteria, two thirds of primary interventions in this setting were evidence based is perhaps less interesting than the flaws in their study that were highlighted by my attempt to emulate it.

Firstly, Gill and colleagues cite individual randomised controlled trials and state that they did not attempt to assess the methodological quality of the trials identified. In my study at least four diagnosis-intervention pairs could be supported or contraindicated depending on which of two conflicting randomised controlled trials one chose to quote. Differences in the date of publication were not great enough to dictate the choice; an accurate assessment of trial strength is vital in such cases. Ellis et al's solution to this problem was to use overviews in addition to randomised controlled trials.2

Secondly, the treatments that fell into Gill and colleagues' category (ii)—“intervention based on convincing non-experimental evidence”—were decided by a consensus of practitioners. Because of the nature of interventions in the paediatric department that I studied, this was the criterion that I adopted. The inclusion criteria for this category were therefore vastly different from those of Ellis et al, whose category (ii) interventions, such as cardiopulmonary resuscitation, were those “whose face validity is so great that randomised trials were unanimously judged by the team to be both unnecessary and, if a placebo would have been involved, unethical.”2 The general practice study, like mine, therefore included within the authors' definition of evidence based interventions a large number of treatments that proponents of evidence based medicine would call non-evidence based.

Such studies are useful for assessing the scientific basis of treatment. When, however, randomised controlled trials are not examined for power and a consensus of practitioners is substituted for such trials in some cases, the finding that two thirds or more of interventions are evidence based is less a cause of satisfaction than a source of debate.