BMJ  2004;328:473 (28 February), doi:10.1136/bmj.328.7438.473

Editorial

What doesn't work and how to show it

Ineffectiveness is hard to prove and accept

Archie Cochrane, the British epidemiologist, posed three key questions to ask about a healthcare intervention: "Can it work?" "Does it work in practice?" and "Is it worth it?"¹ It would be great if the answers to these were always positive, but life isn't like that. The possible answers might be classed as "yes," "not sure," and "no." The rules for deciding "yes" are relatively clear and well known, but less has been written about deciding that something doesn't work. This theme issue looks at examples of interventions that don't appear to work, the dilemmas of trying to decide between an answer of "not sure" and "no," and what to do when we are not sure.

Doust and Del Mar admit (p 474)² that "Don't just do something, stand there!" sounds ludicrous. But this is sometimes good advice, as their round up of commonly used but apparently ineffective interventions shows. This issue also includes new evidence of varying kinds and strengths on other interventions that didn't seem to have a marked effect: a systematic review of mentoring for children with behavioural problems (p 512);³ randomised controlled trials of adenoidectomy for children with recurrent otitis media (p 487);⁴ rocking exercises in late pregnancy to turn babies from a posterior lie (p 490);⁵ low dose ramipril to reduce cardiovascular and renal complications among diabetic people with microalbuminuria (p 495);⁶ and an n-of-1 trial of vitamin B-6 for nausea and vomiting in pregnancy (p 503).⁷

In each case the type of evidence, its quality, and the potential benefits and harms might lead us to be more or less convinced that an intervention does more good than harm. This is very much a judgment, and if people want to believe there is an effect it can be very hard to persuade them that any effect is too small to be important. As a school textbook puts it when discussing the dilemma of industries or official bodies trying to convince the public that something poses no risk: "It is very hard to persuade people that a factor has no effect. This involves `proving a negative'."⁸

When it is difficult to judge effectiveness and benefit from new research, we need all the help we can get from existing evidence. Two articles this week develop this point. McPherson and Hemminki argue that if trials done by pharmaceutical companies to obtain drug licences had to include adequate data on harms and ineffectiveness we could learn much more quickly what we need to know about new drugs (p 518).⁹ Pound et al propose systematic reviews and meta-analyses of the animal studies that supposedly underpin much human clinical research.¹⁰ Such reviews could assess the validity and generalisability to humans of animal research and could prevent unnecessary treatment trials.

Even with all the evidence we can muster, we are often left being uncertain about the right treatment choice.¹¹ As US defense secretary Donald Rumsfeld said at a press briefing on Iraq: "Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know."¹² This much ridiculed statement won the Plain English Campaign's Foot in Mouth award for 2003, but it does make sense when read carefully. Perhaps what we don't know we don't know would be a good topic for a BMJ theme issue: let us know if you agree.

UK Cochrane Centre, Oxford OX2 7LG (palderson{at}cochrane.co.uk)

BMJ (tgroves{at}bmj.com)

---

Competing interests: None declared.

References

1. Haynes B Can it work? Does it work? Is it worth it? BMJ 1999;319: 652-3.[Free Full Text]
2. Doust J, Del Mar C. Why do doctors use treatments that do not work? BMJ 2004;328: 474-5.[Free Full Text]
3. Roberts H, Liabo K, Lucas P, DuBois DL, Sheldon TA. Mentoring to address anti-social behaviour in childhood. BMJ 2004;328: 512-4.[Free Full Text]
4. Koivunen P, Uhari M, Luotonen J, Kristo A, Raski R, Pokka T, Alho OP. Adenoidectomy versus chemoprophylaxis and placebo for recurrent acute otitis media in children aged under 2 years: randomised controlled trial BMJ 2004;328: 487-90.[Abstract/Free Full Text]
5. Kariminia A, Chamberlain ME, Keogh JM, Shea A. Randomised controlled trial of the effect of hands-and-knees posturing on the incidence of occiput posterior position at birth. BMJ 2004;328: 490-3.[Abstract/Free Full Text]
6. Marre M, Lievre M, Chatellier G, Mann JFE, Passa P, Menard J. Effects of low-dose ramipril on cardiovascular and renal outcomes in patients with type 2 diabetes and elevated excretion of urinary albumin: a randomised, double-blind, placebo, controlled trial (the DIABHYCAR study). BMJ 2004;328: 495-9.[Abstract/Free Full Text]
7. Harker N, Montgomery AA, Fahey T. Treating nausea and vomiting during pregnancy: case outcome. BMJ 2004;328: 503-6.[Free Full Text]
8. Hunt A, Millar R, eds. AS science for public understanding. Oxford: Heinemann, 2000: 48.
9. McPherson K, Hemminki E. Synthesising licensing data to assess drug safety. BMJ 2004;328: 518-20.[Free Full Text]
10. Pound P, Ebrahim S, Sandercock P, Bracken M, Roberts I. Where is the evidence that animal research benefits humans? BMJ 2004;328: 514-7.[Free Full Text]
11. Chalmers I. Well-informed uncertainties about the effects of treatment. BMJ 2004;328: 475-6.[Free Full Text]
12. The Acronym Institute. Disarmament documentation. Back to disarmament documentation, June 2002. Defense secretary Rumsfeld press conference, June 6. "Secretary of Defense Donald H. Rumsfeld, press conference at NATO headquarters, Brussels, Belgium, June 6, 2002," US Department of Defense transcript. www.acronym.org.uk/docs/0206/doc04.htm (accessed 20 Feb 2004).

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

If it doesn't work, stop it: Medicine is a science of prediction and intervention

Ludovic Reveiz
BMJ 2004 328: 1015. [Extract] [Full Text]

If it doesn't work, stop it: I don't know

Simon M Loader
BMJ 2004 328: 1016. [Extract] [Full Text]

If it doesn't work, stop it: Summary of rapid responses

Caroline White
BMJ 2004 328: 1016. [Extract] [Full Text]

Nothingness: the role of journals

Richard Smith
BMJ 2004 328: 0. [Extract] [Full Text] [PDF]

Why do doctors use treatments that do not work?

Jenny Doust and Chris Del Mar
BMJ 2004 328: 474-475. [Extract] [Full Text] [PDF]

Treating nausea and vomiting during pregnancy: case outcome

Nicola Harker, Alan Montgomery, and Tom Fahey
BMJ 2004 328: 503. [Extract] [Full Text] [PDF]

Mentoring to reduce antisocial behaviour in childhood

Helen Roberts, Kristin Liabo, Patricia Lucas, David DuBois, and Trevor A Sheldon
BMJ 2004 328: 512-514. [Extract] [Full Text] [PDF]

Synthesising licensing data to assess drug safety

Klim McPherson and Elina Hemminki
BMJ 2004 328: 518-520. [Extract] [Full Text] [PDF]

Randomised controlled trial of effect of hands and knees posturing on incidence of occiput posterior position at birth

Azar Kariminia, Marie E Chamberlain, John Keogh, and Agnes Shea
BMJ 2004 328: 490. [Abstract] [Full Text] [PDF]

Effects of low dose ramipril on cardiovascular and renal outcomes in patients with type 2 diabetes and raised excretion of urinary albumin: randomised, double blind, placebo controlled trial (the DIABHYCAR study)

Michel Marre, Michel Lievre, Gilles Chatellier, Johannes F E Mann, Philippe Passa, and Joël Ménard
BMJ 2004 328: 495. [Abstract] [Full Text] [PDF]

Adenoidectomy versus chemoprophylaxis and placebo for recurrent acute otitis media in children aged under 2 years: randomised controlled trial

Petri Koivunen, Matti Uhari, Jukka Luotonen, Aila Kristo, Risto Raski, Tytti Pokka, and Olli-Pekka Alho
BMJ 2004 328: 487. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Loader, S. M (2004). If it doesn't work, stop it: I don't know. BMJ 328: 1016-1016 [Full text]  
• Reveiz, L. (2004). If it doesn't work, stop it: Medicine is a science of prediction and intervention. BMJ 328: 1015-1015 [Full text]  
• (2004). Hit Parade. BMJ 328: 962-962 [Full text]  

Rapid Responses:

Read all Rapid Responses

Over-reliance on pharmacotherapy in prison-based smoking cessation programs doesn’t work: to prove it, just ask smoker prisoners and prison anti-tobacco workers.

Niyi Awofeso
bmj.com, 27 Feb 2004 [Full text]

Cause and effect?

Susan R Corby
bmj.com, 27 Feb 2004 [Full text]

I agree

Emmanuel UDEZUE
bmj.com, 27 Feb 2004 [Full text]

‘Well after all what do any of us really know?’ ‘That’ll be enough of that, Wilson’

David G Connell
bmj.com, 2 Mar 2004 [Full text]

Unknown Unknowns

Christopher E Nancollas
bmj.com, 2 Mar 2004 [Full text]

So many unknowns, such serious consequences

Elizabeth M Witherington
bmj.com, 2 Mar 2004 [Full text]

What doesn't work and how to show it

Alan R Giles
bmj.com, 5 Mar 2004 [Full text]

Patient uncertainty - scientific uncertainty

Robert Fox
bmj.com, 7 Mar 2004 [Full text]

I don't know

Simon M Loader
bmj.com, 7 Apr 2004 [Full text]

Response to "What doesn't work and how to show it"

Joaquim P Ribeiro, et al.
bmj.com, 14 Jun 2004 [Full text]