What doesn't work and how to show it
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7438.473 (Published 26 February 2004) Cite this as: BMJ 2004;328:473
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Alderson & Groves suggest that ‘what we don’t know we don’t know
would be a good topic for a BMJ theme issue’.
But, do you know what? It couldn’t be done. For to write about what we
don’t know, we must sureley know we don’t know it first, otherwise how
could it be an issue?
The only way it could work would be that those who know they don’t know
something, but think the rest of us don’t know we don’t know it, write
about it so that the rest of us then also know we don’t know it. Then
everything in that issue will no longer be unknown unknowns, but known
unknowns.
Do you know what I mean?
Competing interests:
None declared
Competing interests: No competing interests
I read the collated articles about the assessment of clinical
effectiveness in the February 28th edition with great pleasure. Alderson
& Groves stated that ‘the rules for deciding "yes" are relatively
clear and well known, but less has been written about deciding that
something doesn't work.’ They discuss the reasons why it can be difficult
to determine that an intervention is ineffective. I would like to propose
a possible reason why we cannot be certain about effectiveness.
The double-blind, placebo-controlled, random allocation trial is seen
as the gold standard of study design for testing new interventions. It is
thought to circumvent the problems of observer bias and to control for the
placebo effect. However, as well as a positive placebo effect, can we be
sure that the effect of any uncertainty of random allocation on the
patient does not have a negative effect on the efficacy of the active
agent? And could such an effect would underscore the effectiveness of the
active agent?
One could imagine a study of a novel intervention with a 40%
effectiveness, which is reduced by one third by the uncertainty of
allocation, for a symptom with a placebo response rate of 40%. The result
of a placebo trial would show as a 51% response for the active arm (27%
true effect and 24% placebo effect) and 40% for the placebo. If properly
powered, the research would conclude that the active agent was effective.
An open study would have tended to overestimate the effectiveness but a
blinded study would underestimate the value of the drug. The result of
such an effect could erroneously lead to a conclusion that an agent is
insufficiently effective (NNT too high), or result in an underestimate of
cost-effectiveness. It might also limit conclusions of n=1 trials. If such
an effect does exist one might expect it to apply more to the subjective
assessment of symptoms such as pain or nausea as opposed to objective
disease markers.
We should always bear in mind that it is not nature we onserve but
nature exposed to our method of questioning.
Competing interests:
None declared
Competing interests: No competing interests
It is a pity that in what is otherwise an excellent and thought
provoking editorial, Alderson & Groves appear to provide credibility
for the ruminations of Donald Rumsfeld by suggesting that most of us have
missed his point.1 Surely we can only be unaware of what we do not know by
simply not asking the question either by default or deceit. We would be
deluding ourselves, however, if we assumed that politicians have the
monopoly of such manoeuvres.
During my career in medicine, I have oscillated through a number of roles,
starting in very basic clinical science and then passing through primary
care and pharmaceutical medicine before settling in academic medicine as,
I hope I can describe myself, a clinical scientist. During these varied
experiences I have seen many examples of the situations described in the
articles of this excellent issue of the BMJ. I would draw on two that I
believe are germane to the debate.
During my stay in primary care, which came relatively soon after
qualification, I became very quickly aware and threatened by how
relatively little I actually knew. This was particularly with regard to
the recurrence of rubellaform rashes in otherwise relatively well children
where rubella had been ‘diagnosed’ some months or years previously. This
lead me to adopt an ‘educational’ policy including an open discussion with
the anxious parent and a frank admission of how little we knew about such
things thus denying a firm diagnostic title. Sometimes it worked but more
often than not it did not. The likely reason for this became clear when I
noticed that one particularly dissatisfied parent had re-consulted my
senior partner the same day and left satisfied with a ‘diagnostic title’
and prescription for antibiotics for her child. On challenging this
decision, I was given paternalistic advice on the ‘realities’ of general
practice and the need to prioritise my time commitment between education
and action. Fortunately, I left general practice before I was forced to
consider this advice seriously but I am sure my advisor was not alone in
following the path of least resistance rather than “just standing there
and doing nothing”.
The second, more positive, experience occurred when I re-entered academic
medicine. Perhaps overly sensitive to criticism of my time spent off the
academic ladder, initially I had a tendency to bluster when on uncertain
ground. My mentor, who had a propensity for scientific and personal
directness, quickly provided the diagnosis and the correct remedial
treatment. Simply put, I was advised never to be afraid of admitting
ignorance (an ugly word but to the point!) no matter how ignominious the
immediate consequences could be. The long-term benefits were twofold.
Firstly it was a question of self-preservation. History tends to judge
individuals kindly when ignorance has been freely admitted. It is less
charitable when this is initially denied but uncovered later. Secondly,
and probably of primary importance, it is only by admitting what is not
known that the need to know is appreciated and investigated to the limit
of each individual’s resources. It is not good enough to say, “we don’t
know what we don’t know”. Denial is a powerful force in both the clinical
state and its managers. In both cases, its release, when handled
appropriately, can be extraordinarily beneficial.
It would be foolish to underestimate the pressures under which physicians
and primary care physicians in particular work and the burgeoning
responsibility imparted by the general public’s faith in their physician’s
complete understanding of the ‘secrets of life’. Over the history of
medicine we have used this faith, or should we call it trust, to benefit
our patients in their hours of medical need.
Unfortunately, it is also
open to significant abuse. Although still woefully incomplete, there has
been an exponential increase in our knowledge of human disease and its
management. Now is surely the time for us to shed defensive postures and
admit we do not know when we do not because we have asked the question and
the answers were not forthcoming. This lesson should be taught in our
medical schools. It was not a message I was taught in the sixties and I
suspect nothing has changed. It should.
1. Alderson P. Groves T. What doesn’t work and how to show it. BMJ
2004;328:473
Competing interests:
None declared
Competing interests: No competing interests
EDITOR- 'What we don't know we don't know' is a
familiar concept in General Practice. I remember being
introduced to this idea as a GP trainee in 1989: at first it
seemed almost a nonsense - of course we know what
we don't know- how could we not, after so many years'
training? It's only through experience that we come to
appreciate the extent of our ignorance. It is a further,
more difficult step to be able to accept this in our work:
we are trained to 'know'. In medical training and
practice, ignorance is so often equated with failure.
Perhaps this is why we are reluctant to admit our
limitations. Yet, what we don't know we don't know is at
the heart of so many examples of poor communication
and adverse outcomes for patients, eg
-the doctor who is 'often wrong but never in doubt'.
-the attitude that, because the patient's symptoms don't
add up to a diagnosis I can recognise, the patient must
be either a) depressed b) mad or c) 'swinging the lead'.
There is nothing so instantly depressing as being told
by a doctor that troubling symptoms do not exist or
matter, and that the consultation is closed. It is more
time-consuming, but ultimately more constructive to
admit ignorance, (having, as far as possible, excluded
important serious conditions), enable patients to get on
with their lives in spite of their symptoms and permit
review if things change or get worse.
- the classic GP consultation where the presenting
symptom is just an 'entry ticket' for the patient to test out
the doctor's receptiveness to a much more troubling or
embarrassing problem. How many brief 'pointless'
consultations are required before the true complaint is
broached?
-communication between medical colleagues: the
absent or vanishingly brief GP letter which
compromises care for emergency admissions.
-the absent or incomplete hospital discharge
communication which contributes to adverse events.
Local examples include patients who died because the
discharge information did not include significant results
or monitoring recommendations. The GP cannot know
what information is missing. Our local audit shows that
about 15% of older patients had problems with
medication within 4 days of discharge.
With the new GP contract, patients are less likely to see
a doctor who knows them out of hours. With changes in
junior doctors' working patterns, immediate discharge
information (the 'TTO form') is less likely to be written by
a doctor with extensive knowledge of the patient.
This combination increases the risks for those already
at greater risk of adverse events, ie complex, often older
patients, who are more likely to be discharged to
different destinations (intermediate and residential
care), and new GPs. The TTO form is often the only
form of discharge communication available for several
days or weeks.
It is at the same time more important and more difficult
to get it right for these patients.
In summary, communication between doctors has
been shown to be poor for at least 20 years. New
working practices mean that there are even more
things that we don't know we don't know about patients,
which are putting lives at risk. If ever there was a time to
push for improvement, it is now.
Competing interests:
None declared
Competing interests: No competing interests
Sir,
It would be an excellent idea to have a theme issue on medical unknowns.
After 25 years in medicine, it seems to me that there are far more unknown
unknowns than there are known knowns, or, to put it slightly differently,
there are very few 'facts' I learnt at medical school which are facts now.
In a world where targets and NSF's are driving the practice of medicine,
and patients expect certainty, it is important to acknowlege what we don't
know.
You must be the only publication which has used Donald Rumsfeld's phrase
seriously, which I admire because I always thought it made sense. In
Cornwall we used to have a phrase ' He don't know enough to know he knows
bugger all' which is sort of the same thing.
Yours sincerely,
Chris Nancollas
Competing interests:
None declared
Competing interests: No competing interests
‘Well after all what do any of us really know?’ ‘That’ll be enough of
that, Wilson’(1)
I write to offer my wholehearted support to a BMJ issue exploring
‘What we don’t know we don’t know.’ Could I suggest en passant for the
dismissive non-believer that some small consolation is offered by a print
run on a slightly more absorbent paper?
I was ashamed to speak up that day and admit that Rumsfeld’s
cumbersome statement held distinct resonance.
As a GP I spend my day being appraised of those areas of my knowledge
that are deficient. It is very educational. I am now being rewarded for
keeping a running tally of all those things I didn’t know.
It strikes me that it is harder for my Specialist colleagues to speak
up about their educational needs, but how would I know that?
It strikes me that Richard Feynman (2)was close to the truth in 1963:
‘In talking about the impact of ideas in one field on ideas in
another field, one is always apt to make a fool of oneself.’
‘Some people say ‘How can you live without knowing?’ I do not know
what they mean. I always live without knowing. That is easy. How you get
to know is what I want to know’
There is much to discuss in this arena – perhaps too much, but let us
at least sow the first seeds of humility for future generations of
patients and doctors to reap.
And I do still like the tale of the ‘Emperors New Clothes’- and if
you are sitting comfortably children our story today will take us through
– the Johari window.
David G Connell
1 Dad’s Army BBC All of our lives
2 Feynman, Richard P. A Scientist looks at Society Cal.Tech 1963
Competing interests:
Ego,Id and the ridicule of my colleagues
Competing interests: No competing interests
Doctors need to be aware of, and avoid, intellectual pride. True
education should humble us, not make us proud. More knowledge makes us
more aware of the things we do not know. And it is the quest to find out
more about these unknowns that extend the frontiers of knowledge.
Competing interests:
None declared
Competing interests: No competing interests
Human resource management interventions are even more problematic
than healthcare interventions. Nevertheless, the Department of Health is
commissioning research on improving health though human resource
management (HRM). It wants to find out which human resource (HR)
interventions are most effective in “increasing productivity and achieving
better outcomes for patients”.
Meanwhile researchers from Aston University UK in a study funded by
the National Health Service (NHS) Executive (north Thames region) have
already found “strong links” between the presence of sophisticated
personnel practices, particularly appraisals, teamwork and training, and
lower patient mortality in acute hospitals. Between 1999-2000 the
researchers collected information on personnel policies, procedures and
practices from 61 NHS trusts. In addition, they collected performance
data on patient mortality, using complex statistical procedures which
included standardising for such variables as patients’ ages and primary
diagnosis.
Yet just because the researchers found an association does not mean
that there is a link, whatever their claims. Statistical associations can
be spurious and misleading as King’s Fund researchers demonstrated. They
found an association between the World Health Organisation’s (WHO’s)
rankings of the health system performance of a country and the country’s
football rankings by the Federation Internationale de Football Association
(FIFA).
Also, there could be reverse causation. The Aston researchers’
information on personnel practices referred to the period 1999-2000 and
the mortality data spanned the period 1995-2000. So lower patient
mortality could have caused sophisticated personnel policies.
Unfortunately an HR intervention is even more difficult to measure
than a medical intervention, not least because of the problems of
measuring health service productivity. (Is it quality or quantity?)
Indeed, research by the University of Greenwich into how one HR
intervention, a new pay system, was evaluated found that health service
HR practitioners were sceptical that links could be demonstrated between a
personnel policy and practice and productivity or health outcomes.
The Department of Health, however, is not deterred. In conjunction
with the Chartered Institute of Personnel and Development and the
Association of Health Human Resource Management it is offering funds of
up to £250,000 for research on how human resource management can
contribute to “positive employee attitudes and behaviours and to improved
health service performance” with the possibility that the research may be
extended with additional funding.
Researchers from cash strapped universities are queuing up. Perhaps
the money would be better spent on reducing patient queues.
References
www.doh.gov.uk/callsforproposals.htm (accessed 18 Feb 2004)
West MA, Borrill C, Dawson J, Scully J, Carter M, Anelay S. et al.
The link between the management of employees and patient mortality in
acute hospitals. Int. J. of Human Resource Management 2002;13:1299-1310
Appleby J, Street A. Health system goals: life, death and… football.
J of Health Services Research & Policy 2001;6:220-225
Corby S, White G, Dennison P, Douglas F, Druker J, Meerabeau E. Does
it work? evaluating a new pay system. London: University of Greenwich,
2002
Competing interests:
None declared
Competing interests: No competing interests
“Professor Aristotle would have avoided the tragic misconception that men had three more teeth than women if he had simply, and courteously, asked Mrs. Aristotle to open her mouth” - Anonymous.
Although there is a clear political and professional risk in trying to challenge medical and public health zeitgeists1, failure to do so when proof of their ineffectiveness is evident can only harm the progress of the health sciences, as reasons for the lag of polio research in the early decades of the 20th century demonstrate.2
Over the past decade, interest in implementing tobacco control programs, and basic smoking cessation (‘Quit’) programs in Western prison settings has heightened, due largely to litigious and public health reasons. For instance, the United Kingdom’s Department of Health has recently agreed to fund Nicotine Replacement Therapy (NRT) for 73,000 British prisoners, about 80% of who smoke (http://www.sepho.org.uk/viewResource.asp?).
The addiction model is a dominant framework adopted by most health workers involved with smoking cessation programs (http://www.rcplondon.ac.uk/pubs/books/nicotine/4-addiction.htm), and all but two of BMJ’s “ABC of smoking cessation” series have emphasized this model. Consequently, provision of NRT to prison populations - epicenters of an ignored public health epidemic3 – is ‘naturally’ regarded by health workers as the antidote to one of the greatest hurdles to setting up prison Quit programs. However, inmates have markedly different perceptions, as a recent survey in an Australia prison shows (Table 1).
Table 1: Smoking cessation obstacles in an Australian prison – prisoners’ survey. Smoker prisoners Non-smoker prisoners Average Inmates’ count during survey week 160 Number of survey respondents 67 20 Number that believe it is more difficult to quit in prison than on the ‘outside’ 58 7 Of participants that believe that it is more difficult to quit in prisons than on the ‘outside’, reasons given (i.e. frequency of each listed response): 1) Boredom 32 6 2) Stress 39 4 3) Everyone else smokes 24 5 4) Addiction 10 5) Helps cope with depressing prison environment 9 1 5) Nothing better to spend money on in prisons 6
Only 7% of responses by smoker inmates were related to addiction as an obstacle for smoking cessation, compared with 23% for boredom, 28% for stress, and 17% for high smoking prevalence. Although the enhanced accessibility of NRT in this prison (for the 40% of smoker inmates serving sentences of six moths or greater) explains the fact that only one of 139 responses cited NRT cost as an obstacle to smoking cessation, only about 9% cited NRT availability as a major obstacle in the author’s unpublished 2001 survey of 1068 Australian prisoners, when NRT was not provided free of charge to inmates in the surveyed prisons. It is also noteworthy that while 88% of smoker prisoners surveyed believed that it was more difficult to quit tobacco use in prison compared with community settings, only 35% of non-smoker prisoners shared the same view.
If prison-based smoking cessation efforts are to yield optimal results it is important for health workers to adopt a social marketing framework4, and address Quit obstacles through smoker prisoners’ perspectives. This paradigm shift should involve a greater appreciation of the core values of prisoners, and the promotion of smoking cessation programs in such a manner that it accords with those core values. Furthermore, the primary objectives of prison-based smoking cessation programs should be framed as having a clear benefit for inmates not necessarily from a health perspective, but perhaps more importantly, in terms of satisfying prisoners’ core values.
Over-emphasis on an addiction model, or a health education model (inadequate knowledge about tobacco’s health effects was regarded as a major impediment to Quit efforts by only about 2% of inmates in the author’s 2001 unpublished survey), at the expense of obstacles accorded greater importance by inmates, such as boredom or stress, is indicative of a “selling approach” to facilitating social change, and is therefore less likely to result in successful prison-based Quit program outcomes. Admittedly, the reality of prison life is such that most major obstacles perceived by inmates cannot be fully surmounted without compromising prison security. Thus, the use of strategies to enhance motivation and self-efficacy to sidestep such obstacles5 should be integral to prison Quit programs.
References
1. Alderson P., Groves T. What doesn’t work and how to show it. British Medical Journal 2004; 328: 473.
2. Benison S. Polio research in the United States: appraisal and lessons. In: Holton G, editor. The 20th century sciences – studies in the biography of ideas. New York, W. W. Norton & Company, 1972: 308-343.
3. Cropsey K., Eldridge GD, Ladner, T. Smoking among female prisoners: an ignored public health epidemic. Addictive behaviors, 2004; 29: 425-31.
4. Awofeso, N. Implementing smoking cessation programmes in prison settings. Addiction Research and Theory, 2003; 11: 119-130.
5. Secker-Walker RH, Flynn, BS, Solomon L.J. et al . Helping women quit smoking: baseline observations for a community health education project. American Journal of Preventive Medicine, 1996; 12: 367-377.
Competing interests:
None declared
Competing interests: Table 1: Smoking cessation obstacles in an Australian prison – prisoners’ survey. Smoker prisoners Non-smoker prisonersAverage Inmates’ count during survey week 160 Number of survey respondents 67 20Number that believe it is more difficult to quit in prison than on the ‘outside’ 58 7Of participants that believe that it is more difficult toquit in prisons than on the ‘outside’, reasons given (i.e.frequency of each listed response): 1) Boredom 32 62) Stress 39 43) Everyone else smokes 24 54) Addiction 10 5) Helps cope with depressing prison environment 9 15) Nothing better to spend money on in prisons 6
Response to "What doesn't work and how to show it"
I agree with you when you say "Perhaps what we don't know we don't
know".
I found it curious that I also finish my article "Normalities,
abnormalities and interspace diseases" with a similar phrase. It was
publised in "Anamnesis, year 9, nº 94, November 2000, page 15".
I said in portuguese: "desconhece-se muito do que se julga conhecer e
desconhece-se ainda mais do que se julga desconhecer"; (In english
something like: "great is our unawareness of how little we actually know,
but greater is still our unawareness of how much we do not know".)
Best Regards
J Palmeiro Ribeiro
Competing interests:
None declared
Competing interests: No competing interests