Building a framework for trust: critical event analysis of deaths in surgical care
BMJ 2005; 330 doi: https://doi.org/10.1136/bmj.330.7500.1139 (Published 12 May 2005) Cite this as: BMJ 2005;330:1139All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Thompson and Stonebridge, or perhaps the sub-editors at the BMJ,
headlined this article:
'The British public's confidence in doctors and hospitals has been
dented in recent years. Use of an independent review of deaths before,
during, or after surgery reflects an attempt to improve care in this area
and may also help to restore the public's trust in their health service.'
While not denying the current attempts to develop robust ways of
measuring medical outcomes, imagining they will improve the 'public's
trust' is misguided. The public trust the health service; it is the
politicians and media who misuse this perceived distrust for their own
purposes: the politicians are jealous of the trust placed in doctors; the
media need to sell their product, which makes bad news good news.
We don't need surrogate reasons. We need to develop ways of measuring
medical outcome because otherwise we don't know whether what we are doing
is worth doing, or is being done properly. However, in much of medicine,
the measurement may be too difficult. You can be sure that if we sort out
the easier outomes, such as deaths after surgery, the politicians and
media will taunt us with the more difficult ones.
Competing interests:
None declared
Competing interests: No competing interests
Two days before the BMJ published Baxter's editorial
expressing doubts as to whether the Scottish Audit of
Surgical Mortality (SASM) was applicable elsewhere the
Western Australian Audit of Surgical Mortality (WAASM)
published its second annual report
<http://www.surgeons.org/cgi-bin/redirector.cgi>.
WAASM, which is based on SASM, commenced in
2001. It has has already shown clear changes in local
practice. This despite a very different surgical
environment. For example, unlike the NHS a majority of
surgical operations in Australia are undertaken in the
private sector.
Current participation (96% of surgeons submitted 60%
of all deaths) is not as complete as Scotland. This
needs to be considered against the very different
background of a county that, unlike Scotland, did not
have an established culture of regional surgical audit.
A particular problem at the outset was the very
aggressive medical legal environment in Australia.
Qualified privilege was an essential prerequisite.
The principal area of public and media interest was the
level of surgeons participation. The clear expectation is
that surgeons participate in the audit process. Little
attention was directed to the adverse events
themselves,
The Royal Australasian College of Surgeons has
announced its intention of establishing the Australian
and New Zealand Audit of Surgical Mortality. This will
be based on WAASM methodology, which although now
modified for local practice, retains the concepts of
SASM at it core.
RJ Aitken
Chairman,
WA Audit of Surgical Mortality
Competing interests:
None declared
Competing interests: No competing interests
SASM unsuitable for revalidation
The Scottish Audit of Surgical Mortality has many serious flaws that
make it quite unsuitable for any use in revalidation (1). Correctly
labelled in the accompanying commentary as a survey of critical incidents
(2), large national projects like SASM and the National Confidential
Enquiry into Post-operative Deaths (NCEPOD) have had considerable value in
determining institutional factors associated with avoidable deaths, like
deficiencies in Recovery, intensive care and high dependency facilities.
It is quite another matter to extend these projects into areas for which
they are not designed or suited.
Even as a peer reviewed series of surgical deaths, SASM has serious
flaws. The review process is inadequate. Each operative case is reviewed
by a surgeon and an anaesthetist, which means there is essentially only
one reviewer for each specialty. The reviewers are encouraged but not
obliged to support their conclusions with published evidence (3). Without
supporting evidence, the reviewer’s conclusions are no more than personal
opinion. A random 10% of reviews are subject to a second independent
assessment as a quality assurance audit, and there is a high level of
agreement of these two reviews. As has been pointed out before in this
context (4), good inter-rater agreement is no guarantee that these
judgements are correct. Lagasse provides a good discussion of how to set
up a well structured and reliable peer review process (5).
The forms completed by the anaesthetist seek the judgements and
opinions of the responsible anaesthetist, and the other information
collected would not enable the reviewer to make an independent judgement
if there were any anaesthetic deficiencies. To rely on the insight of the
responsible clinician to identify deficiencies in the care means that
there must be an unknown number of deaths that have areas of concern and
are not detected by SASM. The forms should capture data only, not the
opinions of those who performed the procedure. It is the responsibility of
the reviewers to determine the areas of concern, not the clinicians to
point them out.
Normal doctors all make errors and misjudgements, and in medicine, we
have not begun to accept the implications of this, If tomorrow, during a
difficult case, I am distracted and connect the epidural infusion of local
anaesthetics to the central venous catheter instead of the epidural
catheter, the patient may well die. Critical comment from SASM will be
the least of my problems as I will probably be suspended, referred to the
GMC and may well be charged with manslaughter. These slip-lapse errors are
routine, and although it may be the only error I make in my entire career,
it will be treated essentially as a criminal act at all levels within
medicine and the law. We persist with the paradigm that good doctors do
not make mistakes and only bad doctors make errors, and projects like SASM
essentially perpetuate this concept. Because we do not know ‘normal’
error and misjudgement rates, using SASM process for revalidation is quite
wrong. Measuring the quality of an individual’s practice is difficult, and
only the cardiac surgeons have developed any sort of reliable methods.
Robust statistical measures of process control like cusum analysis (6) or
Mahalanobis distance (7) are needed. These are not well understood but the
only valid way to measure an individuals performance, and separate the
good practitioner who has had a bad run from a poor practitioner who has
had a good run. One of the lessons from the statistical enquiry at the
Bristol Inquiry was that there were episodes of good performance within a
background of overall poor performance. For this reason alone, a ‘good’
report from SASM can never be taken as evidence of good clinical
performance, and a ‘bad’ report cannot be taken as evidence of poor
performance of an individual.
The limitations of the SASM type approach can be seen in comparison
with the recent Dutch study using multivariate analyses of the factors
influencing anaesthetic morbidity and mortality (8). This study indicates
appropriate data that could be used to estimate the quality of an
individuals' practice, and SASM should be developing this type of
monitoring rather than trying to justify using existing methods just
because they exist.
1 Thompson A M, Stonebridge P A. Building a framework for trust:
critical event analysis of deaths in surgical care. BMJ 2005;330:1139-
1142.
2 Lakhani M. Commentary: Excellent review scheme for critical
incidents but insufficient for revalidation. BMJ 2005:330:1143.
3 Scottish Audit of Surgical Mortality. Guidelines for assessors.
http://www.sasm.org.uk/Process/Assessors%20guidelines.htm (accessed 24th
May 2005).
4 Keats AS : Mortality in perspective. Anesth Analg 1990 ;71: 113–9.
5 Lagasse RS - Anesthesia safety: model or myth? A review of the
published literature and analysis of current original data Anesthesiology
2002;97:1609-17.
6 Poloniecki, J, Sismanidis C, Bland M, Jones P. Retrospective
cohort study of false alarm rates associated with a series of heart
operations: the case for hospital mortality monitoring groups BMJ
2004;328:361-2.
7 Rousseeuw PJ, Leroy AM. Robust regression and outlier detection.
New York: Wiley, 1987.
8 Arbous MS, Meursing AEE, van Kleef JW, de Lange JJ , Spoormans
HHAJM, Touw P et al. Impact of anesthesia management characteristics on
severe morbidity and mortality. Anesthesiology 2005;102:257-268.
Competing interests:
None declared
Competing interests: No competing interests