Systematic review of clinical effectiveness of pressurised metered dose inhalers versus other hand held inhaler devices for delivering corticosteroids in asthma
BMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7318.896 (Published 20 October 2001) Cite this as: BMJ 2001;323:896All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
On the 20th of October the BMJ published two striking examples of
meta-madness(1; 2). The systematic review industry continues to pick over
the corpses of long dead studies hoping to resuscitate silk purses despite
good evidence that the whole flawed process is a little better than the
toss of a coin when it comes to deciding treatment options(3). These
recent studies however plumb new depths in data distortion. The combined
studies look at over 100 randomised controlled comparisons of metered dose
inhalers and a variety of dry powder devices. The authors conclude that
metered dose inhalers are the most cost effective treatment and should be
continued as the first line delivery devices. This is an astonishing
conclusion considering none of the studies they quote used cost
effectiveness as an end point.
To design an experiment which demonstrates the dose relationship,
never mind the cost effectiveness, of two different types of inhaler in
asthma is extraordinarily difficult. The main problem is that
conventional doses of both beta-agonists and inhaled steroids are near the
maximum of the dose response curve. Thus, in their classic paper
demonstrating the two to one efficacy of the turbohaler over the metered
dose inhaler Agertoft and Pederson(4) found that half the children failed
to exacerbate on reduction of their budesonide dose i.e. they were being
over treated. In those patients who did exacerbate turbohaler at half the
dose was more effective in clinical measures than the full conventional
dose delivered by an MDI. Even this carefully designed study did not
truly examine the cost effectiveness of the two different treatment
strategies but merely demonstrates that this particularly inhaler was
clinically more effective at half the dose. Any such studies which may
have hinted at relative efficacy are drowned out by the large volume of
studies without such careful entry criteria.
Why have the majority of these comparative studies been performed?
They are undertaken to demonstrate the equivalence of an MDI and a dry
powder inhaler. This is because the regulatory authorities insist on
novel inhalers being 'equivalent' to the standard preparation, almost
invariably the MDI. Dry powder inhalers are deliberately 'dumbed down' to
provide equivalence since much less clinical information is required for
such an application. Devices which claim higher efficiency such as Qvar
or the turbohaler are made to undergo many more costly studies. These
bizarre regulations, which are almost entirely irrelevant clinically
because asthma medication should be adjusted according to the individual
patient response, have been previously criticised by myself and others(5).
There are a number of manoeuvres that can be used to minimise the risk of
demonstrating a difference in such licensing application studies. If high
doses are used maximum clinical effect will be achieved whatever the
device. It is interesting to note that the authors comment that ten of
the twenty studies in adult asthma using inhaled corticosteroids were at
doses so high they not reflect clinical practice. Power is obviously kept
to a minimum in order to maximise the chance of a type II error. Such
studies do not add to our knowledge of the relative clinical effectiveness
of inhalers. However, their inclusion in a meta-analysis significantly
degrades the power of that analysis to detect any true effect.
One of the great joys of performing a prospective clinical trial as
opposed to a retrospective analysis is that you do not know the answer
before you start. In this regard it is perfectly reasonable to pick
relative change from baseline or absolute change from baseline as your
measure of efficacy in a particular study. Inconveniently for the authors
three studies demonstrated greater efficacy with dry powder inhalers.
These studies are subjected to post hoc analysis resulting in them
becoming insignificant. This is data dredging of the worst sort; picking
studies which do not fit your hypothesis and manipulating them until they
do. At the very least any post hoc analysis should have been applied
evenly over the whole data set.
In all studies an entry criteria will have been that the patients
were able to use the inhaler devices correctly. Problems with MDI
training are well documented, indeed, legions of nurses are employed
specifically for this purpose (how cost effective is that?). Thus the
studies will have excluded the many patients for whom an MDI is
inappropriate and yet the authors generalise their speculative conclusion
to the whole asthmatic population.
Finally, I was astonished to read that the authors felt that they had
no competing interests. The Department of Health recommends metered dose
inhaler usage on the erroneous and unsupported belief that it is more cost
effective. The NHS Health and Research Development Health Technology
Assessment Programme receives considerable funding from the Department of
Health and pays the salary of some of the authors. It is not just big
pharma which butters bread.
Yours sincerely
PROFESSOR ALYN H MORICE
Head of Academic Department of Medicine
Competing interests
AHM has been reimbursed for attending symposia, received fees for
speaking, consulting as well as receiving research funding from Astra
Zeneca, Boehringer Ingelheim, Boots, GlaxoSmithKline, Merck Sharp Dohme,
ML Pharmaceuticals, Novartis, Proctor & Gamble, Schering Plough, 3M.
Reference List
1. Brocklebank D, Wright J, Cates C. Systematic review of clinical
effectiveness of pressurised metered dose inhalers versus other hand held
inhaler devices for delivering corticosteroids in asthma. Br Med J
2001;323:896
2. Ram FSF, Wright J, Brocklebank D, White JES. Systematic review of
clinical effectiveness of pressurised metered dose inhalers versus other
hand held inhaler devices for delivering beta 2 agonists bronchodilators
in asthma. Br Med J 2001;323:901
3. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F.
Discrepancies between meta-analyses and subsequent large randomized,
controlled trials. N.Engl.J.Med. 1997;337:536-542.
4. Agertoft L, Pedersen S. Importance of the inhalation device on
the effect of Budesonide. Arch Dis Child 1993;69:130-133.
5. Morice AH. CFC transition. Thorax 1901;56:501-502.
Competing interests: No competing interests
Dear Sir
The results in the two papers from the National Health Technology
Assessment Inhaler Review Group may seem counter-intuitive to clinicians
in practice.1,2 For many patients simple metered-dose inhalers (MDIs) are
not as effective as breath activated inhalers and powder inhalers because
they are too difficult to use. Why do these systematic reviews come to a
different conclusion? The answer may be that the types of patient for
whom the clinicians in every day practice choose to prescribe alternative
devices were excluded from the clinical trials which these systematic
reviews examined. The entry criteria for randomised controlled trials in
asthma are likely to exclude patients who have difficulty learning, or
patients who have difficulty in keeping diaries (this may be an aspect of
learning difficulty), or particularly patients who have poor reversibility
with a simple MDI. Patients with learning difficulties may find learning
to use an MDI impossible. Patients who have poor reversibility with a
simple MDI may have excellent reversibility with an alternative device if
they can not use a simple MDI effectively. As many as 30% of patients in a
general practice setting may have poor technique with an MDI.3 The
generalisability of the results of randomised controlled trials begins
with the entry criteria for patients in those trials.
It is interesting that in the same issue of the British Medical
Journal the generalisability of randomised controlled trials is discussed
in relation to recombinant human activated protein C in sepsis in
intensive care.3 In that article the importance of knowing the details of
the subjects in a trial and their comparability with subjects in a
clinical setting is highlighted. So also is it essential to know the
details of the subjects in these papers on the clinical effectiveness of
asthma inhalers before a judgement can be made. No details of the
subjects in the original trials are given and the possible relevance of
the trial subjects is not considered in the discussion. I am disappointed
to conclude from these two papers that they can add little to my clinical
practice. Systematic reviews can only offer limited insights if they fail
to report the characteristics of the subjects in the trials they review.
Yours sincerely,
Patrick White
Senior Lecturer
1. Brocklebank D, Wright J, Gates C on behalf of the National Health
Technology Assessment Inhaler Review Group. Systematic review of clinical
effectiveness of pressurised metered dose inhalers versus other hand held
inhaler devices for delivering corticosteroids in asthma. BMJ
2001;323:896-900.
2. Ram FSF, Wright J, Brocklebank D, White JES on behalf of the National
Health Technology Assessment Inhaler Review Group. Systematic review of
clinical effectiveness of pressurised metered dose inhalers versus other
hand held inhaler devices for delivering 2 agonist
bronchodilators in asthma. BMJ 2001;323:901-5.
3. Hilton S. An audit of inhaler technique among asthma patients of 34
general practitioners. Br J Gen Pract 1990;40:505-506.
4. Padkin A, Rowan K, Black N. Using high quality clinical databases to
complement the results of randomised controlled trials: the case of
recombinant human activated protein C. BMJ 2001;323:923-6.
Competing interests: No competing interests
In the systematic review by Brocklebank et al(1) , the meta-analysis
of dose ratios of HFA and CFC BDP present a number of questions for
healthcare professionals. The only HFA beclomethasone available in the UK
is Qvar from 3M, licensed as equivalent in efficacy to CFC
BDP at half the dose. If and when another HFA BDP is approved, the CMO’s
guidance(2) concerning brand prescribing will ensure that there is no
confusion during transition away from CFC products.
We question the applicability of meta-analysis on HFA vs CFC BDP
studies to meet the review’s objective in stable asthma, when only 2 of
the 6 publications examined included stable patients. Busse studied
patients with moderate to severe symptomatic asthma, who had undergone a
steroid washout period prior to receiving treatment. Patients in the
studies by Davies and Gross were symptomatic, had an oral steroid burst,
and were then studied to examine maintenance of asthma control on inhaled
steroids. Patients in studies by Demedts and Dahl (a cross over study)
were all well controlled at study entry. Dosing frequencies and
statistical methods also varied.
The HFA vs CFC review also contains significant errors. The largest
study reviewed was by Demedts (not Damedts). The Busse study was designed
to, and demonstrated, the comparative dose responses FEV1 of CFC and HFA
BDP across 100, 400 and 800 mcg. Brocklebank meta-analysed this study as
four separate studies, using the secondary, patient diary recorded
variable of peak expired flow. In addition to this erroneous
interpretation, patient numbers and change from baseline in Figure 3 have
been transposed, biasing the standardised mean difference and the subtotal
plot. The review and supporting HTA also ignored the considerable
correspondence(3) concerning Milanowski.
Finally, the focus on efficacy, ignoring the comparative safety of
HFA and CFC BDP is a significant omission.
I trust that the work put into this HTA will not be devalued by
errors such as those above and that a corrected assessment is forthcoming.
Accurate reviews are crucial for evidence based clinical decision making.
Dr Richard Spiers
Medical Director
3M Health Care
References:
1. Brocklebank D, Wright J, Cates C. Systematic review of clinical
effectiveness of pressurised metered dose inhalers versus other hand held
inhaler devices for delivering corticosteroids in asthma. Br. Med. J.
2001;323:896-902.
2. CMO’s Update 17 February 1998. Propellant changes in metered dose
inhalers
3. Letters to the Editor, Respiratory Medicine 2000; 94 (2)
Competing interests: No competing interests
The recent meta-analysis by Brocklebank et al1 has many virtues, not
least the energy and thoroughness with which it has been prosecuted. Also,
in their more extensive Health Technology Assessment report, available on
the web, much further helpful information is recorded by the authors in a
detail that is rare outside the drug regulatory context. Nevertheless,
there are inappropriate features in the analysis which, in my view,
reflect inherent dangers in the Cochrane Collaboration approach to meta-
analysis. Since it has been claimed by a paper2 in this journal that in
asthma Cochrane reviews are superior to others, it is worth looking at the
problems with the analysis by Brockelbank et al.
As is common in Cochrane reviews, the authors have relied on the
software RevMan. Rather than proceeding from using treatment contrasts and
standard errors as the raw input for a meta-analysis this requires means,
standard deviations and numbers for each treatment group. This approach is
only valid for single-centre parallel-group trials and is completely
inappropriate for cross-over trials, for example3. Yet the authors have
forced cross-over trials into this straight-jacket claiming that it is
necessary to do so because within-patient errors are not available. Where,
however, treatment differences and standard errors of differences are
already recorded this is not true provided one is not committed to using
RevMan. The requirement of RevMan to have everything presented as a
parallel group trial has led the authors further astray. For example, to
enable the results of the Vidgren et al trial4, which had three arms, to
be "understood" by RevMan, they have entered this twice as two separate
trials but using the same control data. This means that in their overall
summary of figure 2 the control data are counted twice, a very serious
methodological error. (Two typos in this figure need correcting: the upper
confidence limits for the first two trials have minus signs missing.) A
further aspect of their approach betrays a misunderstanding of measurement
in clinical trial. Contrary to what they claim, provided that the same
measure of lung-function is used "absolute and improvement from baseline"
measure exactly the same thing, as may be simply seen when we consider
that in the perfectly balanced trial they give the same answer5. There is
thus no need to use the method of standardise mean differences to combine
them.
Finally , out of 28 standard deviations quoted in Figure 2, 10 have a
value of 100.0. Can these be real? Surely they must be imputed. It seems
to me that more space could have been spent discussing this rather
important point and less discussing the use of Cochrane' preferred quality
assessment index, which places virtually no premium on proper analysis3.
1. Brocklebank D, Wright J, Cates C. Systematic review of clinical
effectiveness of pressurised metered dose inhalers versus other hand held
inhaler devices for delivering corticosteroids in asthma. Br. Med. J.
2001;323:896-902.
2. Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, et
al. Systematic reviews and meta-analyses on treatment of asthma: critical
evaluation. Br. Med. J. 2000;320(7234):537-40.
3. Senn SJ. Review is biased. Br. Med. J. 2000;321:p297.
4. Vidgren P, Silvasti M, Poukkula A, Laasonen K, Vidgren M.
Easyhaler Powder Inhaler - a New Alternative in the Antiinflammatory
Treatment of Asthma. Acta Ther. 1994;20(3-4):117-131.
5. Senn SJ. Baseline comparisons in randomized clinical trials. Stat.
Med. 1991;10(7):1157-9.
Declaration of interest. The author is a consultant to the
pharmaceutical industry
Competing interests: No competing interests
Both of the articles in this weeks copy of the BMJ have the same
inherent flaw, which has also been pointed out by Prof. Lipworth in his
response. It has long been true that the most expensive inhaler is the one
that sits in the patient's bedside table and is not used because of
inability to use it properly, or perceived lack of response. Many
publications have backed up the view that patients cannot use properly and
efficiently a pMDI, and that even after correct instruction in technique
their "ability" declines rapidly with time.
There is a huge danger with meta-analyses such as these, that they
will be quoted with glee by PCO prescribing advisers as a push to move
prescribing down the price range back to pMDI, but the evidence is that
asthma control will deteriorate as a result. This is acknowledged in the
conclusion of both articles "....the cheapest inhaler device that
patients can use adequately should be used as first line treatment", but
it is sad that this essential fact does not appear either in the abstract
or in the "What this study adds" (the sections most likely to be read in
Primary Care)in fact the latter has the very dangerous conclusion that
"Pressurised metered dose inhalers (or the cheapest device) should be
first line treatment in all patients with stable asthma.
Competing Interests: Dr Williams has chaired meetings and been a
guest at respiratory care meetings funded by Glaxo Wellcome, and Astra
Zeneca.
Competing interests: No competing interests
The systematic review of Brocklebank et al is a good example of how
meta-analysis can be misleading in terms of what we already know from
everyday practice.The most common reason in my clinic for failure to
respond to inhaled corticosteroid is inabilty to use a pMDI .Moreover even
after educational input on correct use,most patients are still unable to
correctly use a pMDI.The pMDI was cutting edge technology in the 1960's
,but nowadays there are several other portable breath actuated pressurised
and dry powder inhalers,which are much easier to use,require less
educational input and provide better lung depostion as well as having an
actuation counter.All of these facotrs are likely to result in better long
term compliance .In this respect the data from clinical trials using
pMDI's will not reflect real life because patients were usually selected
on the basis of being able to use a pMDI properly as an inclusion criteria
-ie the results would always represent the best possible scenario for the
clinical efficacy of pMDI as compared to other devices.In everyday
practice one has to take into consideration the indirect costs of more
intensive patient education and treatment failures when using pMDI's as
compared to using the direct costs of using more expensive breath actuated
devices ,especially in the long term.Pointedly none of the included
studies evaluated long term effects over several years .Most of my
specialist colleages in pulmonology gave up using pMDI's several years ago
and I very much doubt whether this meta-analysis will have any impact on
prescribing in secondary care.Whether or not primary care prescibers are
more gullible to this type of analysis is open to debate.Sometimes one has
to stand back from so called evidence based data and use a bit of common
sense .Technology has moved on over the past four decades and this is
something we should embrace rather than going back to the bad old days
where patients could not their inhalers.
BJL has recieved grant,eduacational and equipment support from
companies who make pMDIs and breath actuated inhalers,including
AstraZeneca,GlaxoSmithkline,Schering Plough,Boehringer
Ingelheim,Novartis,Innovata Biomed,Aerogen,Aventis ,Orion,Yamanouchi.The
spouse and mother of BJL have shares in GlaxoSmithkline.
Competing interests: No competing interests
Oh Dear
Editor
‘Oh dear….’
The 'systematic' review of inhalers for delivering inhaled
corticosteroids (ICS) by Brocklebank et al1 is a worthy, relatively
comprehensive, ultimately disappointing and indeed misleading example of
secondary research. Leaving aside the validity of their conclusions for
the moment the concept of undertaking systematic reviews of
‘effectiveness’ of therapy with no consideration of safety is at best
naïve and will certainly lead to breaches of our commitment to 'do no
harm'. One of the principle reasons for developing holding chambers in
the early 1980's was to improve the 'therapeutic index' of beclomethasone
in response to the overt side effects such as thinning of the skin and
purpura seen in the elderly. Conventional beclomethasone remains the most
widely prescribed ICS in the UK and it is clear that the ‘therapeutic
index’ changes considerably if a spacer is not used. A systematic review
of the effectiveness of therapy for morning sickness would almost
certainly indicate that thalidomide would be a valid cost effective
option. As is so often is the case if you do not ask the right question
you do not get the right answer.
Perhaps more concerning is the authors and, to an even greater
extent, the journals lack of insight into the 'generalisability' of the
results. If secondary research is to be of value and influence clinical
practice it must consider the characteristics of the studies that may
influence the 'generalisability' of the results. There are two important
factors which are not considered pertaining to inhaled therapy. The
journal stresses that ' pMDIs should be the first line in all patients
with stable asthma'. However the vast majority of studies utilise a pMDI
and spacer as it is widely accepted that many patients cannot use a pMDI
effectively. pMDIs are simple to used but difficult to use effectively.
More importantly, there is no consideration of the inclusion criteria
used for each study. Though I have not had the opportunity to review all
the papers I suspect the majority, if not all, of the studies have as an
inclusion criteria the ability to use the devices under consideration. In
many cases it may be necessary to approach the original investigators for
this information. It is entirely predictable that we will see only small
differences if we review studies which only include subjects who can
effectively use both devices at the commencement of the study and who are
reviewed at frequent intervals over the short period of the study. This
does not represent practice in the UK where lack of competence with
devices, particularly in the elderly, and contriving to use a device
ineffectively (contrivance), as when patients choose not to use the
spacer, are both very common2. Moreover few patients receive regular
intensive training. The authors do note however the very high presumably
supra-maximal levels of ICS used in so many of the studies which will mask
any significant differences between devices in terms of effectiveness as
the dose used in clinical practice should, almost certainly, be ‘the
lowest dose that works’.
As a consequence the findings of this piece of secondary research are
of little relevance to clinical practice. The journal’s bullet points
represent another example of taking one piece out of a complex jigsaw and
over interpreting the results. Though there are proponents of DPIs these
are again not without their problems. At present we do not have the
evidence on which to make blanket recommendations as the appropriate
trials have not been undertaken. The choice of inhaler for ICS is very
important – if a patient chooses to comply with therapy it is our duty to
provide them with a device that will reliably and effectively deliver the
ICS to the lungs. More over the ‘therapeutic index’ of the device/drug
combination should be acceptable. At present there is no convincing
evidence that any one ICS is safer than another, if used with an
appropriate delivery system and is used at the ‘lowest dose that works’.
The choice of delivery system for agonists is a lifestyle
choice as these are administered at supra-maximal doses and patients can
titrate the dose against response both of which will compensate for poor
technique
The reality is that none of the current devices are ideal and that
there is still a need to develop devices that are intuitive to use so that
all patients can and will use them effectively. In the meantime choice of
delivery system for ICS is dependent upon selecting an appropriate device
that a patient can and will use effectively.
1. Brocklebank D, Wright J, Cates C. Systematic review of clinical
effectiveness of pressurissed metered dose inhalers versus other hand held
inhaler devices for delivery of corticosteroids in asthma. BMJ 2001; 323:
896-900
2 Everard ML. CFC transition: the Emperor’s new clothes. Thorax 2000;
55: 811-14
Competing interests: No competing interests