Rapid responses are electronic comments to the editor. They enable our users
to debate issues raised in articles published on bmj.com. A rapid response
is first posted online. If you need the URL (web address) of an individual
response, simply click on the response headline and copy the URL from the
browser window. A proportion of responses will, after editing, be published
online and in the print journal as letters, which are indexed in PubMed.
Rapid responses are not indexed in PubMed and they are not journal articles.
The BMJ reserves the right to remove responses which are being
wilfully misrepresented as published articles or when it is brought to our
attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not
including references and author details. We will no longer post responses
that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Sensky and Scott1 suggest that my questioning of the CBT literature
uses arguments which are instances of “idiosyncratic use of the research
evidence” and calls for “clinicians and commentators to understand and
respect the critical appraisal of the evidence base”. This turns my
argument on its head as I was suggesting that it is CBT researchers who
show a lack of respect for their chosen method, including presenting
idiosyncratic versions of randomised control trials (RCTs) and being
creative with their use of statistical analysis.
Sensky and Scott remind me of the anacronym2 effect - they seem to
think that if they give the impression that there is unequivocal evidence
for the effectiveness of CBT this will make it so. However, like
Garfield3 I find that “just a cursory glance” at the literature leads to
questioning of the claims made for validated therapies/evidence based
psychotherapy. There was no need for me to be selective about the papers
I cited as claiming more than their data suggest. Papers I have quoted
previously and will go on to quote below either immediately come to hand
or are those suggested by reviews of CBT and rated as good evidence for
CBT. For example, Leff et al4 is worth returning to as it illustrates
many of the problems in psychotherapy RCTs as well as being a study in
which CBT, although marketed as user friendly, was unable to engage
participants in its arm of the study. This trial is described as an RCT
and reached the conclusion that “couple therapy is much more acceptable
than anti depressant drugs and is at least as efficacious, if not more so,
both in treatment and maintenance phases”. In this study there was not a
no treatment control group; two outcome measures were used but the p
values were not adjusted; only one of the two outcome measures rejected
the null hypothesis, at the five percent level; and, furthermore, the
outcome measure that gave the significant effect did not have a
significant group x occasion interaction. Therefore, this study is not
strictly speaking a randomised control trial (it does not have a no
treatment condition) and it does not strictly conform to the principles of
statistical analysis. Two outcome measures are used and the p value
should be halved as a consequence. This would mean that the main effect
could not be reported as statistically significant.
Furthermore, as there
is not a significant occasion x treatment condition interaction it is not
possible to state that the observed difference between treatment groups is
a consequence of the treatment interventions. Since I started looking
more closely at the evidence based literature, rather than accepting
authors’ assertions, I have been surprised to see how frequently those who
advocate validation of treatments via RCTs stretch the definition of RCT
and are creative in their use of statistics.
I have looked, as Sensky and Scott challenge me to, at the abstracts
of the Cochrane Reviews5 of cognitive therapy. I noted the Cochrane
disclaimer – the reviews are open to different interpretations. While
Sensky and Scott obviously feel the evidential pot is half full, I see it
as half empty. I particularly noted the Cochrane review of Cognitive
Therapy in Schizophrenia6 and did not find the ringing endorsement their
promotion of CBT led me to expect – “Reviewers' conclusions: Cognitive
behavioural therapy is a promising but under evaluated intervention.
Currently, trial-based data supporting the wide use of cognitive
behavioural therapy for people with schizophrenia or other psychotic
illnesses are far from conclusive. More trials are justified, especially
in comparison with a lower grade supportive approach. These trials should
be designed to be both clinically meaningful and widely applicable.”
These conclusions remind me of earlier examples of the literature, in
particular the British Psychological Society’s review of psychological
approaches to psychotic experiences7. This included an assertion of the
effectiveness of CBT, although examination of the papers cited revealed
less papers than expected (each time a reference was mentioned it was
given a new number – creating the impression that more papers were being
cited than was actually the case) and much weaker evidence than readers
were led to anticipate. That is, in section “12.3.4 The effectiveness of
CBT” 12 references were cited, “Several published reports of randomised
controlled trials (the ‘gold standard ‘ of clinical research) are
available” - but only two were cited. The 12 references were of two
RCTs, three treatment manuals, a review, a drug study and a controlled
trial. Three papers were quoted twice and two of those (ie four of the 12
references) were, in fact, different parts of the same study. The
“RCT”, which presented the more robust data, used multiple outcome
measures and did not include a no treatment control group. The other RCT
used numerous outcome measures, only one of which distinguished between
groups, did not include a no treatment control and did not assess outcomes
blind. That is, although readers were given to understand that the
effectiveness of CBT is based on ‘several….randomised control trials’
actually 2 RCTs were referenced and neither were, strictly speaking, a
RCT.
That review7 stated, as Sensky and Scott seem to agree, “There is
convincing evidence that psychological interventions (ie CBT), are
effective for many people in reducing psychotic experiences and the
distress and disability they cause”. I think this overestimates the
evidence (as indicated by a Cochrane Review6), particularly when the
evidence for other forms of psychotherapy is not even considered to
deserve a mention. This claim is based on ‘gold standard’ studies which
are not RCTs and the outcomes of which are not as clear cut as is
suggested. For example, the first ‘RCT’ I refer to above, a study
reported by Sensky and colleagues8, showed no difference between
cognitive therapy (CT) and the befriending control at the end of the
treatment phase, only at follow up did a difference emerge. The latter
study also reports multiple outcome measures for which the p values have
not been adjusted. Had the p values been appropriately adjusted it is
unlikely that the authors could have reported a positive outcome at follow
-up. Furthermore, this study reports the percentage of patients “who
showed 50% or greater reduction in outcome scores at follow-up
examination” as if supporting the use of CT, despite the analyses finding
a statistically significant advantage for CT on only one of four measures
(had the p values been adjusted for multiple outcomes none of the measures
would have reached significance at the 5% level).
Many CBT reports are published claiming positive outcomes in studies
using multiple outcome measures with p values which have not been
adjusted, and in which equivocal findings (some measures suggesting
rejection of the null hypothesis and others not ) are described as
supporting the favoured intervention. It seems to me that those who
insist that psychotherapy research can only proceed using the RCT approach
are not prepared to accept its judgement.
The recent Department of Health review of psychological therapies9
included an allegiance examination of its own work and found CBT
practitioners the most likely to favour their own approach and the least
likely to consider the possible utility of any other approach. A number
of CBT studies appear to provide support for common factors in
psychotherapeutic effectiveness10, but this possibility is ignored by CBT
researchers8. Perhaps in future studies Sensky and colleagues will have
followed the Cochrane Review suggestion of taking more seriously the
possible benefits of a “lower grade supportive approach”.
CBT’s endorsement of the acute illness/drug research model, which
many non-CBT psychotherapists consider inappropriate, gives it a special
status in psychotherapy research. Evidence from non RCT investigations,
particularly the case reporting approach traditional in psychotherapy, has
been dismissed as not scientific and, therefore, irrelevant. CBT is the
dominant approach as it reports far more ‘RCTs’ than any other form of
psychotherapy, even if the status of many these RCTs is questionable.
Perhaps because of allegiance effects, as well as the quantity of CBT
studies, CBT also does better than other approaches when others are
available for comparison in Cochrane Reviews. For example, CBT is
supported by a Cochrane Review for use with Chronic Fatigue Sycndrome11 on
the basis of three studies; but when Bulimia Nervosa and Bingeing12 is
considered three studies supporting non CBT-psychotherapies are dismissed
as insufficient and CBT is once more the favoured therapy.
Sensky and Scott1 ask that the challenge is met to understand and
respect the critical appraisal of the evidence base. Understanding
Cochrane reviews and many of the CBT ‘RCTs’, for example the trial of
Sensky and his colleagues8 that I have already mentioned, certainly is a
challenge. A criticism of the CBT literature13 is that data are not
presented in a clear and straightforward manner. Intricate
transformations of the data, partial reporting of the results and complex
statistical methods are used which the average clinician is unlikely to be
able to follow. If the benefits of CBT are robust and ubiquitous why can
this not be demonstrated in a way that is obvious to the reader? The
study14 to which Sensky and Scott refer, as if it is definitive evidence,
is open to the criticisms above and, if it shows anything clearly, it is
not that cognitive therapy is effective in preventing relapse in residual
depression rather that an additional psychological intervention improves
on medication alone.
It might be hoped that the CBT research literature will stimulate
those working with different psychotherapeutic approaches to undertake
similar studies, including co-operating in multi-arm trials. A major
problem is that the CBT literature can also be interpreted as
demonstrating the impossibility of conducting RCTs with psychotherapy.
I agree that rigorous research methods should be applied to all forms of
psychotherapy - but hoping for progress with acceptable evidence confined
to that from RCTs is like asking Eriksson to prepare England for the next
World Cup only using five-a-side teams and pitches.
1. Sensky T, Scott J. The evidence base of cognitive behavioural
therapy. BMJ.com 2002; 7 Feb.
2. Goodman B. Acronym acrimony. Scientific American 2001;285(5):16.
3. Garfield S. Some problems associated with “validated” forms of
psychotherapy. Clinical Psychology: Science and Practice 1996; 3: 218-
229.
4. Leff J, Vearnals S, Brewin C, Wolff G, Alexander B, Asen E, et al.
The London depression intervention trial. Br J Psychiatry 2001; 177: 95-
100.
5. Index to Abstracts of Cochrane Reviews. The Cochrane Library Issue
3, 2002. www.update-software.com.
6. Cormac I, Jones C, Campbell C. Cognitive behaviour therapy for
schizophrenia. The Cochrane Library, Issue 2, 2002.
7. Kinderman P, Cooke A. Recent advances in understanding mental
illness and psychotic experiences: a report by the British Psychological
Society Division of Clinical Psychology. Leicester: BPS, 2001.
8. Sensky T. Turkington D. Kingdon D. Scott JL. Scott J. Siddle R.
O'Carroll M. Barnes TR. A randomized controlled trial of cognitive-
behavioral therapy for persistent symptoms in schizophrenia resistant to
medication. Archives of General Psychiatry 2000; 57(2):165-72.
9. Department of Health. Treatment Choice in Psychological Therapies
and Counselling: Evidence Based Clinical Practice Guideline. London: DOH,
2001.
10. Seligman M. The Effectiveness of Psychotherapy: The Consumer
Report Study. American Psychologist 1995; 50: 965-974.
11. Price R, Couper J. Cognitive behaviour therapy for chronic
fatigue syndrome in adults. The Cochrane Library, Issue 2, 2002.
12. Hay J, Bacaltchuk J. Psychotherapy for bulimia nervosa and
binging. The Cochrane Library, Issue 2, 2002.
13. Johnson D. Peer review of "Cognitive therapy and recovery from
acute psychosis". Br J Psychiatry 1996; 169: 608-609.
14. Paykel E, Scott J, Teasdale J, Johnson A, Garland A, Moore R, et
al. Prevention of relapse in residual depression by cognitive therapy: a
controlled trial. Arch Gen Psychiatry 1999; 56: 829-835.
Competing interests:
No competing interests
02 October 2002
Nick Bolsover
Consultant Clinical Psychologist in Psychotherapy Hull and East Riding Community Healthcare NHS Trus
Repeated claims for the benefits of CBT do not strengthen the weak evidence
Sensky and Scott1 suggest that my questioning of the CBT literature
uses arguments which are instances of “idiosyncratic use of the research
evidence” and calls for “clinicians and commentators to understand and
respect the critical appraisal of the evidence base”. This turns my
argument on its head as I was suggesting that it is CBT researchers who
show a lack of respect for their chosen method, including presenting
idiosyncratic versions of randomised control trials (RCTs) and being
creative with their use of statistical analysis.
Sensky and Scott remind me of the anacronym2 effect - they seem to
think that if they give the impression that there is unequivocal evidence
for the effectiveness of CBT this will make it so. However, like
Garfield3 I find that “just a cursory glance” at the literature leads to
questioning of the claims made for validated therapies/evidence based
psychotherapy. There was no need for me to be selective about the papers
I cited as claiming more than their data suggest. Papers I have quoted
previously and will go on to quote below either immediately come to hand
or are those suggested by reviews of CBT and rated as good evidence for
CBT. For example, Leff et al4 is worth returning to as it illustrates
many of the problems in psychotherapy RCTs as well as being a study in
which CBT, although marketed as user friendly, was unable to engage
participants in its arm of the study. This trial is described as an RCT
and reached the conclusion that “couple therapy is much more acceptable
than anti depressant drugs and is at least as efficacious, if not more so,
both in treatment and maintenance phases”. In this study there was not a
no treatment control group; two outcome measures were used but the p
values were not adjusted; only one of the two outcome measures rejected
the null hypothesis, at the five percent level; and, furthermore, the
outcome measure that gave the significant effect did not have a
significant group x occasion interaction. Therefore, this study is not
strictly speaking a randomised control trial (it does not have a no
treatment condition) and it does not strictly conform to the principles of
statistical analysis. Two outcome measures are used and the p value
should be halved as a consequence. This would mean that the main effect
could not be reported as statistically significant.
Furthermore, as there
is not a significant occasion x treatment condition interaction it is not
possible to state that the observed difference between treatment groups is
a consequence of the treatment interventions. Since I started looking
more closely at the evidence based literature, rather than accepting
authors’ assertions, I have been surprised to see how frequently those who
advocate validation of treatments via RCTs stretch the definition of RCT
and are creative in their use of statistics.
I have looked, as Sensky and Scott challenge me to, at the abstracts
of the Cochrane Reviews5 of cognitive therapy. I noted the Cochrane
disclaimer – the reviews are open to different interpretations. While
Sensky and Scott obviously feel the evidential pot is half full, I see it
as half empty. I particularly noted the Cochrane review of Cognitive
Therapy in Schizophrenia6 and did not find the ringing endorsement their
promotion of CBT led me to expect – “Reviewers' conclusions: Cognitive
behavioural therapy is a promising but under evaluated intervention.
Currently, trial-based data supporting the wide use of cognitive
behavioural therapy for people with schizophrenia or other psychotic
illnesses are far from conclusive. More trials are justified, especially
in comparison with a lower grade supportive approach. These trials should
be designed to be both clinically meaningful and widely applicable.”
These conclusions remind me of earlier examples of the literature, in
particular the British Psychological Society’s review of psychological
approaches to psychotic experiences7. This included an assertion of the
effectiveness of CBT, although examination of the papers cited revealed
less papers than expected (each time a reference was mentioned it was
given a new number – creating the impression that more papers were being
cited than was actually the case) and much weaker evidence than readers
were led to anticipate. That is, in section “12.3.4 The effectiveness of
CBT” 12 references were cited, “Several published reports of randomised
controlled trials (the ‘gold standard ‘ of clinical research) are
available” - but only two were cited. The 12 references were of two
RCTs, three treatment manuals, a review, a drug study and a controlled
trial. Three papers were quoted twice and two of those (ie four of the 12
references) were, in fact, different parts of the same study. The
“RCT”, which presented the more robust data, used multiple outcome
measures and did not include a no treatment control group. The other RCT
used numerous outcome measures, only one of which distinguished between
groups, did not include a no treatment control and did not assess outcomes
blind. That is, although readers were given to understand that the
effectiveness of CBT is based on ‘several….randomised control trials’
actually 2 RCTs were referenced and neither were, strictly speaking, a
RCT.
That review7 stated, as Sensky and Scott seem to agree, “There is
convincing evidence that psychological interventions (ie CBT), are
effective for many people in reducing psychotic experiences and the
distress and disability they cause”. I think this overestimates the
evidence (as indicated by a Cochrane Review6), particularly when the
evidence for other forms of psychotherapy is not even considered to
deserve a mention. This claim is based on ‘gold standard’ studies which
are not RCTs and the outcomes of which are not as clear cut as is
suggested. For example, the first ‘RCT’ I refer to above, a study
reported by Sensky and colleagues8, showed no difference between
cognitive therapy (CT) and the befriending control at the end of the
treatment phase, only at follow up did a difference emerge. The latter
study also reports multiple outcome measures for which the p values have
not been adjusted. Had the p values been appropriately adjusted it is
unlikely that the authors could have reported a positive outcome at follow
-up. Furthermore, this study reports the percentage of patients “who
showed 50% or greater reduction in outcome scores at follow-up
examination” as if supporting the use of CT, despite the analyses finding
a statistically significant advantage for CT on only one of four measures
(had the p values been adjusted for multiple outcomes none of the measures
would have reached significance at the 5% level).
Many CBT reports are published claiming positive outcomes in studies
using multiple outcome measures with p values which have not been
adjusted, and in which equivocal findings (some measures suggesting
rejection of the null hypothesis and others not ) are described as
supporting the favoured intervention. It seems to me that those who
insist that psychotherapy research can only proceed using the RCT approach
are not prepared to accept its judgement.
The recent Department of Health review of psychological therapies9
included an allegiance examination of its own work and found CBT
practitioners the most likely to favour their own approach and the least
likely to consider the possible utility of any other approach. A number
of CBT studies appear to provide support for common factors in
psychotherapeutic effectiveness10, but this possibility is ignored by CBT
researchers8. Perhaps in future studies Sensky and colleagues will have
followed the Cochrane Review suggestion of taking more seriously the
possible benefits of a “lower grade supportive approach”.
CBT’s endorsement of the acute illness/drug research model, which
many non-CBT psychotherapists consider inappropriate, gives it a special
status in psychotherapy research. Evidence from non RCT investigations,
particularly the case reporting approach traditional in psychotherapy, has
been dismissed as not scientific and, therefore, irrelevant. CBT is the
dominant approach as it reports far more ‘RCTs’ than any other form of
psychotherapy, even if the status of many these RCTs is questionable.
Perhaps because of allegiance effects, as well as the quantity of CBT
studies, CBT also does better than other approaches when others are
available for comparison in Cochrane Reviews. For example, CBT is
supported by a Cochrane Review for use with Chronic Fatigue Sycndrome11 on
the basis of three studies; but when Bulimia Nervosa and Bingeing12 is
considered three studies supporting non CBT-psychotherapies are dismissed
as insufficient and CBT is once more the favoured therapy.
Sensky and Scott1 ask that the challenge is met to understand and
respect the critical appraisal of the evidence base. Understanding
Cochrane reviews and many of the CBT ‘RCTs’, for example the trial of
Sensky and his colleagues8 that I have already mentioned, certainly is a
challenge. A criticism of the CBT literature13 is that data are not
presented in a clear and straightforward manner. Intricate
transformations of the data, partial reporting of the results and complex
statistical methods are used which the average clinician is unlikely to be
able to follow. If the benefits of CBT are robust and ubiquitous why can
this not be demonstrated in a way that is obvious to the reader? The
study14 to which Sensky and Scott refer, as if it is definitive evidence,
is open to the criticisms above and, if it shows anything clearly, it is
not that cognitive therapy is effective in preventing relapse in residual
depression rather that an additional psychological intervention improves
on medication alone.
It might be hoped that the CBT research literature will stimulate
those working with different psychotherapeutic approaches to undertake
similar studies, including co-operating in multi-arm trials. A major
problem is that the CBT literature can also be interpreted as
demonstrating the impossibility of conducting RCTs with psychotherapy.
I agree that rigorous research methods should be applied to all forms of
psychotherapy - but hoping for progress with acceptable evidence confined
to that from RCTs is like asking Eriksson to prepare England for the next
World Cup only using five-a-side teams and pitches.
1. Sensky T, Scott J. The evidence base of cognitive behavioural
therapy. BMJ.com 2002; 7 Feb.
2. Goodman B. Acronym acrimony. Scientific American 2001;285(5):16.
3. Garfield S. Some problems associated with “validated” forms of
psychotherapy. Clinical Psychology: Science and Practice 1996; 3: 218-
229.
4. Leff J, Vearnals S, Brewin C, Wolff G, Alexander B, Asen E, et al.
The London depression intervention trial. Br J Psychiatry 2001; 177: 95-
100.
5. Index to Abstracts of Cochrane Reviews. The Cochrane Library Issue
3, 2002. www.update-software.com.
6. Cormac I, Jones C, Campbell C. Cognitive behaviour therapy for
schizophrenia. The Cochrane Library, Issue 2, 2002.
7. Kinderman P, Cooke A. Recent advances in understanding mental
illness and psychotic experiences: a report by the British Psychological
Society Division of Clinical Psychology. Leicester: BPS, 2001.
8. Sensky T. Turkington D. Kingdon D. Scott JL. Scott J. Siddle R.
O'Carroll M. Barnes TR. A randomized controlled trial of cognitive-
behavioral therapy for persistent symptoms in schizophrenia resistant to
medication. Archives of General Psychiatry 2000; 57(2):165-72.
9. Department of Health. Treatment Choice in Psychological Therapies
and Counselling: Evidence Based Clinical Practice Guideline. London: DOH,
2001.
10. Seligman M. The Effectiveness of Psychotherapy: The Consumer
Report Study. American Psychologist 1995; 50: 965-974.
11. Price R, Couper J. Cognitive behaviour therapy for chronic
fatigue syndrome in adults. The Cochrane Library, Issue 2, 2002.
12. Hay J, Bacaltchuk J. Psychotherapy for bulimia nervosa and
binging. The Cochrane Library, Issue 2, 2002.
13. Johnson D. Peer review of "Cognitive therapy and recovery from
acute psychosis". Br J Psychiatry 1996; 169: 608-609.
14. Paykel E, Scott J, Teasdale J, Johnson A, Garland A, Moore R, et
al. Prevention of relapse in residual depression by cognitive therapy: a
controlled trial. Arch Gen Psychiatry 1999; 56: 829-835.
Competing interests: No competing interests