BMJ  2003;326:1171-1173 (31 May), doi:10.1136/bmj.326.7400.1171

Paper

Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications

¹ Medical Products Agency, Box 23, S-751 03 Uppsala, Sweden

Abstract

Objectives To investigate the relative impact on publication bias caused by multiple publication, selective publication, and selective reporting in studies sponsored by pharmaceutical companies.

Design 42 placebo controlled studies of five selective serotonin reuptake inhibitors submitted to the Swedish drug regulatory authority as a basis for marketing approval for treating major depression were compared with the studies actually published (between 1983 and 1999).

Results Multiple publication: 21 studies contributed to at least two publications each, and three studies contributed to five publications. Selective publication: studies showing significant effects of drug were published as stand alone publications more often than studies with non-significant results. Selective reporting: many publications ignored the results of intention to treat analyses and reported the more favourable per protocol analyses only.

Conclusions The degree of multiple publication, selective publication, and selective reporting differed between products. Thus, any attempt to recommend a specific selective serotonin reuptake inhibitor from the publicly available data only is likely to be based on biased evidence.

Introduction

Drug treatment should rely on solid evidence, and it is now generally recognised that the standard basis for treatment guidelines is systematic literature reviews or meta-analyses of all randomised controlled trials. However, as meta-analyses are usually limited to publicly available data, several factors can give rise to biased conclusions. These include selection of studies submitted or accepted for publication,^{1 2} inclusion of undetected duplicate publications,^{3 4} and selective reporting (such as failure to report intention to treat results). Several actors (editors, investigators, and sponsors) affect whether and how scientific results reach the public domain. In clinical trials of drugs the role of the sponsor is especially important. The sponsor usually has access to all data on a specific product and has an obvious conflict of interest.⁵

The objective of our study was to investigate the relative impact on bias caused by multiple publication, selective publication, and selective reporting in studies sponsored by the pharmaceutical industry.

Material and methods

Studies submitted to drug regulatory authority
Five selective serotonin reuptake inhibitors were approved in Sweden between 1989 and 1994 for treating major depression. Forty two short term (4-8 weeks), placebo controlled clinical trials with the approved doses were submitted to the Swedish drug regulatory authority and formed the basis for the approvals. When applying for marketing authorisation, the applicants are obliged to submit full reports of all studies performed by the applicants as well as all available information on any study not performed by the applicants. Thus, it is reasonable to assume that the submitted studies have not been subject to selection bias.

Studies published
We identified published versions of the submitted studies through a search in Medline, Embase, and PsycINFO; scrutiny of reference lists; and inquiries to the sponsoring companies.

Comparison of studies
We chose the percentage of patients responding to treatment as the criterion for comparing results from submitted reports with those from published articles. Response was defined as at least a 50% reduction of the initial score on the Hamilton depression rating scale (HDRS), on the Montgomery Åsberg depression rating scale or the clinical global impression of change.

Results

We identified 38 publications presenting data from 38 of the 42 studies submitted to the drug regulatory authority (see bmj.com). The sponsoring companies confirmed the completeness of our search.

Multiple publication
The degree of multiple publication varied from no duplicate publication (drug 3) to extensive multiple publication (drug 1) with three stand alone publications appearing twice and two subsets of studies published as pooled publications three times each (fig 1).

View larger version (43K): [in this window] [in a new window]   Fig 1 Publication pattern for studies of the five selective serotonin reuptake inhibitors approved in Sweden between 1989 and 1994 for treating major depression

 

For drug 1, there were no cross references between the pooled analyses of the same subsets of studies. For each of the subsets, the first author was different in two of the pooled analyses, and the third publication had a single author. Many of the studies had appeared previously as stand alone publications, but reference to these in the pooled publications was given in two cases only, once for each subset. Some of the analyses were presented as a pooled analysis of stand alone centres and some as a multicentre study. For both subsets of studies, the pooled results differed slightly between the publications.

For drug 2, eight studies resulted in three pooled publications based on different combinations of studies. The pooled analyses based on two and eight studies appeared simultaneously (as "a double blind comparison" and "a large multicentre study" respectively) with one author in common but without cross reference. Later, the five study analysis was presented as an intention to treat reanalysis of the per protocol analysis in the eight study publication without revealing that three studies were omitted. Nor was it said that two of the included studies had been published earlier as stand alone publications.

The pooled publication of studies of drug 4 was denoted as a review of multicentre controlled studies without identification of the included studies. Two of the studies later appeared as stand alone publications without acknowledgement of their earlier inclusion in a pooled publication. There was no author name in common in the pooled and stand alone publications.

For drug 5, the pooled analysis was presented as a meta-analysis of the five available placebo controlled studies, clearly identified by the name of the principal investigator. Reference was given to one previous stand alone publication. The other stand alone publication appeared seven years later without reference to the pooled publication.

Selective publication
Of the 42 submitted studies, 21 found the test drug to be significantly more effective than placebo (fig 1). Nineteen of these studies appeared as stand alone publications. Only six of the 21 studies not showing significant results were published as stand alone publications. Of the four studies that never reached the public domain, all showed non-significant results with respect to the primary variable.

Selective reporting
All but one of the study reports submitted to the regulatory agency presented results from two or more alternative analyses (intention to treat and per protocol). Only two of the stand alone publications presented an intention to treat analysis as well as a per protocol analysis. The remaining stand alone publications presented only one analysis, which tended to be the more favourable per protocol analysis. In the 15 stand alone and five pooled publications reporting differences in percentage response, patients who withdrew or who could not be evaluated were usually ignored in the calculations of the response rates. This could result in large overestimates compared with the intention to treat analysis based on the submitted reports, where patients who withdrew or could not be evaluated were considered to be non-responders (fig 2).

View larger version (28K): [in this window] [in a new window]   Fig 2 Difference in estimated size of treatment effect (% response to drug minus response to placebo) from published studies and estimate from intention to treat analysis of submitted studies plotted against total sample size.

 

Comparison of pooled results from submitted and published studies
For each drug, we compared a pooled analysis of all studies submitted to the regulatory agency with a pooled analysis of a correct selection of published studies in which all duplicates were excluded. We also made a pooled analysis of published studies including those duplicates that probably could not be identified as such without access to information about all studies. In this second selection we excluded duplicates with at least one author in common and only minor differences with respect to patient numbers and efficacy results but included any duplicates unidentifiable because of lack of cross reference between pooled publications and stand alone publications.

The pooled analyses of published data generally gave larger differences in response rate (drug minus placebo) than did the estimates from all submitted studies (fig 3). The result of the comparison was conspicuous for drug 2 and drug 4. Since the estimates based on the published studies for these two drugs included data from all the submitted studies, the overestimations are due to selective reporting rather than selective publication. Overall, there were only minor differences in response rates between the correct selections of published studies and the plausible selections. Thus, in this material there is no indication of any major bias due to multiple publication.

View larger version (26K): [in this window] [in a new window]   Fig 3 Differences (95% confidence intervals) in response rate (% response to drug minus response to placebo). Results from pooled analyses of all submitted studies, correct selection of published studies (duplicates excluded), and plausible selection of published studies (including probably undetectable duplicates)

 

Discussion

In a cohort of studies submitted to the Swedish regulatory agency to secure marketing approval for five selective serotonin reuptake inhibitors for the treatment of major depression we have found evidence of duplicate publication, selective publication, and selective reporting. In our material this selective reporting was the major cause for bias in overall estimates based on published data.

Strengths and limitations of study
To our knowledge, access to full reports and study protocols for all studies, published as well as unpublished, is unique to our investigation. This has enabled us to study the impact of different sources of publication bias. It also allowed us to elucidate the sometimes complex pattern of publications. Our investigation is restricted to one class of antidepressant drugs, but there is no reason to believe that drug manufacturers have different policies for reporting and publishing studies of different drugs. Indeed, in a review of an antiemetic drug a similar pattern of duplicate publication has been reported.⁴ Thus, our results are likely to be valid for other classes of drugs with a similar structure of the efficacy documentation—that is, several studies with small to medium sample size.

Conclusions
The outcome of our investigation should not be used to dispute the value of systematic literature reviews and meta-analyses in general. However, for anyone who relies on published data alone to choose a specific drug, our results should be a cause for concern. Without access to all studies (positive as well as negative, published as well as unpublished) and without access to alternative analyses (intention to treat as well as per protocol), any attempt to recommend a specific drug is likely to be based on biased evidence. The probable choice of a specific selective serotonin reuptake inhibitor based on a pooled analysis of publicly available data is not likely to be supported by an analysis considering the total body of evidence.

What is already known on this topic Duplicate publication, selective publication, and selective reporting are likely to introduce bias in systematic literature reviews and meta-analyses Several reports have provided evidence of duplicate publication and selective publication as well as the tendency to publish only studies with significant findings What this study adds Access to full documentation of all studies (published and unpublished) made it possible to investigate the relative impact of the different sources of bias Selective reporting (tendency to publish the more favourable per protocol results only) was a major cause for bias A sponsor in control of all studies does not seem to improve the situation with respect to duplicate publication, selective publication, and selective reporting

---

This is an abridged version; the full version is on bmj.com

Contributors: see bmj.com

Funding: None.

Competing interests: None declared.

References

1. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;263: 1385-9.[Abstract]
2. Dickersin K, Chalmers I. Under-reporting research is scientific misconduct. JAMA 1990;263: 1405-8.[Abstract]
3. Gøtzsche PC. Multiple publication of reports of drug trials. Eur J Clin Pharmacol 1989;36: 429-32.[CrossRef][ISI][Medline]
4. Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997;315: 635-40.[Abstract/Free Full Text]
5. Davidoff F, DeAngelis C, Drazen J, Hoet J, Højgaard L, Horton R, et al. Sponsorship, authorship, and accountability [commentary]. Lancet 2001;358: 854-6.[CrossRef][ISI][Medline]

(Accepted March 6, 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Management of depression in adults

Markku Timonen and Timo Liukkonen
BMJ 2008 336: 435-439. [Extract] [Full Text] [PDF]

Efficacy of antidepressants in adults

Joanna Moncrieff and Irving Kirsch
BMJ 2005 331: 155-157. [Extract] [Full Text] [PDF]

The influence of big pharma

R E Ferner
BMJ 2005 330: 855-856. [Extract] [Full Text] [PDF]

Reporting of clinical trials of drugs shows bias
BMJ 2003 326: 0. [Full Text]

This article has been cited by other articles:

• Noble, J. H. Jr, Sharav, V. H. (2008). Protecting People With Decisional Impairments and Legal Incapacity Against Biomedical Research Abuse. Journal of Disability Policy Studies 18: 230-244 [Abstract]  
• Timonen, M., Liukkonen, T. (2008). Management of depression in adults. BMJ 336: 435-439 [Full text]  
• Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. NEJM 358: 252-260 [Abstract] [Full text]  
• Dartnell, J., Hemming, M., Collier, J., Ollenschlaeger, G. (2008). Putting evidence into context: some advice for guideline writers. Evid. Based Nurs. 11: 6-8 [Full text]  
• Timimi, S. (2008). Child psychiatry and its relationship with the pharmaceutical industry: theoretical and practical issues. Adv. Psychiatr. Treat. 14: 3-9 [Abstract] [Full text]  
• Vandenbroucke, J. P., Elm, E. v., Altman, D. G., Gotzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., for the STROBE initiative, (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. ANN INTERN MED 147: W-163-W-194 [Abstract] [Full text]  
• Dartnell, J., Hemming, M., Collier, J., Ollenschlaeger, G. (2007). Putting evidence into context: some advice for guideline writers. Evid. Based Med. 12: 130-132 [Full text]  
• Hampson, L. A., Joffe, S., Fowler, R., Verter, J., Emanuel, E. J. (2007). Frequency, Type, and Monetary Value of Financial Conflicts of Interest in Cancer Clinical Research. JCO 25: 3609-3614 [Abstract] [Full text]  
• Hall, R., de Antueno, C., Webber, A. (2007). Publication bias in the medical literature: a review by a Canadian Research Ethics Board: [Biais de publication dans la litterature medicale : un compte rendu d'un Comite d'ethique de recherche canadien]. Canadian J. Anesthesia 54: 380-388 [Abstract] [Full text]  
• Bero, L A (2006). Clinical trial registration at Tobacco Control. Tobacco Control 15: 417-418 [Full text]  
• Abbass, A. (2006). Merits of psychotherapies.. CMAJ 175: 61-61 [Full text]  
• Chalmers, I. (2006). From optimism to disillusion about commitment to transparency in the medico-industrial complex. JRSM 99: 337-341 [Full text]  
• Psaty, B. M., Rennie, D. (2006). Clinical Trial Investigators and Their Prescribing Patterns: Another Dimension to the Relationship Between Physician Investigators and the Pharmaceutical Industry. JAMA 295: 2787-2790 [Full text]  
• Heres, S., Davis, J., Maino, K., Jetzinger, E., Kissling, W., Leucht, S. (2006). Why Olanzapine Beats Risperidone, Risperidone Beats Quetiapine, and Quetiapine Beats Olanzapine: An Exploratory Analysis of Head-to-Head Comparison Studies of Second-Generation Antipsychotics. Am. J. Psychiatry 163: 185-194 [Abstract] [Full text]  
• Bryan-Brown, C. W., Dracup, K. (2006). Disentangling The Web. Am J Crit Care 15: 7-9 [Full text]  
• MESHNICK, S. R., ALKER, A. P. (2005). AMODIAQUINE AND COMBINATION CHEMOTHERAPY FOR MALARIA. Am J Trop Med Hyg 73: 821-823 [Full text]  
• Goodman, D. C. (2005). When an Asthma Drug Has an Inferiority Complex: A Noninferiority Trial. Pediatrics 116: 493-495 [Full text]  
• Moncrieff, J., Kirsch, I. (2005). Efficacy of antidepressants in adults. BMJ 331: 155-157 [Full text]  
• Spitz, B., Abramson, J. (2005). When Health Policy Is the Problem: A Report from the Field. Journal of Health Politics, Policy and Law 30: 327-366 [Abstract]  
• Ferner, R E (2005). The influence of big pharma. BMJ 330: 855-856 [Full text]  
• Moncrieff, J., Hopker, S., Thomas, P. (2005). Psychiatry and the pharmaceutical industry: who pays the piper?: A perspective from the Critical Psychiatry Network. Psychiatr. Bull. 29: 84-85 [Full text]  
• Lexchin, J. R (2005). Implications of Pharmaceutical Industry Funding on Clinical Research. The Annals of Pharmacotherapy 39: 194-197 [Abstract] [Full text]  
• Lexchin, J., Mintzes, B. (2004). Transparency in drug regulation: Mirage or oasis?. CMAJ 171: 1363-1365 [Full text]  
• Chan, A.-W., Krleza-Jeric, K., Schmid, I., Altman, D. G. (2004). Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171: 735-740 [Abstract] [Full text]  
• van de Laar, F. A, Lucassen, P. L.B.J (2004). No evidence for a reduction of myocardial infarctions by acarbose. Eur Heart J 25: 1179-1179 [Full text]  
• Hanefeld, M.G (2004). Meta-analysis of long-term studies to assess the effect of acarbose on cardiovascular risk reduction - scientifically credible: Reply. Eur Heart J 25: 1179-1180 [Full text]  
• Chan, A.-W., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., Altman, D. G. (2004). Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles. JAMA 291: 2457-2465 [Abstract] [Full text]  
• Bryan-Brown, C. W., Dracup, K. (2004). Evidence-Based Pandemonium. Am J Crit Care 13: 10-12 [Full text]  
• Walker, B. F (2003). Risk of suicide remains high fifteen years after deliberate self-harm. Evid. Based Ment. Health 6: 106-106 [Full text]  
• Wager, E. (2003). Good practice in publication of clinical trial results. Br. J. Psychiatry 183 : 464-465 [Full text]  
• (2003). ABSTRACTS. Obstet Gynecol 102: 637-640 [Full text]  
• Als-Nielsen, B., Chen, W., Gluud, C., Kjaergard, L. L. (2003). Association of Funding and Conclusions in Randomized Drug Trials: A Reflection of Treatment Effect or Adverse Events?. JAMA 290: 921-928 [Abstract] [Full text]  
• (2003). Bias in Pharmaceutical-Sponsored Research?. Journal Watch Cardiology 2003: 4-4 [Full text]  
• (2003). Bias in Pharmaceutical-Sponsored Research?. JWatch General 2003: 4-4 [Full text]  
• Abbasi, K., Smith, R. (2003). No more free lunches. BMJ 326: 1155-1156 [Full text]  

Rapid Responses:

Read all Rapid Responses

EBM: the new religion - a further thought

Luis Vitetta, et al.
bmj.com, 30 May 2003 [Full text]

At least, there is a bias in a lot of results...

Sergio Stagnaro
bmj.com, 1 Jun 2003 [Full text]

Debate about whether antidepressants are better than placebos

D B Double
bmj.com, 2 Jun 2003 [Full text]