Education and debate

meta-analysis bias in location and selection of studies

^a Department of Social Medicine, University of Bristol, Bristol BS8 2PR

Correspondence to: Dr Egger m.egger@bristol.ac.uk

	Introduction

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

Meta-analysis has received a mixed reception since the outset. Some people have rejected what they see as exercises in "mega-silliness,"¹ while the purveyors of a highly distinguished series of meta-analyses of perinatal medical care² have been dismissed as "an obstetrical Baader-Meinhof gang."³ To some clinicians objecting to the findings of meta-analyses, "a tool has become a weapon."⁴ At the other end of the spectrum, the application of a technique that basically consists of calculating a weighted average has been hailed as "Newtonian,"⁵ and it has been suggested that meta-analysis has left no place for the narrative review article.⁶ The truth is likely to lie somewhere between these extreme views.

That meta-analysis holds potential problems can be illustrated by contrasting the conclusions of two meta-analyses comparing low molecular weight heparins and standard heparin in the prevention of thrombosis after surgery.^{7 8} One group concluded that "low molecular weight heparins seem to have a higher benefit to risk ratio than unfractionated heparin in preventing perioperative thrombosis,"⁷ whereas the other considered that "there is at present no convincing evidence that in general surgery patients low molecular weight heparins, compared with standard heparin, generate a clinically important improvement in the benefit to risk ratio."⁸ Various differences exist between these meta-analyses, but the main difference relates to the selection of studies for inclusion (1). Nurmohamed et al based their analysis on a subgroup of trials that they considered to possess the highest methodological strength,⁸ while Leizorovicz et al included all trials in their analysis.⁷ The 1 shows that many other elements—for example, language restrictions or use of unpublished material—could contribute to contrasting conclusions being reached.

	Publication bias

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

The most obvious problem is that some studies never get published. If the reasons that studies remain unpublished are associated with their outcome then the result of a meta-analysis could be seriously biased. Hypothetically, with a putative treatment that has no actual effect on a disease, studies suggesting a beneficial effect might end up being published, while an equal mass of data pointing the other way might remain unpublished. In this situation a meta-analysis of the published trials would identify a spurious beneficial treatment effect. In the field of cancer chemotherapy this has indeed been shown, in a comparison of the results from studies identified in a literature search with those contained in an international trials registry⁹ (box).

A demonstration of publication bias Studies with significant results are more likely to get published than those with non-significant results The inclusion of a study in a trials register can be assumed not to be influenced by its results because registration generally takes place before completion of the study The studies enlisted in a register are therefore likely to constitute a more representative sample of all the studies that have been performed in a given area than a sample of published studies This principle has been tested for trials of different cancer chemotherapies by comparing the results from meta-analysis of trials identified in a literature search and of trials registered with the International Cancer Research Data Bank (4)9 View larger version (10K): [in this window] [in a new window]   Analysis of published clinical trials indicates considerably better survival of patients with advanced ovarian cancer treated with combination chemotherapy compared with monotherapy with alkylating agent. Analysis of registered trials failed to confirm this

Such publication bias has been a matter of concern in education research and psychology for over 30 years.^{10 11 12} Work on the existence and importance of publication bias in the medical literature is more recent. Five separate studies have investigated this by following up research proposals approved by ethics committees.^{13 14 15 16} Of 285 studies approved by one research ethics committee between 1984 and 1987 that had been completed and analysed, 138 (48%) had been published by 1990. Studies with significant (P<0.05) results were more likely to have been published than those with non-significant results.¹³ A meta-analysis of all five studies shows that this is a consistent finding, with little heterogeneity between studies (fig 1). The odds of publishing results were three times greater if the results were significant (combined odds ratio 3.0, 95% confidence intervals 2.3 to 3.9), and such publication bias was found for both clinical trials and observational studies. Interestingly, studies continue to appear in print many years after approval by the ethics committee. Stern et al (ethics committee E) found that about 85% of studies with significant results compared with 65% of studies with null results had been published after 10 years.¹⁶ The median time to publication was 4.8 years for studies with significant results and 8.0 years for studies with null results.

View larger version (15K): [in this window] [in a new window]   Fig 1 Meta-analysis of five studies examining association of significant results and publication among research proposals submitted to ethics committees. The unadjusted odds ratios were combined by using a fixed effects model

The source of funding was associated with publication or non-publication independently of study results.^{13 14 16} Studies sponsored by the pharmaceutical industry were less likely to be published than those supported by the government or by voluntary organisations, with investigators citing the data management by these companies as a reason for non-publication.^{13 14} This is in agreement with a review of publications of clinical trials that grouped them into those which were sponsored by the pharmaceutical industry and those supported by other means.¹⁷ The results of 89% of published industry-supported trials favoured combination chemotherapy over monotherapy with an alkylating agent, compared with 61% of the other trials. Similar results have been reported from an overview of trials of non-steroidal anti-inflammatory drugs.¹⁸ The implication is that the pharmaceutical industry discourages the publication of studies that it has funded which have negative findings. Finally, multicentre studies were more likely to be published than studies from a single centre.¹⁴ However, high quality trials were not more likely to be published than trials of lower quality.¹³

Selective submission of papers rather than selective acceptance of papers by journals seems to be the dominant contributor to publication bias.^{13 14 15 16} However, that selective acceptance does occur is illustrated by the "instructions to authors" section of one major journal on diabetes, which stated that "mere confirmation of known facts will be accepted only in exceptional cases; the same applies to reports of experiments and observations having no positive outcome."¹⁹ Many authors may not submit studies with negative findings because they anticipate rejection.

	Bias in location of studies

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

Although publication bias has long been recognised¹⁰ and much discussed,^{20 21} other factors can contribute to biased inclusion of studies in meta-analyses. Indeed, among published studies the probability of identifying relevant trials for meta-analysis is also influenced by their results. These biases have received much less consideration than publication bias, but their consequences could be equally important.

English language bias
Meta-analyses published in English language journals are often exclusively based on trials published in English. For example, of 36 meta-analyses published in leading English language general medical journals from 1991 to 1993, 26 had restricted their search to studies reported in English.²² Investigators working in a non-English speaking country will, however, publish some of their work in local journals.^{23 24 25} Authors might be more likely to report positive findings in an international, English language journal and negative findings in a local journal. Bias could thus be introduced in meta-analyses based exclusively on reports in English.^{22 24}

View larger version (84K): [in this window] [in a new window]   MARK HUDSON

We have examined this issue for literature published in German. All randomised controlled trials published in five German, Swiss, and Austrian general medical journals from 1985 to 1994 were identified in a manual search.²⁵ The Medline database was then searched for randomised controlled trials published in English during the same 10 year period.²⁶ Comparison of pairs of articles published by the same first author found that 63% of trials published in English had produced significant (P<0.05) results, compared with 35% of trials published in German. In logistic regression analysis the odds for publication in English were 3.8 (95% confidence interval 1.3 to 11.3) times higher if the results were significant. This association was little changed when adjusted for study sample size, design, and quality. Indeed, quality scores were closely similar for English and German language reports.²⁶ These findings show that for publications of randomised trials from German speaking Europe, an English language bias does exist and that ignoring trials published in German is problematic. The same situation is likely in relation to other languages, particularly European languages.

Database bias
Whereas most of the major west European journals that are published in languages other than English are indexed in Embase or Medline, this is not the case for journals published in less developed countries. Among the 3000 to 4000 journals indexed by Medline, Embase, or the Science Citation Index, only about 2% are from the less developed world.²⁷ For example, only 30 journals out of a total of 3861 journals indexed in Medline are published in India, despite the fact that India is the developing country with the largest research output²⁸ and that its medical research is published in English. Studies that are published in journals not indexed in one of the major databases means that these data are effectively hidden from reviewers and meta-analysts. A minority of trials will be published in indexed local or international journals, but results and other characteristics are likely to differ between these two groups. Indeed, trials with significant results might be more likely to be published in an indexed journal, whereas trials with null results are published in non-indexed journals.

Citation bias
In locating studies, researchers often supplement searches of computerised databases by contacting experts in the field and checking the reference lists of other studies and reviews. When reference lists are used, citation bias could have an important role. In the field of cholesterol lowering, trials that are supportive of a beneficial effect are cited more frequently than unsupportive trials, regardless of the size and quality of the studies involved²⁹ (box). Thus the use of reference lists would be more likely to locate supportive studies, which could bias the findings of a meta-analysis.

Citation and selection of studies bias A meta-analysis of seven trials of cholesterol lowering after myocardial infarction30 defined its inclusion criteria as single-factor randomised trials with at least 100 participants per group, with at least three years of follow up, and without the use of hormone treatment to lower cholesterol. The pooled results for all cause mortality showed a favourable trend (odds ratio 0.91, 95% confidence interval 0.82 to 1.02) (5). View larger version (10K): [in this window] [in a new window]   Results from included, missed, and excluded trials in a meta-analysis of cholesterol lowering after myocardial infarction30 One trial met all the entry criteria but was not included.31 In this study the odds ratio for overall mortality was an unfavourable 1.60 (0.95 to 2.70). For the trials included in the analysis the mean annual citation count per study for the period up to five years after publication was 20; for the study that was not included it was less than 1.29 Citation bias may thus have distorted the results of this meta-analysis. Eleven other secondary prevention trials were available at the time but did not meet the somewhat arbitrary inclusion criteria.32 The pooled odds ratio for all cause mortality for these trials is 1.14 (1.03 to 1.26). Inclusion biases may thus have influenced the conclusions of this meta-analysis.

The journals in which papers are published could also influence the ease of their location and their inclusion in meta-analyses. One influential cholesterol lowering trial, for example, was originally planned as a study with primary prevention and secondary prevention arms.³⁴ The results of the primary prevention component were interpreted by the investigators as being favourable, and the results were published in the New England Journal of Medicine in 1987.³⁵ The secondary prevention arm finished at the same time, but in this case the results were clearly unfavourable.³⁶ The findings from this arm were not published until 1993, in the Annals of Medicine,³⁶ a journal with limited circulation. The paper in the New England Journal of Medicine received more than 450 citations in the three years after publication, whereas the article in the Annals of Medicine received 17.

Multiple publication bias
Multiple publications from single studies can lead to bias in several ways.³⁷ Studies with significant results are more likely to lead to multiple publications and presentations,¹³ which makes it more likely that they will be located and included in a meta-analysis. Furthermore, the inclusion of duplicated data may lead to overestimation of treatment effects, as recently shown for trials of ondansetron, a 5-HT3 receptor antagonist that is used for prevention and treatment of postoperative nausea and vomiting.³⁸ It is not always obvious that multiple publications come from a single study, and one set of study participants may thus be included in an analysis twice. This is a particular problem in multicentre trials. In one of the meta-analyses of low molecular weight heparin,⁸ for example, it seems that combined data from a multicentre trial were included, together with a subset of the same data that had also been reported separately from an individual centre which contributed to the trial.³⁹ Indeed, it may be difficult, if not impossible, for meta-analysts to determine whether two papers represent duplicate publications of one trial or two separate trials, as examples exist where two articles reporting the same trial do not share a single common author.^{37 38 40}

Bias in provision of data
Some additional data not reported in print are often needed for meta-analysis—for example, in the case of reports that do not provide adequate stratification of data—but it may prove difficult to obtain this information. Many factors could influence the willingness of investigators to make their data available, but one element could be the direction of the results. Such provision of data bias could also occur in the case of unpublished trials or trials published only as conference abstracts.

	Biased inclusion criteria

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

Although studies might have been located and data obtained, potential for bias might still arise in establishing the inclusion criteria for a meta-analysis. If, as is usual, the inclusion criteria are developed by an investigator familiar with the area under study, the criteria can be influenced by knowledge of the results of the set of potential studies. Manipulating the inclusion criteria could lead to selective inclusion of studies with positive findings and exclusion of studies with negative findings. For example, some meta-analyses of trials of cholesterol lowering treatment have excluded certain studies on the grounds that the treatments used seem to have had an adverse effect independent of cholesterol lowering itself.^{41 42} These meta-analyses have, however, included trials of treatments that are likely to influence favourably the risk of coronary heart disease, independent of cholesterol lowering. Clearly such an asymmetrical approach introduces the possibility of selection bias, with the criteria for inclusion in the meta-analysis being derived from the results of the studies (see box on citation and selection bias).

A more recent example relates to a meta-analysis of trials of dietary interventions in community settings.⁴³ The authors excluded a trial because the randomisation process was considered to be inadequate, but they included a duplicate publication of the same trial with an almost identical description of the randomisation procedure.⁴⁴ Two independent observers assessed eligibility of trials, with disagreements being resolved by a third author. This example shows how vulnerable the selection process can be in meta-analysis.

	Examining for bias

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

The most appropriate way of handling the selection of studies is to include all studies that meet basic entry criteria then perform sensitivity analyses with regard to the different possible entry criteria. Any conclusions from a meta-analysis that are highly sensitive to altering the entry criteria should be treated with caution. In addition to such sensitivity analyses, the likely presence or absence of bias should be examined graphically in funnel plots.

View larger version (13K): [in this window] [in a new window]   Fig 2 Funnel plot of mortality results from trials of ß blockers in secondary prevention after myocardial infarction. The odds ratios are plotted against study sample size

Funnel plots
Funnel plots—simple scatterplots of the trials' effect estimates against their sample size—are useful to detect bias in meta-analysis.⁴⁵ The funnel plot is based on the fact that precision in estimating the underlying treatment effect will increase as the sample size of component studies increases. Results from small studies will scatter widely at the bottom of the graph. The spread will narrow as precision increases among larger studies. In the absence of bias, the plot should thus resemble a symmetrical inverted funnel. If the plot shows an asymmetrical and skewed shape, bias may be present. This usually takes the form of a gap in the wide part of the funnel, which indicates the absence of small studies showing no benefit or harm. The funnel plot is a graphical test for any type of bias that is associated with sample size (box). The publication and location biases described in this article are more likely to affect smaller studies than large trials and may thus lead to funnel plot asymmetry. Another source of asymmetry arises from differences in the methodological quality. Smaller studies are, on average, conducted and analysed with less methodological rigour than larger studies, and trials of lower quality tend to show larger effects.⁴⁶ Statistical methods that provide an objective measure of funnel plot asymmetry have recently become available.^{45 47}

An example—ß blockers after myocardial infarction
In this series we have repeatedly discussed a meta-analysis of 17 trials assessing ß blockers in secondary prevention after myocardial infarction. The sensitivity analysis that we presented earlier⁴⁸ showed that the results were robust to the choice of the statistical method and to the exclusion of trials of lesser quality or of studies terminated early. The funnel plot is shown in figure 3. Visual assessment shows some asymmetry, which indicates that there was selective non-publication of smaller trials with less sizeable benefit. However, in formal statistical analysis^{45 47} the degree of asymmetry is found to be small and non-significant (P>0.1). Furthermore, exclusion of the smaller studies had little effect on the overall estimate.⁴⁸ Bias does not therefore seem to have distorted the findings from this meta-analysis.

	Conclusions

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

Biases in publication, location, and inclusion are a potentially serious problem in meta-analysis^{49 50 51} Critical examination for the presence of such biases in sensitivity and funnel plot analyses should therefore form an integral part of meta-analyses.⁴⁵ The effort of the Cochrane Collaboration to identify as many controlled trials as possible through manual searches of a large number of medical journals published in many different languages is of great importance to reduce such bias. Indeed, the Cochrane Controlled Trials Register⁵² is nowadays likely to be the best single source of published trials for inclusion in systematic reviews and meta-analyses. To eliminate the risk of publication bias, however, trials need to be registered at the time they are established. To ensure registration, the ethical approval of studies could be linked to a requirement that trials are reported to a central register.^{33 49 53} Furthermore, results of completed trials could be submitted to the ethical committee, the reports could be kept centrally, and requests for unpublished trials could be sent to this body. At present, however, much effort is still needed to ensure that the set of studies located for a meta-analysis is not a biased sample of all existing studies.

	Acknowledgements

The department of social medicine at the University of Bristol is part of the Medical Research Council's health services research collaboration.

	References

Top Introduction Publication bias Bias in location of... Biased inclusion criteria Examining for bias Conclusions References

1. Eysenck HJ. An exercise in mega-silliness. Am Psychol 1978;33:517.
2. Chalmers I, Enkin M, Keirse M. Effective care during pregnancy and childbirth. Oxford: Oxford University Press, 1989.
3. Mann C. Meta-analysis in the breech. Science 1990;249:476-80. [Free Full Text]
4. Boden WE. Meta-analysis in clinical trials reporting: has a tool become a weapon? Am J Cardiol 1992;69:681-6. [Medline]
5. Mann C, Peto R. Statistician with a mission. Science 1990;249:479. [Free Full Text]
6. Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias. JAMA 1990;263:1392-5. [Abstract]
7. Leizorovicz A, Haugh MC, Chapuis F-R, Samama MM, Boissel J-P. Low molecular weight heparin in prevention of perioperative thrombosis. BMJ 1992;305:913-20.
8. Nurmohamed MT, Rosendaal FR, Bueller HR, Dekker E, Hommes DW, Vandenbroucke JP, et al. Low-molecular-weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis. Lancet 1992;340:152-6. [Medline]
9. Simes RJ. Confronting publication bias: a cohort design for meta-analysis. Stat Med 1987;6:11-29. [Medline]
10. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J Am Stat Assoc 1959;54:30-4.
11. Rosenthal R. The "file drawer problem" and tolerance for null results. Psychol Bull 1979;86:638-41.
12. Light RJ, Pillemer DB. Summing up. The science of reviewing research. Cambridge, MA; London: Harvard University Press, 1984.
13. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867-72. [Medline]
14. Dickersin K, Min YL, Meinert CL. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA 1992;267:374-8. [Abstract]
15. Dickersin K, Min YL. NIH clinical trials and publication bias. Online J Curr Clin Trials [serial on line] 1993;28 Apr:Doc No 50.
16. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640-5. [Abstract/Free Full Text]
17. Davidson RA. Source of funding and outcome of clinical trials. J Gen Intern Med 1986;1:155-8. [Medline]
18. Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, Minaker KL, et al. A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med 1994;154:157-63. [Abstract]
19. Manuscript guideline. Diabetologia 1984;25:4A.
20. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. Publication bias in clinical trials. Controlled Clinical Trials 1987;8:343-53. [Medline]
21. Begg CB, Berlin JA. Publication bias and dissemination of clinical research. J Natl Cancer Inst 1989;81:107-15. [Abstract/Free Full Text]
22. Grégoire G, Derderian F, LeLorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63. [Medline]
23. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91. [Abstract/Free Full Text]
24. Moher D, Fortin P, Jadad AR, Jüni P, Klassen T, Le Lorier J, et al. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 1996;347:363-6. [Medline]
25. Egger M, Zellweger T, Antes G. Randomised trials in German-language journals. Lancet 1996;347:1047-8. [Medline]
26. Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350:326-9. [Medline]
27. Zielinski C. New equities of information in an electronic age. BMJ 1995;1480-1.
28. Singh R, Singh S. Research and doctors. Lancet 1994;344:546. [Medline]
29. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15-9.
30. Rossouw JE, Lewis B, Rifkind BM. The value of lowering cholesterol after myocardial infarction. N Engl J Med 1990;323:1112-9. [Medline]
31. Woodhill JM, Palmer AJ, Leelarthaepin B, McGilchrist C, Blacket RB. Low fat, low cholesterol diet in secondary prevention of coronary heart disease. Adv Exp Med Biol 1978;109:317-30. [Medline]
32. Davey Smith G, Song F, Sheldon TA. Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ 1993;306:1367-73.
33. Savulescu J, Chalmers I, Blunt J. Are research ethics committees behaving unethically? Some suggestions for improving performances and accountability. BMJ 1996;313:1390-3. [Free Full Text]
34. Manninen V. Clinical results with gefibrozil and background to the Helsinki heart study. Am J Cardiol 1983;52:35-8B.
35. Frick MH, Elo O, Haapa K, Heinonen OP, Heinsalmi P, Helo P, et al. Helsinki heart study: primary-prevention trial with gemfibrozil in middle-aged men with dyslipidemia. N Engl J Med 1987;317:1237-45. [Abstract]
36. Frick MH, Heinonen OP, Huttunen JK, Koskinen P, Manttari M, Manninen V. Efficacy of gemfibrozil in dyslipidaemic subjects with suspected heart disease. An ancillary study in the Helsinki heart study frame population. Ann Med 1993;25:41-5. [Medline]
37. Huston P, Moher D. Redundancy, disaggregation, and the integrity of medical research. Lancet 1996;347:1024-6. [Medline]
38. Tramèr MR, Reynolds DJM, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997;315:635-40. [Abstract/Free Full Text]
39. Leizorovicz A, Haugh MC, Boissel JP. Meta-analysis and multiple publications of clinical trial reports. Lancet 1992;340:1102-3.
40. Felson DT. Bias in meta-analytic research. J Clin Epidemiol 1992;45:885-92. [Medline]
41. Peto R, Yusuf S, Collins R. Cholesterol-lowering trial results in their epidemiologic context. Circulation 1985;72(suppl 3):451.
42. MacMahon S. Lowering cholesterol: effects on trauma death, cancer death and total mortality. Aust N Z J Med 1992;22:580-2. [Medline]
43. Tang JL, Armitage JM, Lancaster T, Silagy CA, Fowler GH, Neil HAW. Systematic review of dietary intervention trials to lower cholesterol in free-living individuals. BMJ (in press).
44. Davey Smith G, Ebrahim S. Dietary change, cholesterol reduction and the public health: what does meta-analysis add? BMJ (in press).
45. Egger M, Davey Smith G, Schneider M, Minder CE. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34. [Abstract/Free Full Text]
46. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12. [Abstract]
47. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994;50:1088-99. [Medline]
48. Egger M, Davey Smith G, Phillips AN. Meta-analysis: principles and procedures. BMJ 1997;315:1533-7. [Free Full Text]
49. Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263:1405-8. [Abstract]
50. Egger M, Davey Smith G. Misleading meta-analysis. Lessons from "an effective, safe, simple" intervention that wasn't. BMJ 1995;310:752-4. [Free Full Text]
51. Pearn J. Publication: an ethical imperative. BMJ 1995;310:1313-5. [Free Full Text]
52. Cochrane Controlled Trials Register. In: The Cochrane Library [database on disk and CD ROM]. The Cochrane Collaboration. Issue 1. Oxford: Update Software, 1997. Updated quarterly.
53. Levy G. Publication bias: its implications for clinical pharmacology. Clin Pharmacol Ther 1992;52:115-9. [Medline]