Ruling a diagnosis in or out with “SpPIn” and “SnNOut”: a note of caution
=================================================================================

* Daniel Pewsner
* Markus Battaglia
* Christoph Minder
* Arthur Marx
* Heiner C Bucher
* Matthias Egger

## Introduction

Dr X is back from her annual leave. Dr Y, the locum doctor, reports on the patients he saw during her absence, including a 40 year old teacher who had sprained her right ankle. Returning from a conference, she had stumbled while walking down the stairs with a heavy bag. Examination revealed a moderately swollen lateral right ankle. The patient was able to walk but was clearly in pain. Her breath smelt of alcohol.

## Ruling diagnoses in and out with SpPIns and SnNOuts

Dr Y had applied the Ottawa ankle rules—decision rules designed to exclude fractures of the malleolus and the midfoot—and found no bone tenderness.1 He had previously visited the website of a centre for evidence based medicine2 and printed out a list of diagnostic tests that can rule out, or rule in, the condition in question without requiring further investigations.

The probability of disease, given a positive or negative test result (post-test probability), is usually obtained by calculating the likelihood ratio of the test result and using formulas based on Bayes's theorem (see box 1), or a nomogram,3 to convert the estimated probability of the suspected diagnosis before the test result was known (pretest probability) into a post-test probability, which takes the result into account.4 Likelihood ratios indicate how many times more likely a test result is to be expected in a patient with the disease compared with a person free of the disease and thus measure a test's ability to modify pretest probabilities.

David Sackett and others have argued that such calculations are unnecessary when a test is highly sensitive or highly specific.4–6 In this situation the likelihood ratio of a negative test will generally be very small, and the likelihood ratio of a positive test very large. A negative test will thus rule out, and a positive result rule in, disease. Two mnemonics that capture the properties of such tests have been coined: SnNOut (high sensitivity, negative, rules out) and SpPIn (high specificity, positive, rules in).4 This concept has become increasingly popular, with many websites for evidence based medicine listing such tests and inviting users to nominate further SpPIn and SnNOut tests. The understanding of the SnNOut principle among medical students was recently examined in a randomised trial.7

#### Summary points

Negative results from highly sensitive tests can rule a diagnosis out (sensitive, negative, out = SnNOut), and positive results from highly specific tests can rule a diagnosis in (specific, positive, in = SpPIn)

Studies quoted as showing SpPIn or SnNOut properties may be affected by spectrum bias, partial verification bias, or incorporation bias. Others may be too small to define test characteristics with sufficient precision

The power of a test to rule a diagnosis out does not depend exclusively on its sensitivity, as suggested by the SnNOut rule, but is reduced by low specificity. Similarly, the power to rule in depends on both specificity and sensitivity

The evidence from studies of a test's accuracy should be critically assessed, and post-test probabilities (with 95% confidence intervals) should be calculated when evaluating potential SnNOut or SpPIn tests

Assuming that a diagnosis can be ruled in or out with confidence, when in reality it cannot, could have serious consequences for patients

The website listed the Ottawa ankle rules as a SnNOut test,2 indicating that in the teacher's case a fracture could safely be ruled out without radiography. Indeed, the patient made an uneventful and full recovery within four weeks. Alerted by the patient's alcoholic breath, Dr Y wondered whether an alcohol problem might have contributed to the accident and used the CAGE questions (see box 2) to investigate this further. According to the same website,2 the CAGE instrument has SpPIn properties, ruling the diagnosis in if two or more questions are answered affirmatively. The patient confirmed that she felt she should cut down on alcohol and that she had felt bad repeatedly about her drinking. Dr X, who had known her for over 10 years, explained that the patient's alcohol intake was moderate and well controlled, and that she was socially well integrated but very health conscious and somewhat anxious. A few days later, Dr X got a telephone call from the patient, who was clearly upset about the locum doctor's suggestion that she had an alcohol problem. A further consultation was required to clarify the situation and restore trust.

### Box 1: Definitions of concepts and terms

*Sensitivity*—The proportion of people with the disease who are correctly identified by a positive test result (“true positive rate”)

*Specificity*—The proportion of people free of the disease who are correctly identified by a negative test result (“true negative rate”)

*SnNOut*—Mnemonic to indicate that a negative test result (N) of a highly sensitive test (Sn) rules out the diagnosis (Out)

*SpPIn*—Mnemonic to indicate that a positive test result (P) of a highly specific test (Sp) rules in the diagnosis (In)

*Likelihood ratios*—Measure of a test result's ability to modify pretest probabilities. Likelihood ratios indicate how many times more likely a test result is in a patient with the disease compared with a person free of the disease.

*Likelihood ratio of a positive test result (LR*+)—The ratio of the true positive rate to the false positive rate: sensitivity/(1-specificity)

*Likelihood ratio of a negative test result (LR*-)—The ratio of the false negative to the true negative rate: (1-sensitivity)/specificity

*Pretest probability (prevalence)*—The probability that an individual has the target disorder before the test is carried out

*Post-test probability*—The probability that an individual with a specific test result has the target condition (post-test odds/[1+post-test odds]) *or*

![Formula][1]</img>  
*Pretest odds*—The odds that an individual has the target disease before the test is carried out (pretest probability/[1-pretest probability])

*Post-test odds*—The odds that a patient has the target disease after being tested (pretest odds×LR)

*Positive predictive value (PPV)*—The proportion of individuals with positive test results who have the target condition. This equals the post-test probability given a positive test result

*Negative predictive value (NPV)*—The proportion of individuals with negative test results who do not have the target condition. This equals one minus the post-test probability given a negative test result

## Critical appraisal of test evaluation studies

In this article, we examine examples of test evaluation studies that websites and a textbook of evidence based medicine4 have cited as showing that the tests had SpPIn or SnNOut properties. The studies were chosen to illustrate methodological issues. We assessed the quality of studies as described elsewhere,8 extracted the two-by-two table from the original publication, and calculated likelihood ratios and post-test probabilities with exact binomial 95% confidence intervals based on the pretest probabilities observed in the studies. Finally, we examined whether the post-test probability of the condition in question in the population studied was compatible with the notion of safely ruling the condition in or out, and considered the transferability of study results to other settings and populations. Tables 1 and 2 summarise the study characteristics and results from our critical appraisal.

View this table:
[Table 1.](http://www.bmj.com/content/329/7459/209/T1)

Table 1. 
Characteristics of test evaluation studies interpreted as demonstrating SnNOut properties (high sensitivity and negative result rules out)

View this table:
[Table 2.](http://www.bmj.com/content/329/7459/209/T2)

Table 2. 
Characteristics of test evaluation studies interpreted as demonstrating SpPIn properties (high specificity and positive result rules in)

### Box 2: CAGE questionnaire for detecting alcohol misuse

1.  Have you ever felt you should **C**ut down on your drinking?

2.  Have people **A**nnoyed you by criticising your drinking?

3.  Have you ever felt bad or **G**uilty about your drinking?

4.  Have you ever had a drink first thing in the morning to steady your nerves or to get rid of a hangover (**E**ye opener)?

### Random error and bias

A diagnostic study may be too small to define test performance with sufficient precision. For example, a website2 and the textbook4 interpreted a study of ankle swelling in patients with suspected ascites10 as demonstrating SnNOut properties. The absence of a history of ankle swelling is thus assumed to rule out ascites.2 4 However, the study was based on only 15 patients with ascites and confidence intervals were wide, with the lower 95% confidence interval of the sensitivity including 68%. This means that absence of ankle swelling is still compatible with a 15.8% probability of ascites, which clearly is unacceptably high (table 1).

Studies with methodological flaws tend to overestimate the accuracy of diagnostic tests.21 Bias can be introduced when tests are evaluated in patients known to have the disease and in people known to be free of it—so called diagnostic case-control studies. In this situation patients with borderline or mild expressions of the disease, and conditions mimicking the disease are excluded, which can lead to exaggeration of both sensitivity and specificity.21 This is called spectrum bias because the spectrum of study patients will not be representative of patients seen in practice. For example, the textbook considered auscultatory percussion in the diagnosis of pleural effusion as a SpPIn test.4 This assessment was based on a study that compared patients who were selected because of the presence or absence of radiological signs of effusion.16 The impressive results (100% specificity and 96% sensitivity (table 2)) may therefore not be reliable. The textbook and a website22 also claim that the presence of retinal vein pulsation in ophthalmoscopy excludes increased intracranial pressure (a SnNOut test). This is based on a study that compared patients known to have increased pressure with people not suspected to have increased intracranial pressure.11

Partial verification bias may be introduced when the reference test or tests are not applied consistently to confirm negative results of the index test. Some patients are either excluded or considered true negatives. This may lead to overestimation of sensitivity and underestimation of specificity or to overestimation of sensitivity and specificity.21 The textbook considered the CAGE questionnaire for diagnosing alcohol misuse (box 2) to be a SpPIn test.4 This is based on a study that subjected only a fraction of CAGE-negative persons to further testing (liver enzymes, medical record review, and physician interviews (table 2)), 17 thus possibly introducing bias.

Similarly, incorporation bias may be present if the test under evaluation is also part of the reference test.21 This will lead to overestimation of test accuracy because experimental and reference tests are no longer independent. For example, a website listed abdominojugular reflux for the diagnosis of congestive heart failure as a SpPIn test,22 on the basis of a study that used clinicoradiographic criteria, including abdominojugular reflux, as the reference test (table 2).18

### Sensitivity and specificity

The likelihood ratio associated with a negative test result does not depend on its sensitivity alone, as suggested by the SnNOut rule, but also on its specificity. For example, a website considered that the clinical criteria for the diagnosis of Alzheimer's disease had SnNOut properties2 based on a sensitivity of 93% (table 1).12 However, despite this high sensitivity, the likelihood ratio of a negative test was a modest 0.3, because of the test's low specificity of 23% (100 - 93/23 = 0.3, see box 1). Indeed, in the population studied, the probability of Alzheimer's disease, given a negative test, was 25% (table 1). The power to rule out a diagnosis thus depends on both sensitivity and specificity.

Similarly, the ability to rule in depends not only on specificity, as suggested by the SpPIn rule, but also on sensitivity. A study examining the presence of a third heart sound in the diagnosis of congestive heart failure (table 2)19—which a website interpreted as demonstrating SpPIn properties23—is an example of a highly specific test (99%) that suffers from a low sensitivity (24%). The figure shows how the power to rule a disease in or out is eroded when highly specific tests are not sufficiently sensitive, or highly sensitive tests are not sufficiently specific.

![Figure1](http://www.bmj.com/https://www.bmj.com/content/bmj/329/7459/209/F1.medium.gif)

[Figure1](http://www.bmj.com/content/329/7459/209/F1)

Ability of a test to rule in disease (measured by the likelihood ratio of a positive test) as a function of specificity and sensitivity (upper panel), and ability of a test to rule out disease (measured by the inverse of the likelihood ratio of a negative test result) as a function of sensitivity and specificity (lower panel)

### Transferability and applicability

The performance of a diagnostic test often varies considerably from one setting to another, which may be due to differences in the definition of the disease, the exact nature of the test, and its calibration and the characteristics of those with and without the disease in a given setting.21 For example, patients attending primary care practices will generally have disease at an earlier stage than patients in secondary and tertiary care, which may reduce a test's sensitivity. Patients free of the disease in tertiary care will tend to have other conditions, which could reduce the specificity of a diagnostic test. Interpreting data on a test's accuracy thus requires defining the exact nature of the test used, the disease, and the patient population studied. For example, the website that listed the CAGE questionnaire as a SpPIn test for alcohol dependence2 cited, as the evidence for this, a study that had been performed in black women admitted to a trauma centre in the United States,20 which may not be applicable to other populations and settings.

Even when we assume that sensitivity and specificity do not change between settings and patient populations, test results will have different interpretations depending on whether a test is performed in a low risk population, such as in primary care, or high risk patients in a referral centre. For example, in the study evaluating the third heart sound in the diagnosis of heart failure,19 the pretest probability or prevalence in a general practice setting was 16%. In this situation, a positive test with a likelihood ratio of 18 will not allow the diagnosis to be ruled in with confidence: the post-test probability is only increased to 77% (table 2). If the pretest probability were 50%, however—such as in a cardiology outpatient clinic—the same positive test would produce a post-test probability of 95% (see box 1 for formula).

The interpretation of studies will be strongly influenced by the nature of the condition and the invasiveness of further investigations. For example, a study assessing urinary albumin:creatinine ratios below 1.8 g/mol for ruling out microalbuminuria in men with type 2 diabetes in primary care13 and the study examining the absence of a history of ankle swelling for ruling out ascites in men admitted to general internal medicine wards10 both produced post-test probabilities of about 3%. In the first case, we accepted a website's conclusion that the urinary albumin:creatinine ratio had SnNOut properties2: we thought that the post-test probability of microalbuminuria was sufficiently low with a negative test result, considering that guidelines recommend regular testing of patients with type 2 diabetes.24 In the second case, however—and unlike the textbook4—we thought that in men with suspected ascites but no history of ankle swelling a probability of ascites of 3%, a sign often associated with serious conditions, was still too high and that sonography should be used to rule the diagnosis in or out.25 As mentioned above, another problem with this study is the small sample size, which resulted in wide confidence intervals.

## Conclusions

Prompted by a colleague's experience with the Ottawa ankle rule and the CAGE questionnaire, we examined diagnostic test evaluation studies, which on websites and in a textbook of evidence based medicine were interpreted as demonstrating tests' ability to conclusively rule a diagnosis in or out (SpPIn or SnNOut tests). We calculated likelihood ratios to measure the tests' power to rule the target conditions in or out and assessed the study designs for possible bias, and we have given examples of tests where these methodological issues raise questions about whether the tests have SpPIn or SnNOut properties.

We believe that the concept of SpPIn or SnNOut tests can help clinicians in interpreting diagnostic test results. However, identifying and promoting tests with SpPIn or SnNOut properties should be based on a careful appraisal of the evidence, including the methodological quality of the test evaluation studies, and not simply on the test's sensitivity or specificity. Likelihood ratios and typical post-test probabilities should be calculated and reported, together with measures of statistical uncertainty, as recommended in the Standards for Reporting Diagnostic Accuracy Studies (STARD).26 Assessments should ideally be based on a systematic review of all available studies, which may include a meta-analysis to increase the precision of estimates of test accuracy. For example, a recent meta-analysis of 27 test accuracy studies of the Ottawa ankle rules confirmed that in many settings this decision aid can indeed exclude fractures and reduce the number of unnecessary radiographs.14 The fact that results may not be transferable to other populations and settings should be stressed, and the information required to judge transferability and applicability should be provided. Clearly, assuming that a diagnosis can be ruled in or ruled out with confidence, when in reality it cannot, could have serious consequences for patients.

## Footnotes

*   We thank Nicola Low for helpful comments on an earlier version of this manuscript.

*   Contributors DP had the idea of critically appraising test accuracy studies that were interpreted as demonstrating SpPIn or SnNOut properties. CM advised on statistical issues. ME and DP wrote the first draft of the article. All authors contributed to the appraisal of the evidence from studies and to writing the final draft of the article.

*   Funding The Krankenfürsorgestiftung der Gesellschaft für das Gute und Gemeinnützige (GGG), Basle, Switzerland, and the Swiss Academy of Medical Sciences supported this study.

*   Competing interests None declared.

## References

1.  1.Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR.A study to develop clinical decision rules for the use of radiography in acute ankle injuries.Ann Emerg Med1992; 21:384–90.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0196-0644(05)82656-3&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=1554175&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1992HL85300008&link_type=ISI) 

2.  2.Centre for Evidence-Based Medicine, University Health Network, Mount Sinai Hospital Toronto. Sensitivity & specificity (SnNouts and SpPins). [www.cebm.utoronto.ca/glossary/spsn.htm](http://www.cebm.utoronto.ca/glossary/spsn.htm) (accessed March 2004).
    
    

3.  3.Fagan TJ.Nomogram for Bayes theorem.N Engl J Med1975; 293:257.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=1143310&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1975AK17800010&link_type=ISI) 

4.  4.Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB.Evidence-based medicine. How to practice and teach EBM.New York:Churchill Livingstone,2000.
    
    

5.  5.Barry HC, Ebell MH.Test characteristics and decision rules.Endocrinol Metab Clin North Am1997; 26:45–65.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0889-8529(05)70233-4&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=9074852&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1997WN28100005&link_type=ISI) 

6.  6.Sackett DL, Haynes RB, Guyatt GH, Tugwell P.Clinical epidemiology, a basic science for clinical medicine.Boston:Little Brown,1992.
    
    

7.  7.Schwartz A, Hupert J.Medical students' application of published evidence: randomised trial.BMJ2003; 326:536–8.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjYvNzM4OC81MzYiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ1OS8yMDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

8.  8.Horvath AR, Pewsner D.Systematic reviews in laboratory medicine: principles, processes and practical considerations.Clin Chim Acta2004; 342:23–39.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/j.cccn.2003.12.015&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=15026264&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000220526400003&link_type=ISI) 

9.  9.Auleley GR, Kerboull L, Durieux P, Cosquer M, Courpied JP, Ravaud P.Validation of the Ottawa ankle rules in France: a study in the surgical emergency department of a teaching hospital.Ann Emerg Med1998; 32:14–8.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0196-0644(98)70093-9&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=9656943&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000074489600003&link_type=ISI) 

10. 10.Simel DL, Halvorsen RA Jr., Feussner JR.Quantitating bedside diagnosis: clinical evaluation of ascites.J Gen Intern Med1988; 3:423–8.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=3049966&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1988Q130700001&link_type=ISI) 

11. 11.Levin BE.The clinical significance of spontaneous pulsations of the retinal vein.Arch Neurol1978; 35:37–40.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1001/archneur.1978.00500250041009&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=619871&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1978EH10300009&link_type=ISI) 

12. 12.Varma AR, Snowden JS, Lloyd JJ, Talbot PR, Mann DM, Neary D.Evaluation of the NINCDS-ADRDA criteria in the differentiation of Alzheimer's disease and frontotemporal dementia.J Neurol Neurosurg Psychiatry1999; 66:184–8.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiam5ucCI7czo1OiJyZXNpZCI7czo4OiI2Ni8yLzE4NCI7czo0OiJhdG9tIjtzOjIyOiIvYm1qLzMyOS83NDU5LzIwOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

13. 13.Bakker AJ.Detection of microalbuminuria. Receiver operating characteristic curve analysis favors albumin-to-creatinine ratio over albumin concentration.Diabetes Care1999; 22:307–13.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZGlhY2FyZSI7czo1OiJyZXNpZCI7czo4OiIyMi8yLzMwNyI7czo0OiJhdG9tIjtzOjIyOiIvYm1qLzMyOS83NDU5LzIwOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

14. 14.Bachmann LM, Kolb E, Koller MT, Steurer J, ter Riet G.Accuracy of Ottawa ankle rules to exclude fractures of the ankle and mid-foot: systematic review.BMJ2003; 326:417.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjYvNzM4Ni80MTciO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ1OS8yMDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

15. 15.McKhann G, Drachman D, Folstein M, Katzman R, Price D, Stadlan EM.Clinical diagnosis of Alzheimer's disease: report of the NINCDS-ADRDA Work Group under the auspices of Department of Health and Human Services Task Force on Alzheimer's Disease.Neurology1984; 34:939–44.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToibmV1cm9sb2d5IjtzOjU6InJlc2lkIjtzOjg6IjM0LzcvOTM5IjtzOjQ6ImF0b20iO3M6MjI6Ii9ibWovMzI5Lzc0NTkvMjA5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. 16.Guarino JR, Guarino JC.Auscultatory percussion: a simple method to detect pleural effusion.J Gen Intern Med1994; 9:71–4.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=8164080&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1994MW50800003&link_type=ISI) 

17. 17.Bush B, Shaw S, Delbanco TL, Aronson MD.Screening for alcohol abuse using the CAGE questionnaire.Am J Med1987; 82:231–5.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/0002-9343(87)90061-1&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=2880504&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1987F979500007&link_type=ISI) 

18. 18.Marantz PR, Kaplan MC, Alderman MH.Clinical diagnosis of congestive heart failure in patients with acute dyspnea.Chest1990; 97:776–81.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1378/chest.97.4.776&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=2182296&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1990CX25300006&link_type=ISI) 

19. 19.Davie AP, Francis CM, Caruana L, Sutherland GR, McMurray JJ.Assessing diagnosis in heart failure: which features are any use?QJM1997; 90:335–9.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToicWptZWQiO3M6NToicmVzaWQiO3M6ODoiOTAvNS8zMzUiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ1OS8yMDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

20. 20.Cherpitel CJ, Clark WB.Ethnic differences in performance of screening instruments for identifying harmful drinking and alcohol dependence in the emergency room.Alcohol Clin Exp Res1995; 19:628–34.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1111/j.1530-0277.1995.tb01559.x&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=7573785&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1995RE22800016&link_type=ISI) 

21. 21.Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J.Sources of variation and bias in studies of diagnostic accuracy: a systematic review.Ann Intern Med2004; 140:189–202.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-140-3-200402030-00010&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=14757617&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000188704500005&link_type=ISI) 

22. 22.Centre for Evidence Based Medicine, University of Oxford. SpPins and SnNouts. [www.cebm.net/sppins\_snnouts.asp](http://www.cebm.net/sppins_snnouts.asp) (accessed January 2003).
    
    

23. 23.Centro per la formazione permanente e l'aggiornamento del personale del servizio santiario (CEFPAS), Caltanissetta, Italy. [http://www.cefpas.it/ebm/diagnosis/dia_appraising.htm](http://www.cefpas.it/ebm/diagnosis/dia_appraising.htm) (accessed March 2004).
    
    

24. 24.National Electronic Library for Health.Full-text guideline collection. Clinical guidelines for type 2 diabetes: renal disease—prevention and early management. [www.nelh.nhs.uk/guidelinesdb/html/fulltextsummary/RcgpRenal.html](http://www.nelh.nhs.uk/guidelinesdb/html/fulltextsummary/RcgpRenal.html) (accessed March 2004).
    
    

25. 25.Williams JW Jr., Simel DL.The rational clinical examination. Does this patient have ascites? How to divine fluid in the abdomen.JAMA1992; 267:2645–8.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1001/jama.1992.03480190087038&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=1573754&link_type=MED&atom=%2Fbmj%2F329%2F7459%2F209.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1992HT82300034&link_type=ISI) 

26. 26.Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al.Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.BMJ2003; 326:41–4.
    
    [FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjYvNzM3OS80MS9hIjtzOjQ6ImF0b20iO3M6MjI6Ii9ibWovMzI5Lzc0NTkvMjA5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)

 [1]: /embed/graphic-1.gif