BMJ  2004;328:924-925 (17 April), doi:10.1136/bmj.38007.694745.F7 (published 23 January 2004)

Paper

Hospitals' star ratings and clinical outcomes: ecological study

¹ Intensive Care National Audit and Research Centre, London WC1H 9HR, ² London School of Hygiene and Tropical Medicine, London WC1E 7HT

Introduction

The English Department of Health is developing global measures of the performance of all NHS bodies, including 166 acute hospital trusts. Since 2000-1, the trusts get zero, one, two, or three stars to indicate performance.¹ This rating may not reflect the effectiveness of clinical care measured in patient outcomes because of the lack of accurate routine data.² One exception is in adult critical care³; we checked whether a hospital's rating provided an indication of its clinical outcomes.

Methods and results

We compared the 2001-2 rating of 102 acute hospital trusts for which we had validated data for that year. We calculated each patient's predicted risk of death before discharge from hospital⁴ and compared it with actual mortality for all admissions in 2001-2 for each unit.

We compared rating with crude mortality at the patient level rather than aggregated by hospital; our sample of hospitals with all hospitals; and university with non-university hospitals using ² tests for trend. We compared rating with size of intensive care unit and mean age of patients, using Spearman's . We calculated confidence intervals for mortality adjusted for risk, using logistic regression of mortality on rating and predicted log odds of mortality. We tested rating and adjusted mortality using the likelihood ratio test.

The distribution of ratings for the 102 acute hospital trusts was similar to that for all 166 trusts (² = 1.7; P = 0.19). Rating was associated with teaching status (university hospitals had more stars than non-university hospitals—52% v 29% had three, 38% v 45% had two, 5% v 19% had one, 5% v 7% had zero; ² = 3.9; P = 0.05) but not size of its critical care unit (Spearman's = 0.09; P = 0.34).

Rating and crude mortality for critical care admissions were significantly associated (² = 4.1; df = 1; P = 0.04) (figure): mortality in trusts with three stars was about 4% lower than in trusts with zero stars. However, case mix of critical care admissions also differed considerably. Rating was inversely associated with the mean age of critical care admissions ( = -0.19; P = 0.04). The association between rating and hospital mortality was no longer significant when case mix differences were taken into account (P = 0.4) (figure).

View larger version (20K): [in this window] [in a new window]   Odds ratio for crude case mix and for case mix adjusted for risk hospital mortality by star rating of acute hospital trust

 

Comment

For adult critical care, star ratings do not reflect the quality of clinical care provided by hospitals. Patients do just as well in a trust with no stars as they do in one with three stars. Crude mortality data are misleading because they ignore the fact that higher rated trusts tend to be teaching institutions with patients who are less severely ill on admission to critical care units.

We did not expect to find an association between the rating of the whole trust and the effectiveness of critical care. Firstly, hospitals are complex organisations containing many services; performance across a hospital will not be uniform—a poorly rated hospital may contain some excellent services and vice versa. Secondly, ratings are determined by a small number of process measures; outcome measures play only a small role and are based on scant poor quality data, which do not adequately account for case mix.

The study's principal limitation is its confinement to one small, though important, group of patients and services. Our findings may be atypical, and trusts' ratings may reflect outcomes elsewhere in hospital care.

If these findings reflect other areas of hospital care, the government is not yet fulfilling its "commitment to provide patients and the general public with comprehensive, easily understandable information on the performance of their local health services."¹ Outcome ought to be a principal concern alongside process indicators, such as waiting times and cleanliness; to fulfil its aim, the government needs to use specialised clinical databases (accessible through www.docdat.org).⁵

---

This article was posted on bmj.com on 23 January 2004: http://bmj.com/cgi/doi/10.1136/bmj.38007.694745.F7

We thank the participating intensive care units and Minesh Tailor at the Intensive Care National Audit and Research Centre.

Contributors: KR and NB devised the study; all authors designed the study and interpreted the data; and NB, KR, and DH wrote the paper. NB is guarantor.

Funding: None.

Competing interests: None declared.

Ethical approval: Not needed.

References

1. Department of Health. NHS performance ratings: acute trusts, specialist trusts, ambulance trusts, mental health trusts 2001/02. London: DoH, 2002.
2. Gill K, Black SA, Clarke P, Corbett CRR. Misleading mortality data in league tables. Ann R Coll Surg Engl 2003;85(suppl): S244-7.[CrossRef]
3. Rowan K, Black N. A bottom-up approach to performance indicators through clinician networks. Health Care UK 2000 Spring: 42-6.
4. Rowan KM. Outcome comparisons of intensive care units in Great Britain and Ireland using the APACHE II method [PhD thesis]. Oxford: University of Oxford, 1993.
5. Black N, Payne M. Directory of clinical databases: improving and promoting their use. Qual Saf Health Care 2003;12: 348-52.[Abstract/Free Full Text]

(Accepted 12 November 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Article

Be sceptical about the star system
BMJ 2004 328: 0. [Full Text]

This article has been cited by other articles:

• Stevens, P., Stokes, L., O'Mahony, M. (2006). Metrics, Targets and Performance. National Institute Economic Review 197: 80-92 [Abstract]  

Rapid Responses:

Read all Rapid Responses

Case-mix adjustment and transferred patients

Jon H Laake
bmj.com, 18 Apr 2004 [Full text]

Clinical databases are required to assess hospital performance

Brian T Collopy
bmj.com, 19 Apr 2004 [Full text]

Hospitals’ star ratings and clinical outcomes: comparing apples with apples

Roy Ilan, et al.
bmj.com, 22 Apr 2004 [Full text]