Education and debate

Fortnightly Review: How to ensure that guidelines are effective

^a Department of Epidemiology and Public Health, School of Health Care Sciences, Medical School, Newcastle University, Newcastle upon Tyne NE2 4HH, ^b North Tyne Health, Newcastle upon Tyne, ^c South of Tyne Health Commission, South Shields, Tyne and Wear

Correspondence to: Dr Thomson.

The assessment of a health practice and the development of a policy for the use of that practice is the final step in a long process that begins with a research idea and ends in actual changes in people's health. All the accumulated research, development, and experience is converted into practical recommendations that largely determine what happens to a patient. In this pivotal position the design of a policy deserves whatever effort is required to ensure that all the proceeding work is put to the best effect¹

There is an explosion of interest in guidelines, reflected in a vivid debate ranging from "the best thing since sliced bread" to cries of "cookbook medicine" and fears of constraint on clinical freedom.^{2 3 4 5 6 7 8} Furthermore, the Department of Health has introduced an initiative to encourage the adoption of evidence based guidelines within purchaser-provider contracts,^{9 10} and a review of the effectiveness of clinical guidelines has recently been published.¹¹

The subject of guidelines is complex. Even for enthusiasts, ensuring that guidelines are effective by addressing each part of the complex chain of development, dissemination, implementation, and evaluation (fig 1) can be a daunting task. There are an increasing number of publications on the subject--for example, on the details of guideline development and effectiveness.^{1 12 13 14 15 16} In particular, the seminal work of Grimshaw and Russell has led to a much quoted taxonomy of the factors that influence the effectiveness of guidelines.¹⁵ We have drawn together the key messages from this literature and represented them in the form of a series of reflective questions to guide readers through this complex but important maze.

View larger version (47K): [in this window] [in a new window]   FIG 1--Chain of events to produce effective guidelines

Why guidelines? What are they?

Before addressing the practicalities, it is worth reminding ourselves of the reasons for the prominence of guidelines. These include an emphasis on audit and improving the quality of health care; medical advances and increasingly complex clinical decision making; unexplained variations in clinical practice; heightened public awareness of, and participation in, decision making; and a more explicit debate about the use of limited resources.

Chambers' English Dictionary defines a guideline as "an indicator of a course that should be followed, or of what future policy will be."¹⁷ In clinical work several terms have been used, including guidelines, practice policies, clinical policies, practice parameters, protocols, and algorithms.

Perhaps the clearest definition is that of the Institute of Medicine, guidelines being "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances."¹⁸ Guidelines also have a role in supporting quality assurance and audit, including providing the framework against which care can be evaluated.

Regardless of these differing definitions, there are common elements. Guidelines can:

* Help patients and professionals to make decisions about health care

* Describe appropriate care based on the scientific evidence and broad consensus, leaving room for justifiable variations in practice

* Focus on specific circumstances while taking into account organisational factors, community characteristics, and other influences on health care delivery

* Act as a focus for quality assessment and improvement activity, including audit.

Development of guidelines

The first step in developing guidelines is choosing the topic or subject (box 1). Choosing a topic simply because there is a desire for guidelines may be inappropriate--the desire may reflect uncertainty due to a lack of empirical evidence to guide practice. If so, no amount of enthusiasm for guidelines will compensate for the lack of evidence. Thus, it may be advisable to start with a topic with a broad consensus of opinion or for which guidelines are already available from another source (see below).

Box 1--Key questions on choice of topic
* Is the topic high volume, high risk, high cost?
* Are there large or unexplained variations in
practice?
* Is the topic important in terms of the process and
outcome of patient care?
* Is there potential for improvement?
* Is the investment of time and money likely to be
repaid?
* Is the topic likely to hold the interest of team
members?
* Is consensus likely?
* Will change benefit patients?
* Can change be implemented?

The development of practice guidelines cannot be successfully undertaken in isolation: it requires a team effort (box 2).¹⁹ The effectiveness of guidelines in improving patient care depends not only on the guidelines themselves but also on the nature of the group producing them.²⁰ For example, whether the guidelines have credibility with a peer group of professionals or whether the guidelines gain the support of those who will be responsible for implementing them may critically depend on who is seen to have been active in their development. Moreover, it is important when developing guidelines to consider gaining the participation of those with the power and authority to implement them or to persuade others so to do.

Box 2--Key questions about guidelines development group

Membership of group                      * Who will be expected to implement or
* Do we have the right people in the     evaluate the guidelines produced? Have
group?                                   we included them or their representatives?
  Doctors?                               Working of the group
  Primary and secondary care?            * Are we clear about the task for this
  Nurses?                                group?
  Therapists?
  Managers?                              * Can we estimate how long we will need
  Purchasers?                            to develop and implement guidelines?
  Patients or patients' representatives? * How long will we need to spend on
  Researchers?                           each stage?
* Can we involve other appropriate         Development? (including review of
people by circulation of drafts or some  potential sources of others' guidelines)
other form of consultation?                Dissemination?
                                           Implementation?
* Do we have appropriate skills?           Review and evaluation?
  Group leadership?                      * Are members of the group prepared to
  Group facilitation?                    commit sufficient time to the process?
  Knowledge of guidelines?               * How many meetings will we need?
  Specialist knowledge of the subject?   * What length of meeting?
  Skills in setting standards?           * What is our proposed timescale?
  Administrative support?
  Skills in reviewing and analysing the  * Do we need to start from scratch or is
literature?                              it a matter of adopting or adapting others'
                                         guidelines?
* Are there too many or too few of us?

The group will need skills in the relevant clinical subject, including specialist experience, to ensure that the guidelines produced have professional credibility. The group may need skills in conducting a literature review and in collecting and analysing data. Above all, the group needs a leader with the organisational and communication skills required to take them through the process of guidelines development and on to the stages of implementation and review.

Some members of the group may be unwilling to commit themselves to a process such as this without some idea of the time entailed. At this stage it is well worth taking the trouble to outline a plan for the process, deciding how often to meet and for how long. The plan should include specific objectives. You will then be in a position to estimate the time it will take to complete the process. Don't underestimate the time and effort needed for implementation and evaluation. Give these stages of the process at least the same, if not more, time than you give to the development stage.

Methods

of guideline development

Before starting guidelines development, explore what has already been done elsewhere. It will be unusual to find a subject in which no work on guidelines development has been undertaken. Seek already available guidelines. Review the literature. Talk to colleagues elsewhere. Consider talking to your royal college's audit office or to specialty led clinicians in your hospital, practice, or region. Even if your search identifies only poorly developed guidelines or failed attempts at development, that knowledge will give you a better start than staring at a blank piece of paper and hoping for inspiration, even if only to identify what not to do.

ADOPTION, ADAPTATION, ASSESSMENT

You may have been able to identify guidelines that have already been developed in your chosen subject. If so, you need to decide whether the guidelines can be adopted unchanged or need to be adapted for local use. Adapting guidelines may entail taking broad based guidelines and turning them into locally applicable guidelines or protocols.¹² Although ownership is more readily achieved among those who have participated in development, this needs to be weighed up against the benefits in adopting or adapting guidelines already available. Adapting available guidelines to local circumstances may be sufficient to stimulate ownership.

It is important to have some means of assessing the quality of guidelines that have already been produced. For example, the Agency for Health Care Policy Research in the United States has outlined key factors to be considered.²¹ Others have suggested alternative or complementary strategies,¹⁵ which help to identify guidelines that have undergone an appropriate and rigorous process of development. However, these frameworks for assessing guidelines may be statements of the ideal. Such thorough appraisal of the quality of guidelines may be both difficult (because of the lack of documentation on the process of development of available guidelines) and fruitless at present (as most guidelines may not have gone through such a rigorous process of development).²²

STARTING FROM SCRATCH

When reference guidelines are not already available you may need to develop them yourselves. There is no single superior method of developing effective guidelines. The method selected will depend on the topic chosen, the experience of the group, the purpose of the guidelines, and the evidence available on the outcomes of clinical practice (box 3). Although ideal approaches have been proposed,¹ the application of these strategies are far from ideal, not least because of limited data on health outcomes in many areas of practice. In the meantime, most current examples of guidelines development use a variable combination of "expert" judgment and literature review.

Box 3--Key questions on development and presentation of
guidelines

* Have we clearly defined the subject of  * Have we clearly defined the end users
the guidelines?                           of the guidelines?

  Patient group?                          * Is the problem or issue clearly defined?
  Diagnosis?                              * Is our presentation clear and unambiguous?
  Investigation?                          Do we need to test its clarity?
  Treatment or intervention?
  Administration?                         * Is the evidence for our statements
                                          presented or readily available?
* Do we have access to guidelines         * Is there a rationale for the guidelines?
already produced on this subject?           What is the estimated impact or
* Have we approached likely sources of    outcome?
guidance or advice?                         What are the current variations and
* Do we need to undertake a literature     uncertainties?
review?                                   * If options are available are the decision
  Computerised bibliographies--for        making factors clear?
example, Medline on CD ROM                * How should we present them? Which
  Medical library                         format will be most appropriate for their
  Do not forget the "grey" literature     proposed use?
* Is the evidence for defining components   Booklet?
of the guidelines available to those        Newsletter?
responsible for their construction?         Pocket card?
* How shall we take this forward?           Posters?
  Nominal group technique?                  Publication in professional journal?
  Delphi technique?                         Computer based?
  Consensus conference?                     Other?
  Evidence based model?                   * Will the chosen medium be sufficiently
* What type of guidelines do we want to   durable for its proposed use?
produce?                                  * Are the authors acknowledge or
                                          stated?

The views of a group of experts may be coalesced through a consensus development process.^{23 24 25} This method is potentially the most straightforward way of developing guidelines but is limited by the knowledge and experience of the group members, both in the subject and in the various stages of guidelines development. It can be enhanced by preparatory collation, dissemination, and summary of existing research evidence, preferably by a formal structured review of the literature.

Achieving consensus requires some compromise and is not easy. Too often in groups the loudest or most authoritative members have their way, leading to a lack of commitment from the others within the group. However, there are recognised methods of getting the best out of a group such as the nominal group technique²⁶ and the Delphi technique.²⁷

Alternatively, the process of development can be more formal and systematic, and entail using thorough review and analysis of the available literature to produce guidelines based on the evidence of high quality studies.^{1 21} In this approach, such as the model developed by the Agency for Health Care Policy and Research in the United States, the development group or team is constructed so as to share the load inherent in detailed and systematic literature review and analysis of both clinical evidence and evidence of cost effectiveness. This approach, however, is both costly and time consuming.

There are trade offs between these approaches and limited evidence on which are more effective or cost effective. Local and consensus groups may lack the skills and resources to undertake a synthesis of the evidence in the literature. In this case the validity of the guidelines produced may be compromised, but it is conceivable that the enhanced local ownership, and its effects on implementation, may compensate for this. This approach emphasises the potential for local adaptation of systematically developed national guidelines and is worthy of further research.

What will the product look like?

There is no single format for presenting guidelines. Several presentations can be used, according to such factors as the target group, the intended use and user, and the topic chosen. A combination of text, algorithms, and option lists can all be used so long as the end result is explicit, logical, and unambiguous. Preferably, a rationale should accompany the guidelines, describing the evidence considered by the group and the way in which it came to the decisions contained within the guidelines. The approach of the Agency for Health Care Policy and Research produces summary guidelines, which are supported by detailed manuals containing the evidence collected and synthesised and the rationale behind the statements, but this thoroughness tends to lead to bulky publications.^{21 28}

When the evidence for the statements is strong and professional judgment almost unanimous, this should be reflected in the way the guidelines are expressed. Some components may be mandatory statements, where judgment is not required and variation in decision making is not clinically justified. An example would be the need for admission for observation of a patient with skull fracture after head injury.

The questions in box 3 will help but should be raised at the planning stage.

Dissemination and implementation of guidelines

Dissemination implies bringing the guidelines to the attention of the intended users (box 4). One of the reasons why guidelines fail to bring about change is that they do not reach the intended users. For example, an audit of the distribution of management guidelines on head injury in the Northern region showed that only 63% of the relevant senior house officers had received copies of the guidelines.²⁹

Box 4--Key questions for dissemination
of guidelines

* Are we clear about the target users of our guidelines?

* How can we best ensure we reach this audience?

* What means do we have of reaching the target
audience?
  Professional bodies?
  Specialty representative groups?
  Audit networks? Local clinical audit committees or
  medical audit advisory groups?
  Personnel departments and induction packs?
  Unit management boards?
  District health authorities or family health services
  authorities
  Local medical committees?
  Undergraduate and postgraduate lectures?
  Conferences and seminars?
  Newsletters?
  Other?

* In what form should we publish and disseminate
the guidelines?
  In junior doctors' handbooks?
  On posters in the relevant departments and wards?
  In newsletters or professional news sheets?
  As local reports or documents?
  In a journal?
  In patient literature?
  Several of the above?
  Other?

* Do we need a system of regular dissemination--for
example, every six months to junior doctors?

* How will we monitor or evaluate dissemination?

* What will dissemination cost?

In this process the opinions of potential users of the guidelines may be helpful. A recent survey of general practitioners in South Tees found that publication in a medical journal was considered to be the least preferred option of disseminating guidelines on referral of patients to orthopaedic outpatient clinics.³⁰ General practitioners preferred having guidelines posted to them and being able to discuss them in specifically arranged seminars. These opinions are consistent with the research evidence on guidelines, which shows that publication in a journal is unlikely to support effective implementation.¹⁵

Adequate dissemination is necessary for guidelines to be used, but implementation requires strategies to facilitate changes in behaviour (box 5). This can be done in several ways.

Box 5--Key questions on implementing
guidelines

* What means can we use to support implementation?

* Can medical records or forms be designed to
support the guidelines?

* Can we provide individual patient feedback?
    Computerised record with forced choice?

* Is there a means of retrospective feedback?
    From audit results or case note review?

* Should we arrange an educational presentation?
    Seminars, lectures, regular discussion and
    debate?

* Are key opinion formers promoting our guidelines?

* What are the incentives to implement the guidelines?

The research evidence is that the greater the educational component of dissemination the greater the likelihood of adoption into practice.¹⁵ Thus, specific educational interventions--for example, seminars concentrating on the particular guidelines that have been targeted at the end users--are more likely to be effective than including guidelines within the general continuing medical education programme. Simply posting the guidelines or publishing them in professional journals is less valuable in reaching the target audience or in stimulating adoption. Furthermore, the promotion and endorsement by peers, particularly by respected local clinicians, can enhance adoption.

Reminders may support implementation, but the form and timing of reminders may be crucial. They range from patient specific reminders at the time of consultation to general non-specific reminders. Patient specific reminders at the time of consultation could include, for example, attachment of guidelines to the medical record,³¹ inclusion of guidelines on a desk top computer,^{32 33} or specially designed clinical records.³⁴ The key factor is that the guidelines are brought to the clinician's attention at the time of the consultation and are thus able to influence the clinical decision prospectively.

The use of guidelines in audit (see below) can increase the likelihood of adoption. Comparative feedback on performance may be a valuable stimulus to change, preferably if linked to the cyclical process of audit with implementation of change and subsequent review.

A review of these various strategies has shown that any of them or combinations of them work to a degree and for varying periods. Strategies specific to individual patients are more likely to encourage the adoption of guidelines and bring about a change in patient care.¹⁵

Purchasers and contracting

An important potential lever to support the implementation of guidelines is the purchaser-provider interaction and contracting.⁹ Purchasers have a legitimate interest in influencing the quality and delivery of care through contracts for services and increasingly through their influence on clinical audit,³⁵ which has been enhanced recently by the transfer of audit funds to purchasers. However, there are problems with this approach that relate to the degree of skills within purchasing organisations, the quality of the interaction with providers, and the methods by which purchasers seek to influence their providers.^{36 37}

Thus, although the opportunities to develop purchasing based on guidelines exist, they may not be as extensive as some have suggested. At the very least, both purchasers and providers need to be cognisant of the factors discussed here that are likely to influence the effectiveness of guidelines. Thus, the imposition of guidelines by purchasers is far less likely to be effective than guidelines that have been developed, adopted, or adapted as a result of shared priority setting and review.

Evaluation

The application of guidelines should produce improved quality of care for patients. However, concentrating on an evaluation of patient outcomes alone as a measure of the success of guidelines is insufficient and may be impractical, given the difficulties of interpreting data on the small numbers of patients in a locality. Thus, the various components of the whole process--that is, development, dissemination, implementation, evaluation, and review--have a part to play (box 6). For example, patient outcome may be unchanged because the target audience has not received the guidelines, because it has not read them, because it has forgotten them, and so on. Without an evaluation of each step in the chain (fig 1), inability to show effectiveness may leave people none the wiser about which link(s) in the chain may have been inadequate.

Box 6--Key questions on evaluation and revision of guidelines
Evaluation                                  What is the expected outcome and how
* How will we know if the guidelines are  can it be measured?
  Received?                               * Who should assess the guidelines?
  Read?                                     Lead clinicians in local audit?
  Respected?                                Managers?
  Used?                                     External organisations--for example,
  Locally evaluated?                      purchasers?
  Locally promoted or endorsed?             Professional bodies?
* What methods are required to assess       Regional audit structures?
each of the above?                          Other?
  Surveys?                                Review and revision
  Questionnaires?                         * How often should the guidelines be
  Case note reviews?                      reviewed or reformulated? What is the
  Cyclical criterion based audit?         likelihood of significant advances in
  Routine monitoring?                     knowledge occurring?
* Do we need advice on methodology?       * What local or wider organisational
* What gaps in knowledge have been        changes will have an effect on the
identified through evaluation?            applicability of the guidelines?
* How will we feed back the results of    * How will their relevance be
evaluation to those responsible for       maintained?
implementation?                           * Who will be responsible for initiating
* How will appropriate changes in each    review? Who will be needed to undertake
step of the chain be identified and       a review?
implemented?                              * Has evaluation suggested need for
* Is there a clear means of evaluation?   review?
  Can explicit standards be assessed?     * How will reviewed guidelines be
  Can locally agreed standards arising    disseminated to replace redundant
from the guidelines be defined?           versions?
  Can key indicators be selected to give a
measure of implementation?

Working within the developing multidisciplinary clinical audit programmes becomes important at this point.³⁸ Audit groups will enhance ownership and enable the dissemination and implementation process to be intimately linked to audit programmes. Furthermore, consideration of the role of audit at an early stage will help to identify which elements of the guidelines are amenable to standard setting and hence be the basis of criterion based audit.³⁹ Evaluation can take the form of comparative evaluation ("Have we improved?") or absolute evaluation ("Have we achieved a defined standard?").

Moreover, the audit process may be valuable not only in assessing whether the guidelines have been adhered to (an audit of the process of care) but also in assessing the effectiveness of other links in the chain. Such an approach is applicable to each component of guideline development, dissemination, implementation, evaluation, and review. A standard for dissemination, for example, could be that 100% of the target audience should receive the guidelines. It may not be possible to evaluate all components, but identification of key standards or indicators should be considered.

Updating guidelines

One of the potential criticisms of guidelines is that they could stifle change and innovation.³ This concern can be dealt with by periodic review of the guidelines. Review may be necessary because of advances in medical knowledge, changes in medical practice, changes in local circumstances, or the results of evaluation.

Review is necessary to prevent outdated processes being applied and to incorporate recent research findings. The frequency and detail of review will depend on the subject of the guidelines, including the fluidity of knowledge in the field and local changes in delivery or organisation of services that may require a reappraisal. When planning a scheduled review for the guidelines, any changes will need to be implemented with the same commitment as the original guidelines (box 6).

In addition to the questions specific to each area above there are a series of general questions that will support the development and implementation of guidelines (box 7).

Box 7--Key questions on sources of
support

* General
Is there access to developed or published guidelines?
Are there sources of advice and support for guideline
development?
Are these international, foreign, national, regional, or
local?

* International
Are there international organisations or specialty
bodies with guideline experience?
  Can they help us?

* National or regional
Have you approached your royal college or faculty
audit office or equivalent?
Have you approached the National Clinical Audit
Information and Dissemination Centre?
Have you approached the Eli Lilly National Medical
Audit Centre?
Are there other national or regional specialty bodies
worth exploring?
Are there active national or regional
patient organisations worth approaching?
Can a member of the regional clinical audit committee
advise?
Is there a local university department with skills--for
example, a department of epidemiology and public
health?
Is there a national or regional database or compendium
of good practice in audit or guidelines?
Can the regional college or faculty adviser help?
Is there a regional specialty led clinician and audit
group?
  Can it help?

* Local
Can the postgraduate or medical library help with
  A literature search?
  Audit and guideline publications?
  Specialist and other journals?
Is there a local audit office with information or to
provide support?
Are there accessible local audit support staff?
Can the local clinical audit committee or medical audit
advisory group help?
Is there a local nursing and therapy audit group?

Conclusion

s

McKeown states: "Medicine must be prepared to face the tests which are inescapable in private enterprise and which it is almost unique among public activities in having evaded hitherto: Is our work well done? Is it worth doing? and Does it pay its way"⁴⁰

The process that leads from selecting a topic to having an impact on patient care is complex and full of potential pitfalls. None the less, there is a growing body of research evidence on appropriate approaches. We have reviewed this literature and produced a series of reflective questions to help incorporate this knowledge into practice. We believe that the potential for guidelines to influence the quality of patient care is considerable, particularly when aligned with the process of audit and quality assurance, in the setting of the purchaser-provider interaction, and within an overall culture of continuous quality improvement (fig 2).³⁷

View larger version (31K): [in this window] [in a new window]   FIG 2--Guidelines in context

This review arose from work undertaken on behalf of the former Northern Regional Health Authority as part of its strategy to support wider use of guidelines. We produced a guidelines resource pack for a regional guidelines conference; the pack includes overhead masters for the boxes and figures in this article and is available from us on request. We thank the members of the Regional Guidelines Group, particularly Dr Pali Hungin, for their valuable comments.

1. Eddy DM. Assessing health practices and designing practice policies; the explicit approach. Philadelphia: American College of Physicians, 1992.
2. Charlton JRH, Hartley RM, Silver R, Holland WW. Geographical variation in mortality from conditions amenable to medical intervention in England and Wales. Lancet 1983;i:691-6.
3. Charlton BG. Management of science. Lancet 1993;342:99-100. [Medline]
4. Delamothe T. Wanted: guidelines that doctors will follow. BMJ 1993;307:218.
5. Haines A, Feder G. Guidance on guidelines. BMJ 1992;305:785-6.
6. Hurwitz B. Clinical guidelines: proliferation and medicolegal significance. Quality in Health Care 1994;3:37-44. [Abstract]
7. Guidelines for doctors in the new world [editorial]. Lancet 1992;339:1197-8. [Medline]
8. Madhok R, Ohnsorg F, Liberati A, Ballard DJ. Practice guidelines. In: Frymoyer JW, ed. Orthopaedic knowledge update 4. Illinois: American Academy of Orthopaedic Surgeons, 1993:35-44.
9. Department of Health. Improving clinical effectiveness. London: HMSO, 1993. (EL(93)115.)
10. Department of Health. Improving the effectiveness of the NHS London: HMSO, 1994. (EL(94)74.)
11. Effective Health Care. Implementing clinical practice guidelines. Leeds: Institute for Health, University of Leeds, 1994.
12. Clinical Resource and Audit Group. Clinical guidelines: report of the working group. Edinburgh: Scottish Office, 1993.
13. Grimshaw J, Russell I. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342:1317-22. [Medline]
14. Grimshaw J, Russell I. Achieving health gain through clinical guidelines. I. Developing scientifically valid guidelines. Quality in Health Care 1993;2:243-8. [Medline]
15. Grimshaw JN, Russell IT. Achieving health gain through clinical guidelines. II. Ensuring that guidelines change medical practice. Quality in Health Care 1994;3:45-52. [Medline]
16. Russell I, Grimshaw J. Effectiveness of referral guidelines: a review of the methods and findings of published evaluations. In: Roland M, Coulter A, ed. Hospital referrals. Oxford: Oxford University Press, 1992:179-211.
17. The Chambers dictionary. Edinburgh: Chambers Harrap, 1993.
18. Field MJ, Lohr KN, eds. Guidelines for clinical practice: from development to use. Washington, DC: National Academy Press, 1992.
19. McNichol M, Layton A, Morgan G. Team working: the key to implementing guidelines? Quality in Health Care 1993;2:215-6.
20. Thomas S. Standard setting in the Netherlands: impact of the human factor on guideline development. Br J Gen Pract 1994;44:242-3. [Medline]
21. Field MJ, Lohr KN. Clinical practice guidelines: directions for a new programme. Washington, DC: National Academy Press, 1990.
22. Cluzeau F, Littlejohns P, Grimshaw JM. Appraising clinical guidelines: towards a "Which" guide for purchasers. Quality in Health Care 1994;3:121-2. [Medline]
23. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. International Journal of Technology Assessment in Health Care 1986;2:53-63. [Medline]
24. Hicks N. Some observations on attempts to measure appropriateness of care. BMJ 1994;309:730-3. [Free Full Text]
25. Fink A, Kosecoff J, Chassin M. Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74:979-83. [Abstract/Free Full Text]
26. Delbecq AL, Van Der Ven AH, Gustafson DH. Techniques for programme planning: a guide to nominal group and Delphi processes. Illinois: Scott Foreman, 1975.
27. Linstone HA, Turoff M, eds. The Delphi method: techniques and applications. Reading, MA: Addison-Wesley, 1975.
28. Hadorn DC, McCormick K, Diokno A. An annotated algorithm approach to clinical guideline development. JAMA 1992;267:3311-4. [Abstract]
29. Madhok R, Thomson RG, Mordue A, Mendelow AD, Barker J. An audit of distribution and use of guidelines for the management of head injury. Quality in Health Care 1993;2:27-30. [Abstract]
30. Madhok R, Green S. Orthopaedic referral guidelines: experience in an English health district. International Journal for Quality in Health Care 1994;6:73-6. [Abstract/Free Full Text]
31. Packer GJ, Goring CC, Gayner AD, Craxford AD. Audit of ankle injuries in an accident and emergency department. BMJ 1991;302:885-7.
32. Brownbridge G, Evans A, Fitter M, Platts M. An interactive computerised protocol for the management of hypertension: effects on general practitioners' clinical behaviour. Journal of the Royal College of General Practitioners 1986;36:198-202.
33. Lilford RJ, Kelly M, Baines A, Cameron S, Cave M, Guthrie K, et al. Effects of using protocols on medical care: randomised trial of three methods of taking an antenatal history. BMJ 1992;305:1181-4.
34. Maclaren RE, Ghoorahoo HI, Kirby NG. Skull x-ray after head injury: the recommendations of the Royal College of Surgeons Working Party Report in Practice. Arch Emer Med 1993;10:138-44.
35. Thomson RG, Cook G, Lelliot P, Baker I, Godwin R. Audit and the purchaserprovider interaction. London: Department of Health, 1993. (EL(93)34. Annex B.)
36. McKee M, Clarke A. Guidelines, enthusiasms, uncertainty, and the limits of purchasing. BMJ 1995;310:101-4. [Free Full Text]
37. Thomson RG. The purchaser role in provider quality: lessons from the United States. Quality in Health Care 1994;3:65-6. [Medline]
38. Department of Health. Clinical audit: meeting and improving standards in health care. London: HMSO, 1993.
39. Shaw CD. Criterion based audit. BMJ 1990;300:649-51.
40. McKeown T. Medicine: dream, mirage, or nemesis. Oxford: Blackwell, 1979.

Related Articles

Ensuring that guidelines are effective

Charlotte Williamson
BMJ 1995 311: 1023. [Extract] [Full Text]

Pilot of patient held guidelines is under way

Jammi N Rao, P Myers, B S Smith, and B A Andreou
BMJ 1995 311: 1023. [Extract] [Full Text]

This article has been cited by other articles:

• Jones, N. E., Suurdt, J., Ouelette-Kuntz, H., Heyland, D. K. (2007). Implementation of the Canadian Clinical Practice Guidelines for Nutrition Support: A Multiple Case Study of Barriers and Enablers. Nutr Clin Pract 22: 449-457 [Abstract] [Full text]  
• Scott, I. (2007). The evolving science of translating research evidence into clinical practice. Evid. Based Med. 12: 4-7 [Full text]  
• Erhardt, L. R. (2005). Managing cardiovascular risk: reality vs. perception. Eur Heart J Suppl 7: L11-L15 [Abstract] [Full text]  
• Gardner, M. J., Brophy, R. H., Demetrakopoulos, D., Koob, J., Hong, R., Rana, A., Lin, J. T., Lane, J. M. (2005). Interventions to Improve Osteoporosis Treatment Following Hip Fracture. A Prospective, Randomized Trial. JBJS 87: 3-7 [Abstract] [Full text]  
• Burgers, J.S., Fervers, B., Haugh, M., Brouwers, M., Browman, G., Philip, T., Cluzeau, F.A. (2004). International Assessment of the Quality of Clinical Practice Guidelines in Oncology Using the Appraisal of Guidelines and Research and Evaluation Instrument. JCO 22: 2000-2007 [Abstract] [Full text]  
• Keeley, P. W (2003). Clinical guidelines. Palliat Med 17: 368-374 [Abstract]  
• Degeling, P., Maxwell, S., Kennedy, J., Coyle, B. (2003). Medicine, management, and modernisation: a "danse macabre"?. BMJ 326: 649-652 [Full text]  
• (2003). Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 12: 18-23 [Abstract] [Full text]  
• Brown, E. M. (2002). Guidelines for antibiotic usage in hospitals. J Antimicrob Chemother 49: 587-592 [Full text]  
• Anthony, D., Brooks, N. (2001). Clinical guidelines in community hospitals. Journal of Research in Nursing 6: 839-851 [Abstract]  
• Durieux, P., Gaillac, B., Giraudeau, B., Doumenc, M., Ravaud, P. (2000). Despite Financial Penalties, French Physicians' Knowledge of Regulatory Practice Guidelines Is Poor. Arch Fam Med 9: 414-418 [Abstract] [Full text]  
• Kerry, S., Oakeshott, P., Dundas, D., Williams, J. (2000). Influence of postal distribution of The Royal College of Radiologists' guidelines, together with feedback on radiological referral rates, on X-ray referrals from general practice: a randomized controlled trial. Fam Pract 17: 46-52 [Abstract] [Full text]  
• Cookson, B. (2000). Infection and antimicrobial prescribing control in the new millennium: nightmare or nirvana?. J. Clin. Pathol. 53: 66-70 [Full text]  
• Haycox, A., Bagust, A., Walley, T. (1999). Clinical guidelines---the hidden costs. BMJ 318: 391-393 [Full text]  
• Unwin, N., Thomson, R., O'Byrne, A. M., Laker, M., Armstrong, H. (1998). Implications of applying widely accepted cholesterol screening and management guidelines to a British adult population: cross sectional study of cardiovascular disease and risk factors. BMJ 317: 1125-1130 [Abstract] [Full text]  
• Kitchiner, D., Bundred, P. (1998). Integrated care pathways increase use of guidelines. BMJ 317: 147b-147 [Full text]  
• Thomson, R., McElroy, H., Sudlow, M. (1998). Guidelines on anticoagulant treatment in atrial fibrillation in Great Britain: variation in content and implications for treatment. BMJ 316: 509-513 [Abstract] [Full text]  
• Campbell, H., Hotchkiss, R., Bradshaw, N., Porteous, M. (1998). Integrated care pathways. BMJ 316: 133-137 [Full text]  
• Sibbald, W. J., Kossuth, J. D. (1998). The Ontario Health Care Evaluation Network and the Critical Care Research Network as Vehicles for Research Transfer. Med Decis Making 18: 9-16 [Abstract]  
• Williamson, C. (1995). Ensuring that guidelines are effective. BMJ 311: 1023-1023 [Full text]  
• Rao, J. N, Myers, P, Smith, B S, Andreou, B A (1995). Pilot of patient held guidelines is under way. BMJ 311: 1023a-1023 [Full text]