Effectiveness of a multiple intervention to reduce antibiotic prescribing for respiratory tract symptoms in primary care: randomised controlled trial
=====================================================================================================================================================

* Ineke Welschen
* Marijke M Kuyvenhoven
* Arno W Hoes
* Theo J M Verheij

## Abstract

**Objectives** To assess the effectiveness of a multiple intervention aimed at reducing antibiotic prescription rates for symptoms of the respiratory tract in primary care.

**Design** Randomised controlled trial.

**Subjects** Twelve peer review groups including 100 general practitioners with their collaborating pharmacists in the region of Utrecht, Netherlands.

**Intervention** The intervention consisted of group education meetings, with a consensus procedure on indication for and type of antibiotics and with training in communication skills; monitoring and feedback on prescribing behaviour; group education for assistants of general practitioners and pharmacists; and educational material for patients. The control group did not receive any of these elements.

**Main outcome measures** Antibiotic prescription rates for acute symptoms of the respiratory tract and patients' satisfaction.

**Results** 89 general practitioners completed the study (89%). At baseline, prescription rates for antibiotics for respiratory tract symptoms did not differ between intervention and control group (27% *v* 29%, respectively). After nine months, the prescription rates in the intervention group fell to 23%, whereas the control group's rose to 37% (mean difference in change −12%, 95% confidence interval −18.9% to −4.0%). Multilevel analysis confirmed the results of the unadjusted analysis (intervention effect −10.7%, −20.3% to −1.0%). Patients' satisfaction was high and did not differ in the two groups at baseline or after the intervention.

**Conclusions** A multiple intervention reduced prescribing rates of antibiotics for respiratory tract symptoms while maintaining a high degree of satisfaction among patients. Further research should focus on the sustainability and cost effectiveness of this intervention.

## Introduction

In the Netherlands, general practitioners prescribe almost 80% of all antibiotics, and up to two thirds of these prescriptions are issued for infections of the respiratory tract.1 These infections are often treated with antibiotics, although this has mostly not been found to be beneficial.2–6 Unnecessary use of antibiotics entails an increased risk of side effects,2 high costs,7 medicalising effects,8 and development of bacterial resistance against antibiotics.9–11 Although antibiotic prescribing rates in the Netherlands are low compared with other European countries12 and the United States, as many as 50% of such prescriptions are estimated to lack an evidence based indication.13 Non-clinical factors such as perceived patients' expectations play an important part in the decision whether or not to prescribe antibiotics.14 15

The Dutch College of General Practitioners developed evidence based guidelines for infections of the respiratory tract.16–19 However, implementation of these guidelines remains difficult.11 20 Educational outreach visits, local opinion leaders, and combinations of interventions have been shown to have the largest impact, but the results and methodological quality of these studies are highly variable.11 21 22 In general, multiple intervention strategies—including local doctors in setting guidelines, involving a leading participant from the peer review group, training doctors in communication skills (including patient centred healthcare strategies), monitoring prescribing behaviour, and sustaining the achieved consensus by means of feedback on prescribing and reminders—are considered most effective in optimising prescribing behaviour.21 23 However, such a strategy has not been evaluated for the management of respiratory tract infections in primary care. In a randomised controlled trial we evaluated the effectiveness of such a multiple intervention aiming at reducing antibiotic prescription rates for respiratory tract symptoms in primary care: the Utrecht antibiotics and respiratory tract infections (ARTI-1) study.

## Methods

### Recruitment

We used the existing nationwide structure of general practitioners' peer review groups, with collaborating pharmacists, which aims to promote rational prescribing through audit and feedback.24 Membership in the peer review group has been stable over time because it is unusual for general practitioners (and their patients) to switch between groups. We selected peer review groups in the region of Utrecht if the group consisted of at least four doctors. All members of the peer review groups had to agree about participation. We analysed administrative claims data from the regional health insurance company (Agis) to obtain a global and non-diagnosis related estimate of volumes of antibiotic prescribing per group.

### Randomisation

In view of the intervention applied and the outcome measures chosen, we thought that it was essential to achieve comparability between the intervention and control arms with regard to the volume of antibiotic prescribing (above or below the median), working in a rural or urban area, and the number of general practitioners per group (above or below median). Random allocation without taking into account the distribution of these variables would incur a risk for groups due to different characteristics (potential confounders). On the basis of the above variables we therefore allocated the 12 peer review groups who agreed to participate to groups A or B, to achieve comparability. We considered all possible compositions of groups A and B and chose the option of those groups resulting in comparability between group A and B in groups with a high or low volume of antibiotic prescribing, rural or urban working groups, and number of general practitioners per group. MMK, who was blinded to the composition of the groups, flipped a coin to determine whether group A became the intervention or control group.

### Intervention

The intervention consisted of a group education meeting with a consensus procedure on, firstly, the indication for and first choice of antibiotics for acute otitis media, sinusitis, tonsillitis, and acute cough, combined with communication skills training. The second component was monitoring and feedback on prescribing behaviour; the third, group education for assistants of general practitioners and pharmacists; and the fourth, education material for patients.

#### Group education meeting

The researchers prepared the education programme in cooperation with a general practitioner member and a pharmacist of each group, who jointly led the actual meeting. After exploring claims data (1999) we discussed an overview of evidence based medicine. In presenting the evidence we used relative and absolute effects of antibiotics by means of the numbers needed to treat and the numbers needed to treat to harm. This discussion resulted in group consensus about indication and first choice antibiotics per disease. Communication skills training, which aimed to learn how to explore patients' worries and expectations and to inform patients about the natural course of the symptoms, self medication, and alarm symptoms, concluded the meeting. After one week, all doctors received a summary of their group's guidelines by mail, and two months after the intervention they received the results of the baseline measurement (see outcome variables) to reinforce the consensus reached.

#### Monitoring prescribing behaviour

Six months after the intervention, general practitioners again received feedback on their prescribing behaviour, based on insurance claims data comparing the period after the intervention (March to May 2001) with the same period before the intervention (March to May 2000). Volumes of different kinds of antibiotics and the extent to which prescribed antibiotics were in line with the consensus about first choice antibiotics were presented at practice level.

#### Assistants

Assistants of participating doctors and pharmacists attended a two hour group education session informing them about Dutch guidelines for general practitioners, followed by skills training in educating patients.

#### Education materials for patients

Education materials for patients consisted of a brochure and accompanying posters, which aimed to inform patients about the self limiting character of most respiratory tract symptoms, self medication, and serious symptoms (“alarm signals”) necessitating a consultation with the general practitioner. The brochure (also translated into Turkish and Arabic) was available in waiting rooms of general practices, pharmacies, and municipal health services in the intervention group.

The control group did not receive any part of the intervention.

### Outcome measures

General practitioners registered all patients presenting with acute symptoms of the respiratory tract (house calls and out of hours activity not included) during three weeks in the autumn and winter of 2000 and 2001. Doctors noted diagnosis and management in patients' records as usual. Research assistants, who were blinded to the intervention status of the practices, scrutinised these files to extract information on age, sex, diagnoses (ICPC-9 codes25), antibiotic prescriptions (ATC codes26), and referrals to hospital doctors.

After the consultation, general practitioners asked patients to rate their satisfaction with the consultation on a scale ranging from 1 (very dissatisfied) to 5 (very satisfied). Turkish and Arabic translations of this questionnaire were available. Parents filled in questionnaires on behalf of their children. Patients' questionnaires went directly to the investigators without being shown to the general practitioner.

Main outcome variables were the proportion of practice encounters for acute symptoms of the respiratory tract for which antibiotics were prescribed and the degree of patients' satisfaction with the encounter. Secondary outcome variables were insurance claims data over the period 2000-2, to assess differences in volumes of antibiotic prescribing before and after the intervention, independent from general practitioners' registrations and referral rates to hospital doctors, because we assumed that reduced antibiotic prescribing in primary care might increase hospital referral rates.

### Sample size

Pre-study sample size calculations including multilevel correction27 showed that 49 general practitioners per arm would enable us to assess a relative reduction of 30% in antibiotic prescription rates (α 0.05; β 0.10; SD25), assuming a baseline antibiotic prescription rate of 55%. We expected a mean number of 10 included patients per general practitioner per week.

### Analysis

We used a *t* test for unpaired samples to test differences between intervention and control group regarding characteristics of general practitioners and patients (means of proportions) in 2000 and 2001. To assess the effectiveness of the multiple intervention, we used a t test for unpaired samples to compare mean changes in antibiotic prescription rates over time (at general practitioner level) in both groups (mean difference in change, 95% confidence interval). We used the same procedure to assess the effect on patients' satisfaction and referral rates. We performed a multilevel analysis to assess effectiveness, adjusted for clustering of general practitioners in practices and peer review groups28 29 and calculated intra-class correlation coefficients to rate the degree of clustering in practices and peer review groups.

To analyse the claims data we used a t test for unpaired samples to determine changes in the mean number of antibiotic prescriptions per 1000 patients over time (practice level) in intervention and control group. We used SPSS, version 10.0, and MlwiN, version 1.10, for our analyses, all of which we performed on an intention to treat basis.

## Results

Forty two of the 48 peer review groups in the region of Utrecht were eligible and invited to participate. Twelve peer review groups (100 general practitioners) agreed to participate (figure). Insurance claims data showed no differences in volumes of antibiotics prescribed in participating compared with non-participating doctors. Out of 100 general practitioners who agreed to participate, 89 completed the study. Eleven were lost to follow up (intervention group 4/46; control group 7/54) because of retirement (one doctor), removal outside the region (three), illness (three), motivational problems (two) or technical problems (two). General practitioners in both arms did not differ at baseline with regard to sex, practice characteristics, and mean period since registration as general practitioner (table 1). They did not differ either regarding the extent to which the group was used to discuss indication and first choice medication in their meetings (table 1). Registered patients in both arms did not differ in 2000 and 2001 regarding age, sex, and type of diagnosis (table 2). Almost 80% (37) of the general practitioners (intervention group) attended all parts of the intervention.

![Figure1](http://www.bmj.com/https://www.bmj.com/content/bmj/329/7463/431/F1.medium.gif)

[Figure1](http://www.bmj.com/content/329/7463/431/F1)

Flow of participants through the trial.

View this table:
[Table 1](http://www.bmj.com/content/329/7463/431/T1)

Table 1 
Baseline characteristics of general practitioners (n=89) participating in the study in 2000

View this table:
[Table 2](http://www.bmj.com/content/329/7463/431/T2)

Table 2 
Patients' characteristics in 2000 and 2001 in the intervention and control groups

At baseline, mean antibiotic prescription rates for registered encounters for respiratory tract symptoms did not differ significantly between the two groups (27% *v* 29%, 95% confidence interval −9.1 to 5.0). In 2001, antibiotic prescription rates in the intervention group fell by 4% and those in the control group rose by 8% (mean difference in change −12%, −18.9% to −4.0%, table 3). Multilevel analysis confirmed the results of the unadjusted analysis (intervention effect −10.7%, −20.3% to −1.0%), while intra-class correlation coefficients showed that variation could be attributed to practice and group levels 0.17 and 0.09, respectively).

View this table:
[Table 3](http://www.bmj.com/content/329/7463/431/T3)

Table 3 
Registration of patients: changes in antibiotic prescription rates and patients' satisfaction in 2000 and 2001. Values are means with standard deviations unless otherwise indicated

Patient satisfaction questionnaires were available from 83% (35) of general practitioners in the intervention group and from 91% (43) of doctors in the control group (table 2). We found no difference in patients' satisfaction in the two arms in 2000 and 2001 (table 3). The multiple intervention did not change patients' degree of satisfaction; they remained very satisfied with the consultation (mean satisfaction grade 4.2). These results were also confirmed by multilevel analysis (intervention effect on patients' satisfaction −0.03, −0.2 to 0.1).

Claims data over 2000 and 2001 were in line with our results: no significant differences occurred in the number of antibiotic prescriptions between intervention and control group in 2000. In 2001, however, the mean number of antibiotic prescriptions had decreased by 9.7 prescriptions per 1000 patients (P = 0.05) in the intervention group, whereas in the control group it had increased (P = 0.60). This increase was also seen in the non-participating general practitioners in the same region (mean difference in change between intervention and control group −11.6 prescriptions/1000 patients, −23.2 to −0.03) and confirmed by multilevel analysis (table 4). After 15 months, the number of antibiotic prescriptions in the intervention group was still lower than in the pre-intervention period (data not shown). Referral rates (about 2%) remained stable over time and did not differ between intervention and control groups (mean difference in change −0.1, −2.0 to 1.8).

View this table:
[Table 4](http://www.bmj.com/content/329/7463/431/T4)

Table 4 
Insurance claims data: changes in mean number of antibiotic prescriptions per 1000 patients in March-April-May 2000 and March-April-May 2001 at practice level. Values are means with standard deviations

## Discussion

Our intervention—which included a group education meeting with a consensus procedure and communication skills training, monitoring, and feedback on prescribing behaviour; group education for the assistants of general practitioners and pharmacists; and education materials for patients—was effective in reducing antibiotic prescribing for respiratory tract symptoms. Prescribing rates of antibiotics for respiratory tract symptoms fell by 12% compared with the control group. The intervention did not affect patients' satisfaction or hospital referral rates. Our results were confirmed by claims data over the same period, which showed a decrease in the intervention group and an increasing trend in both the control group and the non-participating general practitioners in the same region. Such variations in antibiotic prescribing are common,30 which emphasises the need of a controlled design to account for these variations by comparing the mean differences over time between intervention and control group. The randomisation procedure produced two comparable groups of general practitioners and patients. The observed reduction therefore seems attributable to the intervention.

### Potential biases

The fact that general practitioners may not have included all patients during the registration periods could have caused selection bias. This would be the case if doctors in the intervention group in 2001 tended to include patients for whom few antibiotics were prescribed compared with the control group. Importantly, since claims data over the same periods confirmed our findings and the case mix between intervention and control groups was similar, we assume that selection of patients is unlikely to have biased the results.

About half of the patients received and returned the patients' questionnaire. Non-response can at least partly be explained by the fact that not all patients were given a questionnaire by doctors, probably because of time constraints during the encounter; this can be considered as a more or less random phenomenon in both groups. Nevertheless, satisfied patients may be over-represented in our sample because general practitioners may tend to give out questionnaires to patients in whom they expected a high level of satisfaction. Importantly, however, this will not bias the results if satisfied patients are over-represented in both groups, as is most likely. Bias would occur only if general practitioners in one of the two groups were more inclined to administer questionnaires to satisfied patients than the doctors in the other group. We think this is highly unlikely.

Although our study was not powered to assess an intervention effect on individual diagnoses, the percentage of antibiotics prescribed decreased for all diagnoses in the intervention group compared with the control group except for pneumonia (data not shown). This was expected since antibiotic treatment is strongly recommended in cases of pneumonia.

### Role of bacterial resistance

We did not assess the effect of the intervention on the development of resistant micro-organisms because in the Netherlands the resistance rate in the population is very low. In countries where resistance is high, these kinds of interventions could be accompanied by monitoring of resistance patterns.

### Comparison with other trials

A randomised controlled trial of Zwar et al, which evaluated the effectiveness of an intervention consisting of prescribing feedback, educational material, patient education materials, and face to face instruction for high or wrong prescribers, showed the same effect size,31 whereas a more passively delivered complex intervention had little effect in changing practice.32

### Outlook

Uncertainty remains about the sustainability of intervention effects.23 However, claims data over the first trimester of 2002 provided a rough indication about the longer term effectiveness of this intervention. After 15 months, the number of antibiotic prescriptions in the intervention group was still lower than in the period before the intervention. It is possible that the peer review group structure reinforces the changed behaviour. Further research should focus on structural sustainability and cost effectiveness of these interventions.

#### What is already known on this topic

Changing prescribing behaviour is a complex task

Many evidence based guidelines for infections of the respiratory tract are available

Implementation of guidelines is difficult

#### What this study adds

A multiple intervention consisting of four components reduced antibiotic prescribing rates

The intervention also maintained patients' satisfaction

## Footnotes

*   We thank the general practitioners and their patients who participated in this study and our data managers, P Zuithoff and F Verheij, for their help with data processing and analyses. We also want to thank the Municipal Health Service of Utrecht and the Dutch Institute for Rational Drug Use (DGV Nederlands Instituut voor Verantwoord Medicijngebruik) for their co-operation with the development of the intervention. We thank the regional health insurance company (Agis) for supply of the claims data.

*   Contributors MK, AH, and TV conceived the study and developed the protocol. IW, MK, AH, and TV collected data, managed the study, and wrote and interpreted the report. MK is guarantor.

*   Funding Netherlands Organisation for Health Research and Development (Zorg Onderzoek Nederland) project number 2200.0057 and Foundation for the Advancement of Appropriate Prescription Drug Usage in the Central Region of the Netherlands (Stichting Doelmatig Geneesmiddelengebruik Midden Nederland).

*   Competing interests None declared.

*   Ethical approval Research Ethics Committee of the University Medical Center (UMC) Utrecht.

## References

1.  1.De Melker RA, Kuyvenhoven MM. Management of upper respiratory tract infections in Dutch family practice. J Fam Pract 1994;38: 353–7.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=8163959&link_type=MED&atom=%2Fbmj%2F329%2F7463%2F431.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1994NF55900009&link_type=ISI) 

2.  2.Arroll B, Kenealy T. Antibiotics for the common cold. Cochrane Database Syst Rev 2000;(2): CD000247.
    
    

3.  3.Del Mar CB, Glasziou PP, Spinks AB. Antibiotics for sore throat. Cochrane Database Syst Rev 2000;(4): CD000023.
    
    

4.  4.Becker L, Glazier R, McIsaac W, Smucny J. Antibiotics for acute bronchitis. Cochrane Database Syst Rev 2000;(2): CD000245.
    
    

5.  5.Glasziou PP, Del Mar CB, Sanders SL, Hayem M. Antibiotics for acute otitis media in children. Cochrane Database Syst Rev 2004;(1): CD000219.
    
    

6.  6.Williams JW, Aguilar C, Makela M, Cornell J, Hollman DR, Chiquette E, et al. Antibiotics for acute maxillary sinusitis. Cochrane Database Syst Rev 2003;(2): CD000243.
    
    

7.  7.Mainous III G, Hueston WJ. The cost of antibiotics in treating upper respiratory tract infections in a Medicaid population. Arch Fam Med 1998;7: 45–9.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1001/archfami.7.1.45&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=9443698&link_type=MED&atom=%2Fbmj%2F329%2F7463%2F431.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000071554800010&link_type=ISI) 

8.  8.Little P, Gould C, Williamsen I, Warner G, Gantly M, Kinmonth AL. Reattendance and complications in a randomised trial of prescribing strategies for sore throat: the medicalising effect of prescribing antibiotics. BMJ 1997;315: 350–2.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTUvNzEwNC8zNTAiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

9.  9.Belongia EA, Schwartz B. Strategies for promoting judicious use of antibiotics by doctors and patients. BMJ 1998;317: 668–71.
    
    [FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTcvNzE1OS82NjgiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

10. 10.Seppälä H, Klaukka T, Vuopio-Varkula J, Muotiala A, Helenius H, Lager K, et al. The effect of changes in the consumption of macrolide antibiotics on erythromycin resistance in group A streptococci in Finland. N Eng J Med 1997;337: 441–6.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1056/NEJM199708143370701&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=9250845&link_type=MED&atom=%2Fbmj%2F329%2F7463%2F431.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=A1997XQ49900001&link_type=ISI) 

11. 11.Arnold SR, Evans M, Straus SE. Interventions to improve antibiotic prescribing practices in ambulatory care (Protocol for a Cochrane Review). Cochrane Library; Issue 1, 2003. Oxford: Update Software.
    
    

12. 12.Cars O, Molstad S, Melander A. Variation in antibiotic use in the European Union. Lancet 2001;357: 1851–53.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(00)04972-2&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=11410197&link_type=MED&atom=%2Fbmj%2F329%2F7463%2F431.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000169160800017&link_type=ISI) 

13. 13.De Melker RA. Effectiviteit van antibiotica bij veelvoorkomende luchtweginfecties in de huisartspraktijk [effectiveness of antibotics in common infections of the respiratory tract in general practice]. Ned Tijdschr Geneeskunde 1998;142: 452–6.
    
    

14. 14.Butler CC, Rollnick S, Pill R, Maggs-Rapport F, Scott N. Understanding the culture of Prescribing: qualitative study of general practitioners' and patients' perceptions of antibiotics for sore throats. BMJ 1998;317: 637–42.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTcvNzE1OS82MzciO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

15. 15.Davey P, Pagliari C, Hayes A. The patients' role in the spread and control of bacterial resistance to antibiotics. Clin Microbiol Infect 2002;8(suppl): 43–68.
    
    

16. 16.De Bock GH, Van Duijn NP, Dagnelie CF, Geijer RMM, Van der Hell RJ, Labots-Vogelesang SM, et al. NHG-Standaard Sinusitis. Huisarts Wet 1993;36: 255–7. (Guideline on acute sinusitis of the Dutch College of General Practitioners.)
    
    

17. 17.Dagnelie CF, Zwart S, Balder FA, Romeijnders ACM, Geijer RMM. NHG-standaard Acute Keelpijn (eerste herziening). Huisarts Wet 1999;42: 271–8. (Guideline on acute tonsillitis of the Dutch College of General Practitioners, first revision.)
    
    

18. 18.Appelman CLM, Van Balen FAM, Van de Lisdonk EH, Van Weert HCLM, Eizenga WH. NHG-Standaard Otitis Media Acuta (eerste herziening). Huisarts Wet 1999;33: 242–5). (Guideline on acute otitis media of the Dutch College of General Practitioners, first revision.)
    
    

19. 19.Verheij TJM, Salomé PL, Bindels PJ, Chavannes AW, Ponsioen BP, Sachs APE, et al. NHG-Standaard Acuut Hoesten. Huisarts Wet 2003;46: 496–506. (Guideline on acute cough of the Dutch College of General Practitioners.)
    
    

20. 20.Grol R, Dalhuijsen J, Thomas S, In ‘t Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ 1998;317: 858–61.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTcvNzE2Mi84NTgiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

21. 21.Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: a literature review. Int J Qual Health Care 1994;6: 115–32.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiaW50cWhjIjtzOjU6InJlc2lkIjtzOjc6IjYvMi8xMTUiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

22. 22.Gross PA, Pujat D. Implementing practice guidelines for appropriate antimicrobial usage. A systematic review. Med Care 2001;39: II55–69.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=11583122&link_type=MED&atom=%2Fbmj%2F329%2F7463%2F431.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000170313600004&link_type=ISI) 

23. 23.De Meyere M, Guillemot D, Verheij TJM. Interventions in the community related to antibiotic use. Final report of the European conference on antibiotic use in Europe. European Surveillance of Antimicrobial Consumption (ESAC), Brussels, November 15–17, 2001; 38–43.
    
    

24. 24.Coenen S, Kuyvenhoven MM, Butler CC, Van Rooyen P, Verheij Th JM. Variation in European antibiotic use [letter]. Lancet;358: 1272.
    
    

25. 25.Lamberts H, Wood M. ICPC. International classification of primary care. Oxford: Oxford University Press, 1987.
    
    

26. 26.WHO Collaborating Centre for Drug Statistics Methodology. Anatomical therapeutic chemical (ATC) classification system: guidelines for ATC classification and DDD assignment. [www.whocc.no/atcddd/](http://www.whocc.no/atcddd/) (accessed 8 Jan 2004).
    
    

27. 27.Van Houwelingen JC. Dwalingen in de methodologie. III. Randomisatie op het niveau van behandelaars. Ned Tijdschr Geneesk 1998;142: 1662–5.
    
    

28. 28.Hox JJ. Applied multilevel analysis. Amsterdam, TT-Publikaties, 1994.
    
    

29. 29.Goldstein H. Multilevel models in educational and social research. London: Oxford University Press, 1987.
    
    

30. 30.Kuyvenhoven MM, Van Balen FAM, Verheij TJM. Outpatient antibiotic prescriptions from 1992 to 2001 in the Netherlands. J Antimicrob Chemother 2003;52: 675–8.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamFjIjtzOjU6InJlc2lkIjtzOjg6IjUyLzQvNjc1IjtzOjQ6ImF0b20iO3M6MjI6Ii9ibWovMzI5Lzc0NjMvNDMxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. 31.Zwar N, Wolk J, Gordon J, Sanson-Fisher R, Kehoe L. Influencing antibiotic prescribing in general practice: a trial of prescriber feedback and management guidelines. Fam Pract 1999;16: 495–500.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZmFtcHJhY3QiO3M6NToicmVzaWQiO3M6ODoiMTYvNS80OTUiO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

32. 32.Flottorp S, Oxman AD, Havelsrud K, Treweek S, Herrin J. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ 2002;325: 367.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjUvNzM2MC8zNjciO3M6NDoiYXRvbSI7czoyMjoiL2Jtai8zMjkvNzQ2My80MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9)