Full questionnaire details

Table A Sequence and number of completed sets of Berlin Questionnaire by control groups....

Table B Sequence and number of completed sets of Berlin Questionnaire by participants on three courses in ...
 

Development and validation of instrument control groups

Development of instrument

We derived the content from course topics of a published curriculum, which included basic epidemiological, statistical, and informational concepts relevant to evidence based medicine in the clinical situation (diagnostic tests, therapy, prognosis, harm, and systematic reviews).^w1 The five experienced teachers in evidence based medicine—clinicans (ND-B, LF, and RK), clinical epidemiologists (ND-B, RK, H-WH), and a statistician (KW)—drafted the first set of multiple choice questions. We explicitly sought to assess deep learning (the ability to integrate concepts and to apply them in new situations) rather than superficial learning (the ability to recall definitions and facts). When calculations were part of the solution, the answer included the calculation path. Participants had to identify the correct path rather than merely select the correct numerical solution.

We recirculated the questions according to a modified Delphi method and we assessed for unambiguous wording and answers, clinical relevance, and importance for novices in evidence based medicine.^w2 Disagreements were resolved by discussion. We piloted the questions for understanding and feasibility in a group of five doctor volunteers not familiar with evidence based medicine and then modified the questions further. At the end of the first stage the instrument consisted of two sets of 15 test questions, with similar content applied to different case scenarios to allow repeated testing.

Validation of instrument

We assessed the difficulty of each question by calculating the proportion of correct answers for each question. We sought to ensure that scores per question by course participants should not fall below 0.1 or go above 0.9, as scores outside these parameters do not tend to provide additional information to distinguish the more knowledgeable participants from less knowledgeable ones. Equivalence of sets 1 and 2 was determined by calculating the intraclass correlation coefficient for the controls (students and experts), who received the test without intervention. We determined the internal consistency of the instrument, a measure of reliability as described in standard psychometric textbooks.^{w3 w4} We calculated Pearson’s correlation coefficient to assess the relation between an individual item and the instrument as a whole omitting that item (item total correlation^w5) to identify items contributing to a low reliability. As a rule of thumb, the correlation of that item with the total score should be above 0.2.^w3 In addition, we calculated a standard summary measure for internal consistency, the coefficient α (Cronbach’s α ).^w6 We considered a coefficient above 0.7 as a satisfactory level of internal consistency.^w6

Discriminative ability

We assessed the instrument’s ability to discriminate between groups with different expertise in evidence based medicine by assessing experts, students, and course participants. Both sets of the instrument were administered to the expert controls and the student controls. We analysed the following comparisons: experts versus course participants (before test) versus students; course participants before the course (group A versus group B versus group C), using one way analysis of variance. If we found a significant difference, we added a post hoc analysis by Scheffé’s method.

Measurement of educational effect

Educational intervention

The 3 day Berlin course on evidence based medicine was developed in 1997, based on the small group educational model developed at McMaster University, Canada and directed towards doctors at all levels of training.^w7 It covers previously defined core content in 22 hours of small group sessions and 5 hours of plenary presentations.^w1 It uses a problem based learning style with case scenarios of typical patient problems supported by relevant primary studies. We provided hands-on exercises on retrieval of information and refresher courses in basic statistics. Each group was run by two tutors,an experienced methodologist and a doctor with several years’ clinical experience. The sessions introduced doctors to the principles of evidence based medicine (identification and prioritisation of problems, formulation of questions, consideration of clinical decision options) and critical appraisal in the context of the patient scenario and helped them to become skilled users of evidence that had already been appraised.

w1 Kunz R, Fritsche L, Neumayer H-H. Development of quality assurance criteria for continuing education in evidence-based medicine. Z Arztl Fortbild Qualitatssich 2001;95:3712-5.

w2 Greenhalgh T. Consensus research. In: Elwyn G, Greenhalgh T, Macfarlane F, ed. Groups—a guide to small group work in health care management, education and research. Oxford: Radcliffe Medical Press, 2000.

w3 Streiner DL, Norman GR. Health measurement scales. Oxford: University Press, 1995.

w4 Lienert GA, Raatz U. Testaufbau und testanalyse. Weinheim: Psychologie Verlagsunion, 1995.

w5 Nunnally JC. Introduction to psychological measurement. New York: McGraw-Hill, 1970.

w6 Nunnally JC. Psychometric theory. New York: McGraw-Hill, 1978.

w7 The Evidence-Based Medicine Working Group. Users’ guides to the medical literature: a manual for evidence-based clinical practice. Chicago: JAMA and Archives Journals, 2002.

Table A Sequence and number of completed sets of Berlin Questionnaire by control groups. Values in parentheses are number completing set unless stated otherwise

*Sequence of test sets were reversed year on year, so that group completing initial set 1 went on to complete subsequent set 2 and group completing initial set 2 went on to complete subsequent set 1.
 

Table B Sequence and number of completed sets of Berlin Questionnaire by participants on three courses in evidence based medicine. Values in parentheses are number completing set unless stated otherwise