INTRODUCTION

The National Association of Emergency Medical Technicians launched its Advanced Medical Life Support (AMLS) education program in 1999. The AMLS program is now offered in 24 countries in Europe, Central and South America, and the Middle and Far East. The course emphasizes the use of scene size-up, history taking, interactive group discussion about potential treatment strategies, and physical examinations to systematically rule out and consider possibilities and probabilities in treating medical crises. This program employs an initial assessment-based approach that progresses to a diagnostic-based approach in order to quickly develop the best treatment plan for a given scenario. Learners must successfully complete a 50-question multiple-choice written examination as well as a scenario-based patient care exercise emphasizing patient assessment and the process of differential diagnosis. The written examination draws on content from both the textbook and scenario-based slide presentations [1]. All questions contained within the current 50-item set are structured as Type A items, which have one best answer. In order to expand the scope of the written examination to more fully focus upon the application of knowledge, we developed an assessment blueprint for the AMLS 16-hour two-day program and proposed revisions to the current written examination. Assessment blueprinting is a process of identifying and mapping test content against learning objectives in an effort to produce a valid examination [2]. The current written post-test examination for the AMLS program is several years old and has not undergone recent revision. The purpose of this project was to create an assessment blueprint to guide the creation of a new post-test for the AMLS program, thus ensuring an adequate sampling of course content. During this analytical process, new extended matching questions (EMQs) were designed to raise the level of cognitive domain assessment achieved by the post-test [5].

METHODS

Materials and subjects

The source materials for this program consisted of the AMLS first edition textbook [1] and the associated instructor electronic resource, which primarily consists of a series of scenarios used to support the practical stations. Table 1 summarizes how much time was allocated to the lectures and practical stations over the two-day course. This project was carried out from January 2015 to April 2015.

Development of an assessment blueprint

Based upon a review of the literature, we developed an assessment blueprint derived from previous research [3]. Both the tables in the aforementioned study and our blueprint struck a balance between computational complexity and specificity of purpose. Our method for categorizing cognitive objectives into lower- and higher-order objectives was derived from that of Bridge et al., who described objectives at the recall, interpretation, and problem-solving levels [4]. Our lower-order cognitive objectives corresponded to the recall level, while the higher-order cognitive objectives corresponded to a merged category consisting of the interpretation and problem-solving levels. This structure is well-suited to the AMLS program content. For each chapter, the following calculations were performed: first, calculation of the total program minutes devoted to each chapter by source (text vs. classroom vs. practical exercises) and cognitive level (lower vs. higher) (Table 2); second, calculation of the total and percent-estimated lower-order and higher-order program minutes for each chapter by cognitive level; and third, calculation of the number of lower- and higher-order cognitive items by chapter, based upon a 50-item written examination.

Test item design

Existing multiple choice questions were reviewed for structural consistency with the criteria described by Case and Swanson [5], and EMQs were written to address higher-order cognitive objectives.

RESULTS

Table 2 displays the structure of the blueprint. For example, in Chapter 9, out of the 30 total practice minutes, 1.5 minutes were estimated to have addressed lower-order cognitive information, while 28.5 minutes were estimated to have addressed higher-order cognitive information. The practice time predominantly addressed algorithmic diagnosis, which is considered higher-order reasoning by definition, making the estimate of 1.5 minutes devoted to lower-order cognitive reasoning reasonable. Since the testing format is dictated by the 16-hour duration of the class and by practical and logistical considerations, a 50-item examination would be overwhelmed by the 111 course objectives. As described in the methods section, the blueprint was constructed based on the distribution of classroom time devoted to each chapter and a hypothetical estimate of time devoted to student preparation. The following estimations were made. First, with the exception of the items in Chapter 1, which were weighted at 30 minutes for lower-order items versus 30 minutes for higher-order items, the items in each chapter of the textbook were weighted as 12 minutes for lower-order items versus 48 minutes for higher-order items. Second, Chapter 1 was not specifically reviewed in the lectures or classroom practice. The direct study of Chapter 1 was limited to reading the text. Hence, classroom lecture and practice minutes were not allotted for this chapter. Third, in the classroom practical exercises, where the algorithmic diagnosis of medical conditions was emphasized, higher-order cognitive level items were allocated 156.5 minutes, while lower-order cognitive level items were allocated 13.5 minutes.

This blueprint calculation resulted in a distribution of 53 items. The three additional items resulted from rounding error. This is not an entirely unwelcome outcome. One approach to utilizing the 53 items is to consider them a test bank, which allows the instructor some slight discretion in adjusting the mix of questions to suit local preferences. While the calculated total of approximately 50 items was predetermined by the method of calculation, the nearly even mix of lower- and higher-order cognitive level items was an unpredicted outcome. One explanation is that the proportion of 80% lower-order cognitive level to 20% higher-order cognitive level items utilized to calculate the question mix in the textbook counterbalanced the strong emphasis on algorithmic processes present in the classroom materials. Our results indicated that the items for an appropriately constructed written examination should be approximately evenly split between the lower-order and higher-order cognitive domains. An example of an EMQ to address course content reflecting higher-order cognitive domains can be found in Appendix 1.

DISCUSSION

A significant finding of this study was that after the calculation of the assessment blueprint, the AMLS post-course exam questions needed to be enhanced by creating EMQs to assess the higher-order cognitive level. This article describes the first known published account of the application of an assessment blueprint for the Advanced Medical Life Support (AMLS) two-day prehospital emergency medical services training program. It also complements accounts of the introduction of blueprinting in other assessment settings [6]. While prehospital and medical board organizations administering high-stakes examinations utilize educational specialists or psychometricians to ensure the validity of their examinations, many smaller certifying bodies may lack the resources to do so, making this an important contribution. Roberts et al. noted that adequate resources are not always available even at the university level to ensure the valid and reliable assessment of clinical competence in the United Kingdom [7]. Building upon earlier approaches, we proposed a methodology especially suited to programs constrained by program duration relative to a large number of objectives and seemingly endless numbers of possible test questions [3,8]. Based upon the authors’ experience, the 50-item multiple-choice exam is a fairly common fixture in prehospital professional development examinations. A 50-item exam is reasonably sized in light of the warnings of Notar et al. [9] regarding examination length, student attention span, and fatigue. However, the 50-item ceiling can be easily overwhelmed by a large number of learning objectives, potentially resulting in an examination constructed of a set of excessively fine-grained questions that ignore important central themes. Developing the blueprint at the chapter level, as we have done, should lead item writers to be more inclined to construct questions that focus on important central themes or procedures. Assessment blueprinting also offers advantages beyond its reported enhancement of content validity in examinations [2]. Ahmad et al. recently reported that their blueprinting methods, which were somewhat similar to our approach, led to improved student outcomes as measured by test scores and self-reported student satisfaction measures [8]. Furthermore, in light of the potential variability in the construction of assessment blueprints, we concur with these authors and recommend that each testing entity should consider the educational complexity and duration of their programs and courses and choose a blueprinting method best suited to ensuring content validity for each instance of testing.

In conclusion, our approach to blueprint creation, which included EMQs, is computationally straightforward and could be easily adopted by a group of instructors with a basic understanding of lower- and higher-order cognitive constructs. This approach should therefore make item writers more inclined to construct questions that focus on important central themes or procedures.

Acknowledgments

SUPPLEMENTARY MATERIAL

References

• 1. Advanced Medical Life Support Committee of the National Association of Emergency Medcal Technicians. Advanced Medical Life Support: An assessment-based approach. Burlington, MA: Jones & Bartlett Learning; 2011. 545 p.
• 2. DiDonato-Barnes N, Fives H, Krause ES. Using a Table of Specifications to improve teacher-constructed traditional tests: an experimental design. Assess Educ Princ Policy Pract 2014;21:90-108. http://dx.doi.org/10.1080/0969594X.2013.808173 Article
• 3. Fives H, DiDonato-Barnes N. Classroom test construction: The power of a table of specifications. Pract Assess Res Eval 2013;18:3. Available from: http://pareonline.net/getvn.asp?v=18&n=3.
• 4. Bridge PD, Musial J, Frank R, Roe T, Sawilowsky S. Measurement practices: methods for developing content-valid student examinations. Med Teach 2003;25:414-421. http://dx.doi.org/10.1080/0142159031000100337 ArticlePubMed
• 5. Case S, Swanson DB. Constructing written test questions for the basic and clinical sciences. Philadelphia, (PA): National Board of Medical Examiners; 2002. 180 p.
• 6. McLaughlin K, Lemaire J, Coderre S. Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach 2005;27:544-547. http://dx.doi.org/10.1080/01421590500136113 ArticlePubMed
• 7. Roberts C, Newble D, Jolly B, Reed M, Hampton K. Assuring the quality of high-stakes undergraduate assessments of clinical competence. Med Teach 2006;28:535-543. http://dx.doi.org/10.1080/01421590600711187 ArticlePubMed
• 8. Ahmad RG, Hamed OAE. Impact of adopting a newly developed blueprinting method and relating it to item analysis on students’ performance. Med Teach 2014;36 Suppl 1:S55-S61. http://dx.doi.org/10.3109/0142159x.2014.886014 Article
• 9. Notar C, Zuelke D, Wilson J, Yunker B. The table of specifications: Insuring accountability in teacher made tests. J Instr Psychol 2004;31:115-129. https://www.questia.com/library/journal/1G1-119611686/the-table-of-specifications-insuring-accountability

Appendix

APPENDIX 1.

Example of extended matching-type items.

For each scenario, choose the SINGLE most likely diagnosis from the below list of options. Each option may be used once, more than once, or not at all.

List of options (A - R)

• A. Acute coronary syndrome

• B. Aortic dissection

• C. Arrhythmia

• D. Congestive heart failure

• E. Costochondritis

• F. Esophageal rupture

• G. Non- ST-elevation myocardial infarction

• H. Pericardial tamponade

• I. Pericarditis

• J. Pneumonia

• K. Pulmonary embolus

• L. Stable angina

• M. ST-elevation myocardial infarction

• N. ST-elevation myocardial infarction with right ventricular infarct

• O. Strain

• P. Tension pneumothorax

• Q. Thoracic outlet syndrome

• R. Unstable angina

Item 1. A 48-year-old female presents with increasing 4 of 10 dull ache in mid-chest for two days. She denies radiation, nausea, vomiting, and shortness of breath. The pain worsens when supine. Skin is warm and dry, breath sounds are equal and clear with patient complaint of some increased discomfort upon inspiration. Heart sounds are normal, there is no jugular vein distention, peripheral edema is absent and patient denies pain upon palpation of chest wall and abdomen. Twelve-lead EKG (Fig. 1) was recorded during exam. BP was 142/80, pulse 110, R-18 & non-labored, O2 saturation 94% on room air, weight 55 kg. She denies medical history. States has had a bad cold with cough for 5 days. She has been taking acetaminophen for discomfort and Over-the-counter (OTC) cold medication.

Figure 1.

Twelve-lead electrocardiogram of the patient in item 1.

Answer: I – Pericarditis.

Item 2. A 42-year-old male complains of 5 of 10 chest pain at a local restaurant. Pain continues for 20 minutes while eating. He denies radiation, nausea, vomiting and shortness of breath. The pain does not change with breathing or position. Skin is warm and dry, breath sounds are equal. Heart sounds are normal, there is no jugular vein distension, and peripheral edema is absent. He denies pain upon palpation of chest wall and abdomen. Daily meds are aspirin 81 mg, cimetidine 200 mg and s\Simvastatin 20 mg. History includes hypertension and cholesterol with GERD. The pain subsides after administration of 0.4 mg nitroglycerin SL×2 doses. Twelve-lead EKG (Fig. 2) was recorded during exam. BP 174/88, pulse 75, R-14 & non-labored, O2 saturation 96% on room air, weight 55 kg.

Figure 2.

Twelve-lead electrocardiogram of the patient in Item 2.

Answer: L – Stable angina.

Item 3. A 62-year-old male is found sitting in a bedroom in a tripod position complaining of 8 of 10 shortness of breath (SOB). He was awakened from sleep with SOB, denies pain and radiation. SOB worsens when he attempts to lay supine. Skin is cool, pale and diaphoretic. Breathing is labored with accessory muscle use. Breath sounds are diminished in the bases with crackles and mild wheezing is heard to mid lobes and apices. Jugular vein distension is +. Pedal edema is 2+. His single daily med is hydrochlorothiazide. He states he has high blood pressure and was diagnosed with myocardial infarction three years ago. Twelve-lead EKG (Fig. 3) was recorded during exam. BP 190/110, pulse 110, R-36 & non-labored, O2 saturation 88% on room air, weight 85 kg.

Figure 3.

Twelve-lead electrocardiogram of the patient in Item 3.

Answer: D – Congestive heart failure.

Figure & Data

References

Citations

Citations to this article as recorded by  

• Evaluation of Emergency First Response’s Competency in Undergraduate College Students: Enhancing Sustainable Medical Education in the Community for Work Occupational Safety
  Graciano Dieck-Assad, Omar Israel González Peña, José Manuel Rodríguez-Delgado
  International Journal of Environmental Research and Public Health.2021; 18(15): 7814.     CrossRef
• Quality assurance of test blueprinting
  Ghada Eweda, Zakeya Abdulbaqi Bukhary, Omayma Hamed
  Journal of Professional Nursing.2020; 36(3): 166.     CrossRef