Journal of Educational Evaluation for Health Professions

Research articles

Evaluation of medical school faculty members’ educational performance in Korea in 2022 through analysis of the promotion regulations: a mixed methods study: Hye Won Jang, Janghee Park; J Educ Eval Health Prof. 2023;20:7. Published online February 28, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.7; Funded: Soonchunhyang University Research Fund

336 View
59 Download

Abstract PDF Supplementary Material: Purpose
To ensure faculty members’ active participation in education in response to growing demand, medical schools should clearly describe educational activities in their promotion regulations. This study analyzed the status of how medical education activities are evaluated in promotion regulations in 2022, in Korea.
Methods
Data were collected from promotion regulations retrieved by searching the websites of 22 medical schools/universities in August 2022. To categorize educational activities and evaluation methods, the Association of American Medical Colleges framework for educational activities was utilized. Correlations between medical schools’ characteristics and the evaluation of medical educational activities were analyzed.
Results
We defined 6 categories, including teaching, development of education products, education administration and service, scholarship in education, student affairs, and others, and 20 activities with 57 sub-activities. The average number of included activities was highest in the development of education products category and lowest in the scholarship in education category. The weight adjustment factors of medical educational activities were the characteristics of the target subjects and faculty members, the number of involved faculty members, and the difficulty of activities. Private medical schools tended to have more educational activities in the regulations than public medical schools. The greater the number of faculty members, the greater the number of educational activities in the education administration and service categories.
Conclusion
Medical schools included various medical education activities and their evaluation methods in promotion regulations in Korea. This study provides basic data for improving the rewarding system for efforts of medical faculty members in education.

Obstetrics and gynecology residents’ satisfaction and self-confidence after an anal sphincter injury simulation-based workshop in Indonesia: a pre- and post-intervention comparison study: Riska Wahyuningtyas, Eighty Mardiyan Kurniawati, Budi Utomo, Gatut Hardianto, Hari Paraton, Tri Hastono, Djoko Kuswanto; J Educ Eval Health Prof. 2022;19:4. Published online February 14, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.4; Funded: Universitas Airlangga

3,431 View
418 Download

Abstract PDF Supplementary Material: Purpose
Obstetric anal sphincter injury is one of the most common complications during delivery. Simulation models with manikins can be used as an effective medical learning method to improve students’ abilities before encountering patients. The present study aimed to describe the development of an anal sphincter injury model and to assess residents’ satisfaction and self-confidence after a perineal repair workshop with an anal sphincter injury simulator in Indonesia.
Methods
This was a cross-sectional study with evaluation of outcomes before and after the workshop. We created a silicone-latex simulation anal sphincter injury model. Then, we validated this simulation and used it as a simulation model for the workshop. We asked residents about their satisfaction with repairing anal sphincter injuries using a simulation model and residents’ self-confidence when practicing anal sphincter injury repair.
Results
All residents felt the simulation-based workshop was valuable (100%). Most of the scores for the similarity of the simulation model were good (about 8 out of maximum 10). The self-assessment of confidence was measured before and after the workshop. Overall self-confidence increased significantly after the workshop in identifying the external sphincter ani (EAS) (P=0.031), suturing the anal mucosa (P=0.001), suturing the internal sphincter ani (P=0.001), suturing the EAS (P<0.001), and evaluating the sphincter ani tone (P=0.016).
Conclusion
The anal sphincter injury simulator improved residents’ self-confidence in identifying the EAS, suturing the anal mucosa, suturing the internal sphincter ani, suturing the EAS, and evaluating sphincter ani tone.

Medical residents and attending physicians’ perceptions of feedback and teaching in the United States: a qualitative study: Madeleine Matthiesen, Michael S. Kelly, Kristina Dzara, Arabella Simpkin Begin; J Educ Eval Health Prof. 2022;19:9. Published online April 26, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.9; Funded: Massachusetts General Hospital Center, National Institutes of Health

7,019 View
300 Download

Abstract PDF Supplementary Material: Purpose
Residents and attendings agree on the importance of feedback to resident education. However, while faculty report providing frequent feedback, residents often do not perceive receiving it, particularly in the context of teaching. Given the nuanced differences between feedback and teaching, we aimed to explore resident and attending perceptions of feedback and teaching in the clinical setting.
Methods
We conducted a qualitative study of internal medicine residents and attendings from December 2018 through March 2019 at the Massachusetts General Hospital to investigate perceptions of feedback in the inpatient clinical setting. Residents and faculty were recruited to participate in focus groups. Data were analyzed using thematic analysis to explore perspectives and barriers to feedback provision and identification.
Results
Five focus groups included 33 total participants in 3 attending (n=20) and 2 resident (n=13) groups. Thematic analysis of focus group transcripts identified 7 themes which organized into 3 thematic categories: (1) disentangling feedback and teaching, (2) delivering high-quality feedback, and (3) experiencing feedback in the group setting. Residents and attendings highlighted important themes in discriminating feedback from teaching. They indicated that while feedback is reactive in response to an action or behavior, teaching is proactive and oriented toward future endeavors.
Conclusion
Confusion between the critical concepts of teaching and feedback may be minimized by allowing them to each have their intended impact, either in response to prior events or aimed toward those yet to take place.

Is it possible to introduce an interview to the Korean Medical Licensing Examination to assess professional attributes?: a survey-based observational study: Seung-Joo Na, HyeRin Roh, Kyung Hee Chun, Kyung Hye Park, Do-Hwan Kim; J Educ Eval Health Prof. 2022;19:10. Published online May 10, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.10; Funded: Korea Health Personnel Licensing Examination

2,076 View
265 Download

Abstract PDF Supplementary Material: Purpose
This study aimsed to gather opinions from medical educators on the possibility of introducing an interview to the Korean Medical Licensing Examination (KMLE) to assess professional attributes. Specifically following topics were dealt with: the appropriate timing and tool to assess unprofessional conduct; ; the possiblity of prevention of unprofessional conduct by introducing an interview to the KMLE; and the possibility of implementation of an interview to the KMLE.
Methods
A cross-sectional study approach based on a survey questionnaire was adopted. We analyzed 104 pieces of news about doctors’ unprofessional conduct to determine the deficient professional attributes. We derived 24 items of unprofessional conduct and developed the questionnaire and surveyed 250 members of the Korean Society of Medical Education 2 times. Descriptive statistics, cross-tabulation analysis, and Fisher’s exact test were applied to the responses. The answers to the open-ended questions were analyzed using conventional content analysis.
Results
In the survey, 49 members (19.6%) responded. Out of 49, 24 (49.5%) responded in the 2nd survey. To assess unprofessional conduct, there was no dominant timing among basic medical education (BME), KMLE, and continuing professional development (CPD). There was no overwhelming assessment tool among written examination, objective structured clinical examination, practice observation, and interview. Response rates of “impossible” (49.0%) and “possible” (42.9%) suggested an interview of the KMLE prevented unprofessional conduct. In terms of implementation, “impossible” (50.0%) was selected more often than “possible” (33.3%).
Conclusion
Professional attributes should be assessed by various tools over the period from BME to CPD. Hence, it may be impossible to introduce an interview to assess professional attributes to the KMLE, and a system is needed such as self-regulation by the professional body rather than licensing examination.

Simulation-based training using a novel Surabaya hysterectomy mannequin following video demonstration to improve abdominal hysterectomy skills of obstetrics and gynecology residents during the COVID-19 pandemic in Indonesia: a pre- and post-intervention study: Dara Dasawulansari Syamsuri, Brahmana Askandar Tjokroprawiro, Eighty Mardiyan Kurniawati, Budi Utomo, Djoko Kuswanto; J Educ Eval Health Prof. 2022;19:11. Published online May 17, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.11; Funded: Universitas Airlangga

2,749 View
274 Download

Abstract PDF Supplementary Material: Purpose
During the coronavirus disease 2019 (COVID-19) pandemic, the number of abdominal hysterectomy procedures decreased in Indonesia. The existing commercial abdominal hysterectomy simulation model is expensive and difficult to reuse. This study compared residents’ abdominal hysterectomy skills after simulation-based training using the Surabaya hysterectomy mannequin following a video demonstration.
Methods
We randomized 3rd- and 4th-year obstetrics and gynecology residents to a video-based group (group 1), a simulation-based group (group 2), and a combination group (group 3). Abdominal hysterectomy skills were compared between before and after the educational intervention. The pre- and post-tests were scored by blinded experts using the validated Objective Structured Assessment of Technical Skills (OSATS) and Global Rating Scale (GRS).
Results
A total of 33 residents were included in the pre- and post-tests. The OSATS and GRS mean differences after the intervention were higher in group 3 than in groups 1 and 2 (OSATS: 4.64 [95% CI, 2.90–6.37] vs. 2.55 [95% CI, 2.19–2.90] vs. 3.82 [95% CI, 2.41–5.22], P=0.047; GRS: 10.00 [95% CI, 7.01–12.99] vs. 5.18 [95% CI, 3.99–6.38] vs. 7.18 [95% CI, 6.11–8.26], P=0.006). The 3rd-year residents in group 3 had greater mean differences in OSATS and GRS scores than the 4th-year residents (OSATS: 5.67 [95% CI, 2.88–8.46]; GRS: 12.83 [95% CI, 8.61–17.05] vs. OSATS: 3.40 [95% CI, 0.83–5.97]; GRS: 5.67 [95% CI, 2.80–8.54]).
Conclusion
Simulation-based training using the Surabaya hysterectomy mannequin following video demonstration can be a bridge to learning about abdominal hysterectomy for residents who had less surgical experience during the COVID-19 pandemic.

Comparison between residents with a 6-year medical program and a 7-year medical program in terms of objective structured clinical examination performance in postgraduate year training in Taiwan: a 2-group pre- and post-test non-synchronized study: Ya-Ting Chang, Ying-Ying Yang, Chung-Pin Li, Chen-Huan Chen; J Educ Eval Health Prof. 2022;19:13. Published online June 24, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.13; Funded: Taipei Veterans General Hospital, Ministry of Education, Ministry of Science and Technology, Taiwan

1,560 View
188 Download

Abstract PDF Supplementary Material: Purpose
In 2013, medical schools in Taiwan implemented a 6-year medical program that replaced the previous 7-year medical education program. The postgraduate year (PGY) program was also extended from 1 year to 2 years. The new program is characterized by diversified teaching, integration of medical skills, a system-oriented curriculum, and the implementation of primary care and clinical thinking training. The purpose of this study was to examine whether postgraduate residents who learned under the new program have better patient care skills than those who learned under the previous program.
Methods
Of 101 residents in the PGY program at Taipei Veterans General Hospital, 78 were trained in the 6-year program, while 23 were trained in the 7-year program. During the PGY training, 2 objective structured clinical examinations (OSCEs) were used to evaluate clinical reasoning, communication skills, and procedural skills at the beginning of the training and after 11 months of training, respectively. The scores of each OSCE and the rate of improvement of the pre- and post-tests were analyzed.
Results
Residents trained in the new program scored higher on clinical reasoning (P<0.001) and the total scores of the 3 tested skills (P=0.019) on the pre-test. In terms of improvement, residents educated in the previous system improved more in clinical reasoning than those educated in the new education system.
Conclusion
The new medical education program, which emphasizes clinical thinking, improved residents’ clinical skills. The PGY program was effective in improving the clinical performance of residents who were educated in the previous system.

Effect of online education on the knowledge on, attitudes towards, and skills in patient safety for nursing students in Korea: a mixed-methods study: Dan Bi Cho, Won Lee, So Yoon Kim, Sungkyoung Choi; J Educ Eval Health Prof. 2022;19:14. Published online June 30, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.14; Funded: Ministry of Health and Welfare, Korea Health Industry Development Institute

1,628 View
285 Download

Abstract PDF Supplementary Material: Purpose
The purpose of this study was to evaluate the impact of synchronous online education on the patient safety competency (knowledge, attitudes, and skills) of nursing students in Korea and to explore what they thought about this educational method.
Methods
A single-group pre-and post-test design and summative content analysis were implemented. On November 14th, 2020, 110 nursing students completed synchronous online patient safety education. Patient safety competency was measured before and after the intervention using the revised Patient Safety Competency Self-Evaluation tool. The descriptive statistics, paired t-test, and Wilcoxon signed-rank test were used to analyze the data. Students also expressed their opinions about this education based on open-ended questions.
Results
All the patient safety competency scores significantly increased after intervention. A summative content analysis of the open-ended questions yielded 5 categories: educational materials, education level, education time, interaction, and educational media.
Conclusion
This study found that synchronous online patient safety education improves nursing students’ knowledge on, attitudes towards, and skills in patient safety. Nursing students also expressed a variety of positive aspects of the online education method. To improve the efficacy of synchronous online patient safety education, there is a need for further empirical studies on the appropriate class duration and difficulty of the content. It is essential to find a way to combine online education with various learning activities.

Educational/Faculty development material

Environmental management education using immersive virtual reality in asthmatic children in Korea: a randomized controlled study (secondary publication): Seung Hyun Kim, Sang Hyun Park, Insoon Kang, Yuyoung Song, Jaehoon Lim, Wonsuck Yoon, Young Yoo; J Educ Eval Health Prof. 2022;19:15. Published online July 11, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.15; Funded: Environmental Health Center for Childhood Asthma, Ministry of Environmental, Allergy Immunology Center

5,385 View
208 Download
2 Citations

Abstract PDF Supplementary Material

Awareness of environmental control is considered a significant influence on the performance of asthma self-management behaviors, which are involved in maintaining effective asthma control. This study aimed to investigate whether immersive virtual reality (VR) education is effective in environmental control education for asthmatic children in Korea. Thirty asthmatic children aged 9 to 13 years with aeroallergen sensitization were enrolled. Environmental control education for asthmatic participants was performed using immersive VR (VR group) or conventional leaflets provided by asthma specialists (control group). Five questionnaires, on awareness of environmental control, memory, assessment of intent to act, a satisfaction test, and an Asthma Control Test (ACT), were used to estimate the effects of education. The scores for awareness of environmental control, memory, and intent to act significantly increased after education in both groups, and the scores remained high until 4 weeks after education. Both groups’ ACT scores were consistently high before and 4 weeks after education. Satisfaction scores were very high in the VR group. The increased scores in awareness of environmental control and intent to act indicate that the environmental control education using VR is worthy of attention as an effective educational tool for asthma management. Further developed techniques, including active environmental interventions by participants in VR, could be applied to effective asthma management.

Citations

Citations to this article as recorded by

A Systematic Review of Randomized Controlled Trials on Virtual Reality Application in Pediatric Patients
Ashish Varma, Waqar M Naqvi, Salima Mulla, Samana Syed, Sumit Thakur, Sakshi P Arora, Anuj R Varma, Smruti Besekar
Cureus.2022;[Epub] CrossRef
Medical students’ self-assessed efficacy and satisfaction with training on endotracheal intubation and central venous catheterization with smart glasses in Taiwan: a non-equivalent control-group pre- and post-test study
Yu-Fan Lin, Chien-Ying Wang, Yen-Hsun Huang, Sheng-Min Lin, Ying-Ying Yang
Journal of Educational Evaluation for Health Professions.2022; 19: 25. CrossRef

Technical report

Development of examination objectives based on nursing competency for the Korean Nursing Licensing Examination: a validity study: Sujin Shin, Gwang Suk Kim, Jun-Ah Song, Inyoung Lee; J Educ Eval Health Prof. 2022;19:19. Published online August 22, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.19; Funded: Korea Health Personnel Licensing Examination Institute

945 View
166 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to develop the examination objectives based on nursing competency of the Korean Nursing Licensing Examination.
Methods
This is a validity study to develop the examination objectives based on nursing competency. Data were collected in December 2021. We reviewed the literature related to changing nurse roles and on the learning objectives for the Korea Medical Licensing Examination and other health personnel licensing examinations. Thereafter, we created a draft of the nursing problems list for examination objectives based on the literature review, and the content validity was evaluated by experts. A final draft of the examination objectives is presented and discussed.
Results
A total of 4 domains, 12 classes, and 85 nursing problems for the Korean Nursing Liscensing Examination were developed. They included the essentials of objectives, related factors, evaluation goals, related activity statements, related clients, related settings, and specific outcomes.
Conclusion
This study developed a draft of the examination objectives based on clinical competency that were related to the clinical situations of nurses and comprised appropriate test items for the licensing examination. Above results may be able to provide fundamental data for item development that reflects future nursing practices.

Research article

Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study: Janghee Park; J Educ Eval Health Prof. 2022;19:23. Published online August 31, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.23; Funded: Soonchunhyang University Research Fund

1,124 View
135 Download
2 Citations

Abstract PDF Supplementary Material

Purpose
The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE.
Methods
The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results.
Results
The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method.
Conclusion
The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.

Citations

Citations to this article as recorded by

Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
Sun Huh
Journal of Educational Evaluation for Health Professions.2023; 20: 5. CrossRef
Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
Journal of Educational Evaluation for Health Professions.2022; 19: 33. CrossRef

Brief report

Initial steps for integrating academic electronic health records into clinical curricula of physical and occupational therapy in the United States: a survey-based observational study: Stephen Burrows, Lola Halperin, Eric Nemec, Wendy Romney; J Educ Eval Health Prof. 2022;19:24. Published online September 2, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.24; Funded: Sacred Heart University

1,230 View
163 Download

Abstract PDF Supplementary Material: Training programs must be designed to prepare physical and occupational therapy students to use electronic health records (EHR) and interprofessional collaboration. This report aims to describe physical and occupational therapy students’ perceptions of integrating an academic EHR (AEHR) in their problem-based learning (PBL) curricula in the College of Health Professions, Sacred Heart University, Fairfield, Connecticut, the United States. A paper-based case approach to PBL was adapted by creating patient cases in an AEHR. Students were asked to complete chart reviews and review provider notes to enhance their learning. An online survey was conducted to determine their perceptions of using AEHR from May 2014 to August 2015. Eighty-five students completed the survey, and 88.1% felt that using an AEHR was needed, and 82.4% felt that the additional notes enhanced their understanding of the interdisciplinary team. However, 83.5% reported the AEHR system increased the time needed to extract meaningful information. Incorporating an AEHR into curricula is essential to ensure students are adequately prepared for future patient interactions.

Research articles

Medical students’ self-assessed efficacy and satisfaction with training on endotracheal intubation and central venous catheterization with smart glasses in Taiwan: a non-equivalent control-group pre- and post-test study: Yu-Fan Lin, Chien-Ying Wang, Yen-Hsun Huang, Sheng-Min Lin, Ying-Ying Yang; J Educ Eval Health Prof. 2022;19:25. Published online September 2, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.25; Funded: Taipei Veterans General Hospital, Ministry of Education, Ministry of Science and Technology, Taiwan

1,180 View
188 Download

Abstract PDF Supplementary Material: Purpose
Endotracheal intubation and central venous catheterization are essential procedures in clinical practice. Simulation-based technology such as smart glasses has been used to facilitate medical students’ training on these procedures. We investigated medical students’ self-assessed efficacy and satisfaction regarding the practice and training of these procedures with smart glasses in Taiwan.
Methods
This observational study enrolled 145 medical students in the 5th and 6th years participating in clerkships at Taipei Veterans General Hospital between October 2020 and December 2021. Students were divided into the smart glasses or the control group and received training at a workshop. The primary outcomes included students’ pre- and post-intervention scores for self-assessed efficacy and satisfaction with the training tool, instructor’s teaching, and the workshop.
Results
The pre-intervention scores for self-assessed efficacy of 5th- and 6th-year medical students in endotracheal intubation and central venous catheterization procedures showed no significant difference. The post-intervention score of self-assessed efficacy in the smart glasses group was better than that of the control group. Moreover, 6th-year medical students in the smart glasses group showed higher satisfaction with the training tool, instructor’s teaching, and workshop than those in the control group.
Conclusion
Smart glasses served as a suitable simulation tool for endotracheal intubation and central venous catheterization procedures training in medical students. Medical students practicing with smart glasses showed improved self-assessed efficacy and higher satisfaction with training, especially for procedural steps in a space-limited field. Simulation training on procedural skills with smart glasses in 5th-year medical students may be adjusted to improve their satisfaction.

Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study: Song Yi Park, Sang-Hwa Lee, Min-Jeong Kim, Ki-Hwan Ji, Ji Ho Ryu; J Educ Eval Health Prof. 2022;19:26. Published online September 8, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.26; Funded: Busan-Gyeongnam Clinical Skill Examination Consortium

906 View
178 Download

Abstract PDF Supplementary Material: Purpose
This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT).
Methods
This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada).
Results
All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%.
Conclusion
The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study: Ying-Ying Yang, Pin-Hsiang Huang, Ling-Yu Yang, Chia-Chang Huang, Chih-Wei Liu, Shiau-Shian Huang, Chen-Huan Chen, Fa-Yauh Lee, Shou-Yen Kao, Boaz Shulruf; J Educ Eval Health Prof. 2022;19:27. Published online October 17, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.27; Funded: Ministry of Health and Welfare, Ministry of Science and Technology, Taiwan, Ministry of Education, Taiwan Association of Medical Education

825 View
85 Download

Abstract PDF Supplementary Material: Purpose
Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method.
Methods
The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station.
Results
On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts.
Conclusion
The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills.

Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study: Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park; J Educ Eval Health Prof. 2022;19:33. Published online December 12, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.33; Funded: Korea Health Personnel Licensing Examination Institut

314 View
68 Download
1 Citations

Abstract PDF Supplementary Material

Purpose
This study aims to apply the yes/no Angoff and Hofstee methods to actual Korean Medical Licensing Examination (KMLE) 2022 written examination data to estimate cut scores for the written KMLE.
Methods
Fourteen panelists gathered to derive the cut score of the 86th KMLE written examination data using the yes/no Angoff method. The panel reviewed the items individually before the meeting and shared their respective understanding of the minimum-competency physician. The standard setting process was conducted in 5 rounds over a total of 800 minutes. In addition, 2 rounds of the Hofstee method were conducted before starting the standard setting process and after the second round of yes/no Angoff.
Results
For yes/no Angoff, as each round progressed, the panel’s opinion gradually converged to a cut score of 198 points, and the final passing rate was 95.1%. The Hofstee cut score was 208 points out of a maximum 320 with a passing rate of 92.1% at the first round. It scored 204 points with a passing rate of 93.3% in the second round.
Conclusion
The difference between the cut scores obtained through yes/no Angoff and Hofstee methods did not exceed 2% points, and they were within the range of cut scores from previous studies. In both methods, the difference between the panelists decreased as rounds were repeated. Overall, our findings suggest the acceptability of cut scores and the possibility of independent use of both methods.

Citations

Citations to this article as recorded by

Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
Sun Huh
Journal of Educational Evaluation for Health Professions.2023; 20: 5. CrossRef

First
Prev
Page of 8
Next
Last

Funded articles