Journal of Educational Evaluation for Health Professions

Research articles

Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination: Janghee Park, Mi Kyoung Yim, Na Jin Kim, Duck Sun Ahn, Young-Min Kim; J Educ Eval Health Prof. 2020;17:28. Published online October 5, 2020; DOI: https://doi.org/10.3352/jeehp.2020.17.28

9,957 View
223 Download
9 Web of Science
8 Crossref

Purpose
The Korea Medical Licensing Exam (KMLE) typically contains a large number of items. The purpose of this study was to investigate whether there is a difference in the cut score between evaluating all items of the exam and evaluating only some items when conducting standard-setting.
Methods
We divided the item sets that appeared on 3 recent KMLEs for the past 3 years into 4 subsets of each year of 25% each based on their item content categories, discrimination index, and difficulty index. The entire panel of 15 members assessed all the items (360 items, 100%) of the year 2017. In split-half set 1, each item set contained 184 (51%) items of year 2018 and each set from split-half set 2 contained 182 (51%) items of the year 2019 using the same method. We used the modified Angoff, modified Ebel, and Hofstee methods in the standard-setting process.
Results
Less than a 1% cut score difference was observed when the same method was used to stratify item subsets containing 25%, 51%, or 100% of the entire set. When rating fewer items, higher rater reliability was observed.
Conclusion
When the entire item set was divided into equivalent subsets, assessing the exam using a portion of the item set (90 out of 360 items) yielded similar cut scores to those derived using the entire item set. There was a higher correlation between panelists’ individual assessments and the overall assessments.

Citations

Citations to this article as recorded by

Standard setting for dental knowledge tests: reproducibility of the modified Angoff and Ebel method across judges
Ting Khee Ho, Noor Lide Abu Kassim, Lucy O’Malley, Reza Vahid Roudsari
BMC Medical Education.2025;[Epub] CrossRef
Angoff methods in standard setting in health professional education: a systematic review and meta-analysis
Kannan Sridharan, Gowri Sivaramakrishnan
BMC Medical Education.2025;[Epub] CrossRef
Application of computer-based testing in the Korean Medical Licensing Examination, the emergence of the metaverse in medical education, journal metrics and statistics, and appreciation to reviewers and volunteers
Sun Huh
Journal of Educational Evaluation for Health Professions.2022; 19: 2. CrossRef
Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study
Janghee Park
Journal of Educational Evaluation for Health Professions.2022; 19: 23. CrossRef
Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study
Ying-Ying Yang, Pin-Hsiang Huang, Ling-Yu Yang, Chia-Chang Huang, Chih-Wei Liu, Shiau-Shian Huang, Chen-Huan Chen, Fa-Yauh Lee, Shou-Yen Kao, Boaz Shulruf
Journal of Educational Evaluation for Health Professions.2022; 19: 27. CrossRef
Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
Journal of Educational Evaluation for Health Professions.2022; 19: 33. CrossRef
Presidential address: Quarantine guidelines to protect examinees from coronavirus disease 2019, clinical skills examination for dental licensing, and computer-based testing for medical, dental, and oriental medicine licensing
Yoon-Seong Lee
Journal of Educational Evaluation for Health Professions.2021; 18: 1. CrossRef
Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea
Song Yi Park, Sang-Hwa Lee, Min-Jeong Kim, Ki-Hwan Ji, Ji Ho Ryu
Journal of Educational Evaluation for Health Professions.2021; 18: 25. CrossRef

Linear programming method to construct equated item sets for the implementation of periodical computer-based testing for the Korean Medical Licensing Examination: Dong Gi Seo, Myeong Gi Kim, Na Hui Kim, Hye Sook Shin, Hyun Jung Kim; J Educ Eval Health Prof. 2018;15:26. Published online October 18, 2018; DOI: https://doi.org/10.3352/jeehp.2018.15.26

23,281 View
294 Download
3 Web of Science
3 Crossref

Abstract PDF Supplementary Material

Purpose
This study aimed to identify the best way of developing equivalent item sets and to propose a stable and effective management plan for periodical licensing examinations.
Methods
Five pre-equated item sets were developed based on the predicted correct answer rate of each item using linear programming. These pre-equated item sets were compared to the ones that were developed with a random item selection method based on the actual correct answer rate (ACAR) and difficulty from item response theory (IRT). The results with and without common items were also compared in the same way. ACAR and the IRT difficulty were used to determine whether there was a significant difference between the pre-equating conditions.
Results
There was a statistically significant difference in IRT difficulty among the results from different pre-equated conditions. The predicted correct answer rate was divided using 2 or 3 difficulty categories, and the ACAR and IRT difficulty parameters of the 5 item sets were equally constructed. Comparing the item set conditions with and without common items, including common items did not make a significant contribution to the equating of the 5 item sets.
Conclusion
This study suggested that the linear programming method is applicable to construct equated-item sets that reflect each content area. The suggested best method to construct equated item sets is to divide the predicted correct answer rate using 2 or 3 difficulty categories, regardless of common items. If pre-equated item sets are required to construct a test based on the actual data, several methods should be considered by simulation studies to determine which is optimal before administering a real test.

Citations

Citations to this article as recorded by

An evaluation of the undergraduate medicine licensure examinations development and implementation practices: lessons from Ethiopia with global implications
Shewatatek Gedamu Wonde, Tefera Tadesse, Stefan K. Schauber
BMC Medical Education.2026;[Epub] CrossRef
Application of computer-based testing in the Korean Medical Licensing Examination, the emergence of the metaverse in medical education, journal metrics and statistics, and appreciation to reviewers and volunteers
Sun Huh
Journal of Educational Evaluation for Health Professions.2022; 19: 2. CrossRef
Reading Comprehension Tests for Children: Test Equating and Specific Age-Interval Reports
Patrícia Silva Lúcio, Fausto Coutinho Lourenço, Hugo Cogo-Moreira, Deborah Bandalos, Carolina Alves Ferreira de Carvalho, Adriana de Souza Batista Kida, Clara Regina Brandão de Ávila
Frontiers in Psychology.2021;[Epub] CrossRef

History Article

Medical licensing examination (uigwa) and the world of the physician officers (uigwan) in Korea’s Joseon Dynasty: Nam Hee Lee; J Educ Eval Health Prof. 2015;12:16. Published online May 7, 2015; DOI: https://doi.org/10.3352/jeehp.2015.12.16

33,146 View
182 Download
2 Web of Science
3 Crossref

Abstract PDF

Physicians for ordinary people in Joseon Dynasty (1392-1910) do not need to pass national medical licensing examination. They had done their job after enough period of apprenticeship. Only physician officers were licensed as technical civil servants. These physician officers were middle class, located socially between the nobility and the commoner. They had to pass a national licensing examination to be considered for high-ranking physician officers, that is, those at the rank above the 6th level out of a total of 9 ranks, where the first rank was highest. Royal physicians also had to pass this examination before accepting responsibility for the King’s healthcare. This article aims to describe the world of Physician officers during the Joseon Dynasty. Physician officers enjoyed considerable social status because they dealt with matters of life and death. Owing to the professional nature of their fields and a strong sense of group identity they came to compose a distinct social class. The physician officers’ world was marked by strong group allegiances based on shared professional knowledge; the use of marriage to gain and maintain social status; and the establishment of hereditary technical posts within the medical profession that were handed down from one generation to the next. The medical licensing examination persisted until 1894 when the civil service examination agency, of which it was part, was abolished. Until that time, the testing agency, the number of candidates who were accepted, two-step test procedures, and the method of test item selection were maintained and enforced. These aspects of the test could be considered characteristic of the medical licensing examination.

Citations

Citations to this article as recorded by

Heo Jun: physician of the people
Hyuk Joon Kwon
Journal of Community Hospital Internal Medicine Perspectives.2021; 11(1): 53. CrossRef
Suturing the Nation in South Korean Historical Television Medical Dramas
Kai Khiun Liew
Journal of Medical Humanities.2020; 41(2): 193. CrossRef
LİYAKAT TEMELLİ BÜROKRASİ: KORE KAMU SINAVLARI (GWAGEO) (958-1894) - THE MERIT-BASED BUREAUCRACY: THE CIVIL SERVICE EXAMINATION (GWAGEO) IN KOREA (958-1894)
Murat KAÇER
Mehmet Akif Ersoy Üniversitesi Sosyal Bilimler Enstitüsü Dergisi.2018; 10(26): 754. CrossRef

Technical Report

Introduction and Administration of the Clinical Skill Test of the Medical Licensing Examination, Republic of Korea (2009): Kun Sang Kim; J Educ Eval Health Prof. 2010;7:4. Published online December 3, 2010; DOI: https://doi.org/10.3352/jeehp.2010.7.4

67,527 View
223 Download
15 Crossref

Abstract PDF

The first trial of the clinical skill test as part of the Korean Medical Licensing Examination was done from September 23 to December 1, 2009, in the clinical skill test center located in the National Health Personnel Licensing Examination Board (NHPLEB) building, Seoul. Korea is the first country to introduce the clinical skill test as part of the medical licensing examination in Asia. It is a report on the introduction and administration of the test. The NHPLEB launched researches on the validity of introducing the clinical skill test and on the best implementation methods in 2000. Since 2006, lists of subjects of test items for the clinical skill test has been developed. The test consisted of two types of evaluation, i.e., a clinical performance examination (CPX) with a standardized patient (SP) and objective structured clinical examination (OSCE). The proctor (medical faculty member) and SP rate the examinees??proficiency for the OSCE and CPX respectively. Out of 3,456 applicants, 3,289 examinees (95.2%) passed the test. Out of 167 examinees who failed the clinical skill test, 142 passed the written test. This means that the clinical skill test showed characteristics independent from the written test. This successful implementation of the clinical skill test is going to improve the medical graduates??performance of clinical skills.

Citations

Citations to this article as recorded by

Comparing AI chatbot simulation and peer role-play for OSCE preparation: a pilot randomized controlled trial
Hye-Yoon Lee, Jundong Kim, Hyojae Choi, Hyojin Bae, Aram Jeong, Sungyoul Choi, Ji-Hwan Kim, Chang-Eop Kim
BMC Medical Education.2025;[Epub] CrossRef
Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
Hyunjoo Pai
Journal of Educational Evaluation for Health Professions.2023; 20: 8. CrossRef
Implementation strategy for introducing a clinical skills examination to the Korean Oriental Medicine Licensing Examination: a mixed-method modified Delphi study
Chan-Young Kwon, Sanghoon Lee, Min Hwangbo, Chungsik Cho, Sangwoo Shin, Dong-Hyeon Kim, Aram Jeong, Hye-Yoon Lee
Journal of Educational Evaluation for Health Professions.2023; 20: 23. CrossRef
Authenticity, acceptability, and feasibility of a hybrid gynecology station for the Papanicolaou test as part of a clinical skills examination in Korea
Ji-Hyun Seo, Younglim Oh, Sunju Im, Do-Kyong Kim, Hyun-Hee Kong, HyeRin Roh
Journal of Educational Evaluation for Health Professions.2018; 15: 4. CrossRef
A one-day surgical-skill training course for medical students’ improved surgical skills and increased interest in surgery as a career
Ho Seok Seo, Yong Hwa Eom, Min Ki Kim, Young-Min Kim, Byung Joo Song, Kyo Young Song
BMC Medical Education.2017;[Epub] CrossRef
Presidential address: launching the Korea Health Personnel Licensing Examination Institute, a government-supported special foundation from December 23, 2015
Chang Hwi Kim
Journal of Educational Evaluation for Health Professions.2016; 13: 20. CrossRef
Reforms of the Korean Medical Licensing Examination regarding item development and performance evaluation
Mi Kyoung Yim
Journal of Educational Evaluation for Health Professions.2015; 12: 6. CrossRef
Educational intervention as an effective step for reducing blood culture contamination: a prospective cohort study
W.B. Park, S.J. Myung, M.-d. Oh, J. Lee, N.-J. Kim, E.-C. Kim, J.S. Park
Journal of Hospital Infection.2015; 91(2): 111. CrossRef
Impact of Clinical Performance Examination on Incoming Interns' Clinical Competency in Differential Diagnosis of Headache
Seong-Min Park, Yun-Mi Song, Bo-Kyoung Kim, Hyoeun Kim
Korean Journal of Family Medicine.2014; 35(2): 56. CrossRef
Can a medical regulatory system be implemented in Korea?
Sun Huh, Myung-Hyun Chung
Journal of the Korean Medical Association.2013; 56(3): 158. CrossRef
Power of the policy: how the announcement of high-stakes clinical examination altered OSCE implementation at institutional level
Chi-Wei Lin, Tsuen-Chiuan Tsai, Cheuk-Kwan Sun, Der-Fang Chen, Keh-Min Liu
BMC Medical Education.2013;[Epub] CrossRef
Can computerized tests be introduced to the Korean Medical Licensing Examination?
Sun Huh
Journal of the Korean Medical Association.2012; 55(2): 124. CrossRef
How can high stakes examination in Korean medical society be improved to the international level?
Sun Huh
Journal of the Korean Medical Association.2012; 55(2): 114. CrossRef
The impact of introducing the Korean Medical Licensing Examination clinical skills assessment on medical education
Hoon-Ki Park
Journal of the Korean Medical Association.2012; 55(2): 116. CrossRef
Failed Examinees' Legal Challenge over the Clinical Skill Test in the Korean Medical Licensing Examination
Sun Huh
Journal of Educational Evaluation for Health Professions.2010; 7: 5. CrossRef

Original Articles

Correlations between the scores of computerized adaptive testing, paper and pencil tests, and the Korean Medical Licensing Examination: Mee Young Kim, Yoon Hwan Lee, Sun Huh; J Educ Eval Health Prof. 2005;2(1):113-118. Published online June 30, 2005; DOI: https://doi.org/10.3352/jeehp.2005.2.1.113

44,713 View
167 Download
3 Crossref

Abstract PDF

To evaluate the usefulness of computerized adaptive testing (CAT) in medical school, the General Examination for senior medical students was administered as a paper and pencil test (P&P) and using CAT. The General Examination is a graduate examination, which is also a preliminary examination for the Korean Medical Licensing Examination (KMLE). The correlations between the results of the CAT and P&P and KMLE were analyzed. The correlation between the CAT and P&P was 0.8013 (p=0.000); that between the CAT and P&P was 0.7861 (p=0.000); and that between the CAT and KMLE was 0.6436 (p=0.000). Six out of 12 students with an ability estimate below 0.52 failed the KMLE. The results showed that CAT could replace P&P in medical school. The ability of CAT to predict whether students would pass the KMLE was 0.5 when the criterion of the theta value was set at -0.52 that was chosen arbitrarily for the prediction of pass or failure.

Citations

Citations to this article as recorded by

Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory
Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee
Keimyung Medical Journal.2023; 42(1): 7. CrossRef
Application of Computerized Adaptive Testing in Medical Education
Sun Huh
Korean Journal of Medical Education.2009; 21(2): 97. CrossRef
Estimation of an Examinee's Ability in the Web-Based Computerized Adaptive Testing Program IRT-CAT
Yoon-Hwan Lee, Jung-Ho Park, In-Yong Park
Journal of Educational Evaluation for Health Professions.2006; 3: 4. CrossRef

A score of trial examination is closely correlated with a score of Korean Medical Licensing Examination 2002: Myoung Soo Kim, Chun-Bae Kim, Byung Ho Cha, Ki Chang Park, Sang Ok Kwon, Kye Chul Shin, Hae Yong Lee, Seong Joon Kang, Bong Suk Cha; J Educ Eval Health Prof. 2004;1(1):87-98. Published online January 31, 2004; DOI: https://doi.org/10.3352/jeehp.2004.1.1.87

34,305 View
159 Download
3 Crossref

Abstract PDF

Korean Medical Licensing Examination(KMLE) 2002 focused on evaluation of the integrative medical knowledge such as primary clinical care or problem-solving competence. We analyzed the correlation among the year-wise student academic scores(grade score), trial examination scores and KMLE score by correlation analysis and multiple regression method. Four times of trial examination were taken in 2001, which were composed according to the principles of KMLE. Trial examination scores were significantly correlated with student grade scores (p<0.05). KMLE score also correlated with student grade score a nd trial examination score. The grade score at senior had higher correlation coefficient than the grade score at junior in correlation analysis. In multiple regressions, grade score at senior and mean score of trial examinatio n score were significant variants affecting KMLE score. Based on this result, regression formula such as [KMLE score] = 110.596 + 21.449*[6th grade score of student] + 0.577*[mean of trial examination score] was established (R2=0.764, p<0.001). Our results show that the trial examination is useful evaluation tool for final assessment of medical achievements. Also a trial examination is used as a reference data for student guidance and control in preparing for KMLE.

Citations

Citations to this article as recorded by

North Korean refugee doctors' preliminary examination scores
Sung Uk Chae, Jeong Hee Yang, Joon Seop Hyun, June Hee Kim, Seok Hoon Kang
Korean Journal of Medical Education.2016; 28(4): 373. CrossRef
A Six-Year Study of Relationship between Academic Performance in Dental Hygiene School and Performance on the Korean Dental Hygiene Licensing Examination at Yonsei University
So-Jung Mun, Hie-Jin Noh, Hyun-Sun Jeon, Ji-Eun Heo, Won-Gyun Chung
Journal of dental hygiene science.2014; 14(3): 332. CrossRef
The Relationship between Senior Year Examinations at a Medical School and the Korean Medical Licensing Examination
Ki Hoon Jung, Ho Keun Jung, Kwan Lee
Korean Journal of Medical Education.2009; 21(1): 17. CrossRef

A study on the satisfaction of medical licensing examination and the present condition of skill test in medical schools: Jang Hee Park, Un Mook Kim, Won Chul Lee, Yoon Seong Lee; J Educ Eval Health Prof. 2004;1(1):77-86. Published online January 31, 2004; DOI: https://doi.org/10.3352/jeehp.2004.1.1.77

28,464 View
166 Download
2 Crossref

Abstract PDF

lt has been an issue whether the current medical examination system can evaluate medical students' competencies efficiently. This study was performed to survey on the satisfaction for the current medical examination system and present situation for clinical skill test in medical schools. We conducted a survey for this research and the subjects of this study were deans, medical professors, resident and medical students. We met with interesting results. First, most respondents answered the current medical examination system couldn't evaluate the medical students' competencies efficiently. Second, many residents thought preparing for paper-pencil test was not helpful for training, while experiencing clinical skill test was helpful for it. Third, the current contents and methods to evaluate clinical skill in the medical schools were variable and desirable. We concluded it was high time to change our medical examination system for evaluating the clinical skill performance of medical students.

Citations

Citations to this article as recorded by

The impact of introducing the Korean Medical Licensing Examination clinical skills assessment on medical education
Hoon-Ki Park
Journal of the Korean Medical Association.2012; 55(2): 116. CrossRef
Analysis of First Clinical Skills Examination in the Korean Medical Licensing Examination: Focus on Examinees' Experience in a Medical School
Kyung Ae Jun, Sang Yop Shin
Korean Journal of Medical Education.2011; 23(3): 203. CrossRef

First
Prev
Page of 1
Next
Last

Search