Journal of Educational Evaluation for Health Professions

Research article

Improving item pool utilization for health professions examinations under variable-length computerized adaptive testing designs: a shadow-test approach: Hwanggyu Lim, Kyung (Chris) Tyek Han; J Educ Eval Health Prof. 2025;22:35. Published online November 3, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.35

5,569 View
217 Download

Abstract PDF Supplementary Material: Purpose
The shadow-test approach to computerized adaptive testing (CAT) ensures content validity in health professions examinations but may suffer from poor item pool utilization in variable-length designs, increasing operational costs and security risks. This study aimed to address this challenge by developing algorithms that enhance the sustainability of shadow CAT in variable-length design.
Methods
A simulation study was conducted to evaluate 3 proposed modifications of the α-stratification method designed to improve item pool utilization. These methods, which integrated randomesque selection and multiple-form strategies, were compared with 2 baseline algorithms within a variable-length shadow CAT framework. Performance was assessed in terms of measurement precision, pool utilization, and test efficiency.
Results
The proposed modifications significantly outperformed the baseline methods across all measures of item pool utilization and exposure control. The most effective method (Modification 2) reduced the proportion of unused items from 35.6% to 5.0% and produced more uniform item exposure rates. These substantial gains in operational sustainability were achieved while maintaining measurement precision comparable to the baseline methods.
Conclusion
The proposed algorithms effectively mitigate poor item pool utilization in shadow CAT under variable-length design. This enhanced framework provides a robust, secure, and sustainable solution for high-stakes adaptive assessments in the health professions that remain content-valid, precise, and operationally efficient.

Software report

Special article on the 20th anniversary of the journal

The irtQ R package: a user-friendly tool for item response theory-based test data analysis and calibration: Hwanggyu Lim, Kyungseok Kang; J Educ Eval Health Prof. 2024;21:23. Published online September 12, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.23

8,898 View
344 Download
5 Web of Science
6 Crossref

Abstract PDF Supplementary Material

Computerized adaptive testing (CAT) has become a widely adopted test design for high-stakes licensing and certification exams, particularly in the health professions in the United States, due to its ability to tailor test difficulty in real time, reducing testing time while providing precise ability estimates. A key component of CAT is item response theory (IRT), which facilitates the dynamic selection of items based on examinees' ability levels during a test. Accurate estimation of item and ability parameters is essential for successful CAT implementation, necessitating convenient and reliable software to ensure precise parameter estimation. This paper introduces the irtQ R package (http://CRAN.R-project.org/), which simplifies IRT-based analysis and item calibration under unidimensional IRT models. While it does not directly simulate CAT, it provides essential tools to support CAT development, including parameter estimation using marginal maximum likelihood estimation via the expectation-maximization algorithm, pretest item calibration through fixed item parameter calibration and fixed ability parameter calibration methods, and examinee ability estimation. The package also enables users to compute item and test characteristic curves and information functions necessary for evaluating the psychometric properties of a test. This paper illustrates the key features of the irtQ package through examples using simulated datasets, demonstrating its utility in IRT applications such as test data analysis and ability scoring. By providing a user-friendly environment for IRT analysis, irtQ significantly enhances the capacity for efficient adaptive testing research and operations. Finally, the paper highlights additional core functionalities of irtQ, emphasizing its broader applicability to the development and operation of IRT-based assessments.

Citations

Citations to this article as recorded by

Development and Evaluation of Computerized Adaptive Testing for General English Proficiency Test
Hyeon Park, Woojae Han, Mikyung Park, Dong gi Seo
The Korean Association of General Education.2026; 20(2): 191. CrossRef
Residual‐Based DIF Detection as a Model‐Agnostic Framework: A Comparison of Parametric, Semi‐Parametric, and Nonparametric Methods
Shan Huang, David Goretzko
Journal of Educational Measurement.2026;[Epub] CrossRef
Psychometric characteristics of Korean Medical Licensing Examination items from 2012 to 2022 analyzed using the Rasch model and 2-parameter logistic model
Dong Gi Seo, Jeongwook Choi, Mee Young Kim, Sun Huh
Journal of Educational Evaluation for Health Professions.2026; 23: 23. CrossRef
Tirt R Package: A Tutorial on Item Response and Testlet Response Theory Modeling
Jiawei Xiong, Cheng Tang, Yanyan Tan, Qidi Liu
Measurement: Interdisciplinary Research and Perspectives.2026; : 1. CrossRef
Development of a CAT based Diagnostic System for Assessing Basic Academic Skills in Undergraduate Students
Woo-Jin Han, Jeongwook Choi, Dong-Gi Seo
The Korean Association of General Education.2025; 19(3): 177. CrossRef
Feasibility of applying computerized adaptive testing to the Clinical Medical Science Comprehensive Examination in Korea: a psychometric study
Jeongwook Choi, Sung-Soo Jung, Eun Kwang Choi, Kyung Sik Kim, Dong Gi Seo
Journal of Educational Evaluation for Health Professions.2025; 22: 29. CrossRef

First
Prev
Page of 1
Next
Last

Search