Journal of Educational Evaluation for Health Professions

Research article

Improving item pool utilization for health professions examinations under variable-length computerized adaptive testing designs: a shadow-test approach: Hwanggyu Lim, Kyung (Chris) Tyek Han; J Educ Eval Health Prof. 2025;22:35. Published online November 3, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.35

4,772 View
209 Download

Abstract PDF Supplementary Material: Purpose
The shadow-test approach to computerized adaptive testing (CAT) ensures content validity in health professions examinations but may suffer from poor item pool utilization in variable-length designs, increasing operational costs and security risks. This study aimed to address this challenge by developing algorithms that enhance the sustainability of shadow CAT in variable-length design.
Methods
A simulation study was conducted to evaluate 3 proposed modifications of the α-stratification method designed to improve item pool utilization. These methods, which integrated randomesque selection and multiple-form strategies, were compared with 2 baseline algorithms within a variable-length shadow CAT framework. Performance was assessed in terms of measurement precision, pool utilization, and test efficiency.
Results
The proposed modifications significantly outperformed the baseline methods across all measures of item pool utilization and exposure control. The most effective method (Modification 2) reduced the proportion of unused items from 35.6% to 5.0% and produced more uniform item exposure rates. These substantial gains in operational sustainability were achieved while maintaining measurement precision comparable to the baseline methods.
Conclusion
The proposed algorithms effectively mitigate poor item pool utilization in shadow CAT under variable-length design. This enhanced framework provides a robust, secure, and sustainable solution for high-stakes adaptive assessments in the health professions that remain content-valid, precise, and operationally efficient.

Software report

Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece: Kyung (Chris) Tyek Han; J Educ Eval Health Prof. 2018;15:20. Published online August 17, 2018; DOI: https://doi.org/10.3352/jeehp.2018.15.20

33,229 View
269 Download
10 Web of Science
11 Crossref

Computerized adaptive testing (CAT) technology is widely used in a variety of licensing and certification examinations administered to health professionals in the United States. Many more countries worldwide are expected to adopt CAT for their national licensing examinations for health professionals due to its reduced test time and more accurate estimation of a test-taker’s performance ability. Continuous improvements to CAT algorithms promote the stability and reliability of the results of such examinations. For this reason, conducting simulation studies is a critically important component of evaluating the design of CAT programs and their implementation. This report introduces the principles of SimulCAT, a software program developed for conducting CAT simulation studies. The key evaluation criteria for CAT simulation studies are explained and some guidelines are offered for practitioners and test developers. A step-by-step tutorial example of a SimulCAT run is also presented. The SimulCAT program supports most of the methods used for the 3 key components of item selection in CAT: the item selection criterion, item exposure control, and content balancing. Methods for determining the test length (fixed or variable) and score estimation algorithms are also covered. The simulation studies presented include output files for the response string, item use, standard error of estimation, Newton-Raphson iteration information, theta estimation, the full response matrix, and the true standard error of estimation. In CAT simulations, one condition cannot be generalized to another; therefore, it is recommended that practitioners perform CAT simulation studies in each stage of CAT development.

Citations

Citations to this article as recorded by

Balancing exposure and efficiency: a simulation-based comparative analysis of exposure control methods in computerized adaptive testing
Marianna Karagianni, Ioannis Tsaousis
Frontiers in Education.2026;[Epub] CrossRef
From Development to Validation: Exploring the Efficiency of Numetrive, a Computerized Adaptive Assessment of Numerical Reasoning
Marianna Karagianni, Ioannis Tsaousis
Behavioral Sciences.2025; 15(3): 268. CrossRef
Assessing the Potentials of Compurized Adaptive Testing to Enhance Mathematics and Science Student’t Achievement in Secondary Schools
Mary Patrick Uko, I.O. Eluwa, Patrick J. Uko
European Journal of Theoretical and Applied Sciences.2024; 2(4): 85. CrossRef
Implementing Computer Adaptive Testing for High-Stakes Assessment: A Shift for Examinations Council of Lesotho
Musa Adekunle Ayanwale, Julia Chere-Masopha, Mapulane Mochekele, Malebohang Catherine Morena
International Journal of New Education.2024;[Epub] CrossRef
Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
Hyunjoo Pai
Journal of Educational Evaluation for Health Professions.2023; 20: 8. CrossRef
Students’ perceptions of Computerised Adaptive Testing in higher education
Proya Ramgovind, Shamola Pramjeeth
The Independent Journal of Teaching and Learning.2023; 18(2): 109. CrossRef
Preliminary Development of an Item Bank and an Adaptive Test in Mathematical Knowledge for University Students
Fernanda Belén Ghio, Manuel Bruzzone, Luis Rojas-Torres, Marcos Cupani
European Journal of Science and Mathematics Education.2022; 10(3): 352. CrossRef
Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study
Ioannis Tsaousis, Georgios D. Sideridis, Hannan M. AlGhamdi
Journal of Psychoeducational Assessment.2021; 39(8): 954. CrossRef
Exploring Counselor‐Client Agreement on Clients’ Work Capacity in Established and Consultative Dyads
Uma Chandrika Millner, Diane Brandt, Leighton Chan, Alan Jette, Elizabeth Marfeo, Pengsheng Ni, Elizabeth Rasch, E. Sally Rogers
Journal of Employment Counseling.2020; 57(3): 98. CrossRef
Development of a Computerized Adaptive Testing for Internet Addiction
Yong Zhang, Daxun Wang, Xuliang Gao, Yan Cai, Dongbo Tu
Frontiers in Psychology.2019;[Epub] CrossRef
Updates from 2018: Being indexed in Embase, becoming an affiliated journal of the World Federation for Medical Education, implementing an optional open data policy, adopting principles of transparency and best practice in scholarly publishing, and appreci
Sun Huh
Journal of Educational Evaluation for Health Professions.2018; 15: 36. CrossRef

Review

Components of the item selection algorithm in computerized adaptive testing: Kyung (Chris) Tyek Han; J Educ Eval Health Prof. 2018;15:7. Published online March 24, 2018; DOI: https://doi.org/10.3352/jeehp.2018.15.7

48,943 View
573 Download
27 Web of Science
21 Crossref

Abstract PDF Supplementary Material

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the astratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the KullbackLeibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

Citations

Citations to this article as recorded by

Balancing exposure and efficiency: a simulation-based comparative analysis of exposure control methods in computerized adaptive testing
Marianna Karagianni, Ioannis Tsaousis
Frontiers in Education.2026;[Epub] CrossRef
AllTestSim: Comprehensive Software Tool for Simulating Fixed-Form, Linear-on-the-Fly, Multistage, and Computerized Adaptive Testing
Kyung (Chris) T. Han
Applied Psychological Measurement.2026;[Epub] CrossRef
Efficiency of PROMIS MCAT Assessments for Orthopaedic Care
Michael Bass, Scott Morris, Sheng Zhang
Measurement: Interdisciplinary Research and Perspectives.2025; 23(2): 124. CrossRef
Computerized Adaptive Testing Framework Based on Excitation Block and Gumbel-Softmax
Chengsong Liu, Yan Wei
IEEE Access.2025; 13: 3475. CrossRef
From Development to Validation: Exploring the Efficiency of Numetrive, a Computerized Adaptive Assessment of Numerical Reasoning
Marianna Karagianni, Ioannis Tsaousis
Behavioral Sciences.2025; 15(3): 268. CrossRef
Patient reported outcome measures: from the classics to AI
Conrad J Harrison, Ryan W Trickett
Journal of Hand Surgery (European Volume).2025; 50(6): 807. CrossRef
A hybrid model based on learning automata and cuckoo search for optimizing test item selection in computerized adaptive testing
Chanjuan Jin, Weiming Pan
Scientific Reports.2025;[Epub] CrossRef
Maximin criterion for item selection in computerized adaptive testing
Jyun-Hong Chen, Hsiu-Yi Chao
Behavior Research Methods.2025;[Epub] CrossRef
Development and initial validation of DESCAQ: A Rasch‐based computer‐adaptive questionnaire for assessing digital eye strain
Mariano González‐Pérez, Ana Barrio, Rosario Susi, Carlos Pérez‐Garmendia, Beatriz Antona
Ophthalmic and Physiological Optics.2025; 45(6): 1326. CrossRef
Development of a CAT based Diagnostic System for Assessing Basic Academic Skills in Undergraduate Students
Woo-Jin Han, Jeongwook Choi, Dong-Gi Seo
The Korean Association of General Education.2025; 19(3): 177. CrossRef
Utilizing Real-Time Test Data to Solve Attenuation Paradox in Computerized Adaptive Testing to Enhance Optimal Design
Jyun-Hong Chen, Hsiu-Yi Chao
Journal of Educational and Behavioral Statistics.2024; 49(4): 630. CrossRef
A Context-based Question Selection Model to Support the Adaptive Assessment of Learning: A study of online learning assessment in elementary schools in Indonesia
Umi Laili Yuhana, Eko Mulyanto Yuniarno, Wenny Rahayu, Eric Pardede
Education and Information Technologies.2024; 29(8): 9517. CrossRef
A shortened test is feasible: Evaluating a large-scale multistage adaptive English language assessment
Shangchao Min, Kyoungwon Bishop
Language Testing.2024; 41(3): 627. CrossRef
Implementing Computer Adaptive Testing for High-Stakes Assessment: A Shift for Examinations Council of Lesotho
Musa Adekunle Ayanwale, Julia Chere-Masopha, Mapulane Mochekele, Malebohang Catherine Morena
International Journal of New Education.2024;[Epub] CrossRef
The Effects of Different Item Selection Methods on Test Information and Test Efficiency in Computer Adaptive Testing
Merve ŞAHİN KÜRŞAD
Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi.2023; 14(1): 33. CrossRef
Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
Hyunjoo Pai
Journal of Educational Evaluation for Health Professions.2023; 20: 8. CrossRef
Remote Symptom Monitoring With Ecological Momentary Computerized Adaptive Testing: Pilot Cohort Study of a Platform for Frequent, Low-Burden, and Personalized Patient-Reported Outcome Measures
Conrad Harrison, Ryan Trickett, Justin Wormald, Thomas Dobbs, Przemysław Lis, Vesselin Popov, David J Beard, Jeremy Rodrigues
Journal of Medical Internet Research.2023; 25: e47179. CrossRef
Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study
Ioannis Tsaousis, Georgios D. Sideridis, Hannan M. AlGhamdi
Journal of Psychoeducational Assessment.2021; 39(8): 954. CrossRef
Developing Multistage Tests Using D-Scoring Method
Kyung (Chris) T. Han, Dimiter M. Dimitrov, Faisal Al-Mashary
Educational and Psychological Measurement.2019; 79(5): 988. CrossRef
Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece
Kyung (Chris) Tyek Han
Journal of Educational Evaluation for Health Professions.2018; 15: 20. CrossRef
Updates from 2018: Being indexed in Embase, becoming an affiliated journal of the World Federation for Medical Education, implementing an optional open data policy, adopting principles of transparency and best practice in scholarly publishing, and appreci
Sun Huh
Journal of Educational Evaluation for Health Professions.2018; 15: 36. CrossRef

First
Prev
Page of 1
Next
Last

Search