Journal of Educational Evaluation for Health Professions

Software reports

Introduction to the LIVECAT web-based computerized adaptive testing platform: Dong Gi Seo, Jeongwook Choi; J Educ Eval Health Prof. 2020;17:27. Published online September 29, 2020; DOI: https://doi.org/10.3352/jeehp.2020.17.27

5,343 View
131 Download
3 Web of Science
3 Crossref

This study introduces LIVECAT, a web-based computerized adaptive testing platform. This platform provides many functions, including writing item content, managing an item bank, creating and administering a test, reporting test results, and providing information about a test and examinees. The LIVECAT provides examination administrators with an easy and flexible environment for composing and managing examinations. It is available at http://www.thecatkorea.com/. Several tools were used to program LIVECAT, as follows: operating system, Amazon Linux; web server, nginx 1.18; WAS, Apache Tomcat 8.5; database, Amazon RDMS—Maria DB; and languages, JAVA8, HTML5/CSS, Javascript, and jQuery. The LIVECAT platform can be used to implement several item response theory (IRT) models such as the Rasch and 1-, 2-, 3-parameter logistic models. The administrator can choose a specific model of test construction in LIVECAT. Multimedia data such as images, audio files, and movies can be uploaded to items in LIVECAT. Two scoring methods (maximum likelihood estimation and expected a posteriori) are available in LIVECAT and the maximum Fisher information item selection method is applied to every IRT model in LIVECAT. The LIVECAT platform showed equal or better performance compared with a conventional test platform. The LIVECAT platform enables users without psychometric expertise to easily implement and perform computerized adaptive testing at their institutions. The most recent LIVECAT version only provides a dichotomous item response model and the basic components of CAT. Shortly, LIVECAT will include advanced functions, such as polytomous item response models, weighted likelihood estimation method, and content balancing method.

Citations

Citations to this article as recorded by

Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
Hyunjoo Pai
Journal of Educational Evaluation for Health Professions.2023; 20: 8. CrossRef
Patient-reported outcome measures in cancer care: Integration with computerized adaptive testing
Minyu Liang, Zengjie Ye
Asia-Pacific Journal of Oncology Nursing.2023; 10(12): 100323. CrossRef
Development of a character qualities test for medical students in Korea using polytomous item response theory and factor analysis: a preliminary scale development study
Yera Hur, Dong Gi Seo
Journal of Educational Evaluation for Health Professions.2023; 20: 20. CrossRef

Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece: Kyung (Chris) Tyek Han; J Educ Eval Health Prof. 2018;15:20. Published online August 17, 2018; DOI: https://doi.org/10.3352/jeehp.2018.15.20

28,986 View
244 Download
7 Web of Science
7 Crossref

Abstract PDF Supplementary Material

Computerized adaptive testing (CAT) technology is widely used in a variety of licensing and certification examinations administered to health professionals in the United States. Many more countries worldwide are expected to adopt CAT for their national licensing examinations for health professionals due to its reduced test time and more accurate estimation of a test-taker’s performance ability. Continuous improvements to CAT algorithms promote the stability and reliability of the results of such examinations. For this reason, conducting simulation studies is a critically important component of evaluating the design of CAT programs and their implementation. This report introduces the principles of SimulCAT, a software program developed for conducting CAT simulation studies. The key evaluation criteria for CAT simulation studies are explained and some guidelines are offered for practitioners and test developers. A step-by-step tutorial example of a SimulCAT run is also presented. The SimulCAT program supports most of the methods used for the 3 key components of item selection in CAT: the item selection criterion, item exposure control, and content balancing. Methods for determining the test length (fixed or variable) and score estimation algorithms are also covered. The simulation studies presented include output files for the response string, item use, standard error of estimation, Newton-Raphson iteration information, theta estimation, the full response matrix, and the true standard error of estimation. In CAT simulations, one condition cannot be generalized to another; therefore, it is recommended that practitioners perform CAT simulation studies in each stage of CAT development.

Citations

Citations to this article as recorded by

Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
Hyunjoo Pai
Journal of Educational Evaluation for Health Professions.2023; 20: 8. CrossRef
Students’ perceptions of Computerised Adaptive Testing in higher education
Proya Ramgovind, Shamola Pramjeeth
The Independent Journal of Teaching and Learning.2023; 18(2): 109. CrossRef
Preliminary Development of an Item Bank and an Adaptive Test in Mathematical Knowledge for University Students
Fernanda Belén Ghio, Manuel Bruzzone, Luis Rojas-Torres, Marcos Cupani
European Journal of Science and Mathematics Education.2022; 10(3): 352. CrossRef
Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study
Ioannis Tsaousis, Georgios D. Sideridis, Hannan M. AlGhamdi
Journal of Psychoeducational Assessment.2021; 39(8): 954. CrossRef
Exploring Counselor‐Client Agreement on Clients’ Work Capacity in Established and Consultative Dyads
Uma Chandrika Millner, Diane Brandt, Leighton Chan, Alan Jette, Elizabeth Marfeo, Pengsheng Ni, Elizabeth Rasch, E. Sally Rogers
Journal of Employment Counseling.2020; 57(3): 98. CrossRef
Development of a Computerized Adaptive Testing for Internet Addiction
Yong Zhang, Daxun Wang, Xuliang Gao, Yan Cai, Dongbo Tu
Frontiers in Psychology.2019;[Epub] CrossRef
Updates from 2018: Being indexed in Embase, becoming an affiliated journal of the World Federation for Medical Education, implementing an optional open data policy, adopting principles of transparency and best practice in scholarly publishing, and appreci
Sun Huh
Journal of Educational Evaluation for Health Professions.2018; 15: 36. CrossRef

Review

Components of the item selection algorithm in computerized adaptive testing: Kyung (Chris) Tyek Han; J Educ Eval Health Prof. 2018;15:7. Published online March 24, 2018; DOI: https://doi.org/10.3352/jeehp.2018.15.7

43,833 View
483 Download
13 Web of Science
11 Crossref

Abstract PDF Supplementary Material

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the astratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the KullbackLeibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

Citations

Citations to this article as recorded by

A shortened test is feasible: Evaluating a large-scale multistage adaptive English language assessment
Shangchao Min, Kyoungwon Bishop
Language Testing.2024;[Epub] CrossRef
Efficiency of PROMIS MCAT Assessments for Orthopaedic Care
Michael Bass, Scott Morris, Sheng Zhang
Measurement: Interdisciplinary Research and Perspectives.2024; : 1. CrossRef
The Effects of Different Item Selection Methods on Test Information and Test Efficiency in Computer Adaptive Testing
Merve ŞAHİN KÜRŞAD
Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi.2023; 14(1): 33. CrossRef
Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
Hyunjoo Pai
Journal of Educational Evaluation for Health Professions.2023; 20: 8. CrossRef
Remote Symptom Monitoring With Ecological Momentary Computerized Adaptive Testing: Pilot Cohort Study of a Platform for Frequent, Low-Burden, and Personalized Patient-Reported Outcome Measures
Conrad Harrison, Ryan Trickett, Justin Wormald, Thomas Dobbs, Przemysław Lis, Vesselin Popov, David J Beard, Jeremy Rodrigues
Journal of Medical Internet Research.2023; 25: e47179. CrossRef
Utilizing Real-Time Test Data to Solve Attenuation Paradox in Computerized Adaptive Testing to Enhance Optimal Design
Jyun-Hong Chen, Hsiu-Yi Chao
Journal of Educational and Behavioral Statistics.2023;[Epub] CrossRef
A Context-based Question Selection Model to Support the Adaptive Assessment of Learning: A study of online learning assessment in elementary schools in Indonesia
Umi Laili Yuhana, Eko Mulyanto Yuniarno, Wenny Rahayu, Eric Pardede
Education and Information Technologies.2023;[Epub] CrossRef
Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study
Ioannis Tsaousis, Georgios D. Sideridis, Hannan M. AlGhamdi
Journal of Psychoeducational Assessment.2021; 39(8): 954. CrossRef
Developing Multistage Tests Using D-Scoring Method
Kyung (Chris) T. Han, Dimiter M. Dimitrov, Faisal Al-Mashary
Educational and Psychological Measurement.2019; 79(5): 988. CrossRef
Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece
Kyung (Chris) Tyek Han
Journal of Educational Evaluation for Health Professions.2018; 15: 20. CrossRef
Updates from 2018: Being indexed in Embase, becoming an affiliated journal of the World Federation for Medical Education, implementing an optional open data policy, adopting principles of transparency and best practice in scholarly publishing, and appreci
Sun Huh
Journal of Educational Evaluation for Health Professions.2018; 15: 36. CrossRef

Review article

The impacts of computer adaptive testing from a variety of perspectives: Tetsuo Kimura; J Educ Eval Health Prof. 2017;14:12. Published online May 29, 2017; DOI: https://doi.org/10.3352/jeehp.2017.14.12

39,524 View
375 Download
16 Web of Science
16 Crossref

Abstract PDF

Computer adaptive testing (CAT) is a kind of tailored testing, in that it is a form of computer-based testing that is adaptive to each test-taker’s ability level. In this review, the impacts of CAT are discussed from different perspectives in order to illustrate crucial points to keep in mind during the development and implementation of CAT. Test developers and psychometricians often emphasize the efficiency and accuracy of CAT in comparison to traditional linear tests. However, many test-takers report feeling discouraged after taking CATs, and this feeling can reduce learning self-efficacy and motivation. A trade-off must be made between the psychological experiences of test-takers and measurement efficiency. From the perspective of educators and subject matter experts, nonstatistical specifications, such as content coverage, content balance, and form length are major concerns. Thus, accreditation bodies may be faced with a discrepancy between the perspectives of psychometricians and those of subject matter experts. In order to improve test-takers’ impressions of CAT, the author proposes increasing the target probability of answering correctly in the item selection algorithm even if doing so consequently decreases measurement efficiency. Two different methods, CAT with a shadow test approach and computerized multistage testing, have been developed in order to ensure the satisfaction of subject matter experts. In the shadow test approach, a full-length test is assembled that meets the constraints and provides maximum information at the current ability estimate, while computerized multistage testing gives subject matter experts an opportunity to review all test forms prior to administration.

Citations

Citations to this article as recorded by

The administration of the paper and electronic versions of the Manual Ability Measure-36 (MAM-36) and Fatigue Severity Scale (FSS) is equivalent in people with multiple sclerosis
Andrea Tacchino, Rachele Di Giovanni, Erica Grange, Maria Marcella Spirito, Michela Ponzio, Mario Alberto Battaglia, Giampaolo Brichetto, Claudio Marcello Solaro
Neurological Sciences.2024; 45(3): 1155. CrossRef
The Effect of Computerized Adaptive Testing on Motivation and Anxiety: A Systematic Review and Meta-Analysis
Hanif Akhtar, Silfiasari, Boglarka Vekety, Kristof Kovacs
Assessment.2023; 30(5): 1379. CrossRef
Stealth Literacy Assessments via Educational Games
Ying Fang, Tong Li, Linh Huynh, Katerina Christhilf, Rod D. Roscoe, Danielle S. McNamara
Computers.2023; 12(7): 130. CrossRef
Patient-reported outcome measures in cancer care: Integration with computerized adaptive testing
Minyu Liang, Zengjie Ye
Asia-Pacific Journal of Oncology Nursing.2023; 10(12): 100323. CrossRef
On Measuring Adaptivity of an Adaptive Test
Zhongmin Cui
Measurement: Interdisciplinary Research and Perspectives.2022; 20(1): 21. CrossRef
Performance Decline as an Indicator of Generalized Test-Taking Disengagement
Steven L. Wise, G. Gage Kingsbury
Applied Measurement in Education.2022; 35(4): 272. CrossRef
A content validity evaluation of the PEDI-CAT Speedy Mobility domain
Helene M. Dumas, Maria A. Fragala-Pinkham, Elaine L. Rosen, Pengsheng Ni
Physiotherapy Theory and Practice.2021; 37(4): 517. CrossRef
Item Parameter Estimation in Multistage Designs: A Comparison of Different Estimation Approaches for the Rasch Model
Jan Steinfeld, Alexander Robitzsch
Psych.2021; 3(3): 279. CrossRef
Adaptive Practice Quizzing in a University Lecture: A Pre-Registered Field Experiment
Svenja Heitmann, Niklas Obergassel, Stefan Fries, Axel Grund, Kirsten Berthold, Julian Roelle
Journal of Applied Research in Memory and Cognition.2021; 10(4): 603. CrossRef
Development and Test-Item Analysis of a Freely Available 1900-Item Question Bank for Rheumatology Trainees
Bharat Kumar, Manish Suneja, Melissa L Swee
Cureus.2021;[Epub] CrossRef
Investigating the Viability of Alternating Model Tree As An Item Selection Algorithm for Constructing Computerized Adaptive Psychological Testing
Jeong-Han Youn, Taehun Lee
The Korean Journal of Psychology: General.2021; 40(4): 539. CrossRef
Adaptive practice quizzing in a university lecture: A pre-registered field experiment.
Svenja Heitmann, Niklas Obergassel, Stefan Fries, Axel Grund, Kirsten Berthold, Julian Roelle
Journal of Applied Research in Memory and Cognition.2021; 10(4): 603. CrossRef
Development of an Item Bank to Measure Medication Adherence: Systematic Review
Yu Heng Kwan, Livia Jia Yi Oo, Dionne Hui Fang Loh, Jie Kie Phang, Si Dun Weng, Dan V Blalock, Eng Hui Chew, Kai Zhen Yap, Corrinne Yong Koon Tan, Sungwon Yoon, Warren Fong, Truls Østbye, Lian Leng Low, Hayden Barry Bosworth, Julian Thumboo
Journal of Medical Internet Research.2020; 22(10): e19089. CrossRef
Development of a Computerized Adaptive Version of the Turkish Driving Licence Exam
Nukhet CIKRIKCI, Seher YALÇIN, İlker KALENDER, Emrah GÜL, Cansu AYAN, Gizem UYUMAZ, Merve ŞAHİN KÜRŞAD, Omer KAMIS
International Journal of Assessment Tools in Education.2020; 7(4): 570. CrossRef
Plus ça change, plus c’est pareil: Making a continued case for the use of MCQs in medical education
Debra Pugh, André De Champlain, Claire Touchie
Medical Teacher.2019; 41(5): 569. CrossRef
Review: Patient-reported outcomes in multiple sclerosis care
Emanuele D'Amico, Rocco Haase, Tjalf Ziemssen
Multiple Sclerosis and Related Disorders.2019; 33: 61. CrossRef

Research Article

Estimation of an Examinee's Ability in the Web-Based Computerized Adaptive Testing Program IRT-CAT: Yoon-Hwan Lee, Jung-Ho Park, In-Yong Park; J Educ Eval Health Prof. 2006;3:4. Published online November 22, 2006; DOI: https://doi.org/10.3352/jeehp.2006.3.4

45,852 View
171 Download
5 Crossref

Abstract PDF

We developed a program to estimate an examinee's ability in order to provide freely available access to a web-based computerized adaptive testing (CAT) program. We used PHP and Java Script as the program languages, PostgresSQL as the database management system on an Apache web server and Linux as the operating system. A system which allows for user input and searching within inputted items and creates tests was constructed. We performed an ability estimation on each test based on a Rasch model and 2- or 3-parametric logistic models. Our system provides an algorithm for a web-based CAT, replacing previous personal computer-based ones, and makes it possible to estimate an examinee?占퐏 ability immediately at the end of test.

Citations

Citations to this article as recorded by

Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory
Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee
Keimyung Medical Journal.2023; 42(1): 7. CrossRef
Accuracy and Efficiency of Web-based Assessment Platform (LIVECAT) for Computerized Adaptive Testing
Do-Gyeong Kim, Dong-Gi Seo
The Journal of Korean Institute of Information Technology.2020; 18(4): 77. CrossRef
Computer‐Based Testing and Construction of an Item Bank Database for Medical Education in Korea
Sun Huh
Korean Medical Education Review.2014; 16(1): 11. CrossRef
Can computerized tests be introduced to the Korean Medical Licensing Examination?
Sun Huh
Journal of the Korean Medical Association.2012; 55(2): 124. CrossRef
Application of Computerized Adaptive Testing in Medical Education
Sun Huh
Korean Journal of Medical Education.2009; 21(2): 97. CrossRef

Original Articles

Correlations between the scores of computerized adaptive testing, paper and pencil tests, and the Korean Medical Licensing Examination: Mee Young Kim, Yoon Hwan Lee, Sun Huh; J Educ Eval Health Prof. 2005;2(1):113-118. Published online June 30, 2005; DOI: https://doi.org/10.3352/jeehp.2005.2.1.113

42,540 View
162 Download
3 Crossref

Abstract PDF

To evaluate the usefulness of computerized adaptive testing (CAT) in medical school, the General Examination for senior medical students was administered as a paper and pencil test (P&P) and using CAT. The General Examination is a graduate examination, which is also a preliminary examination for the Korean Medical Licensing Examination (KMLE). The correlations between the results of the CAT and P&P and KMLE were analyzed. The correlation between the CAT and P&P was 0.8013 (p=0.000); that between the CAT and P&P was 0.7861 (p=0.000); and that between the CAT and KMLE was 0.6436 (p=0.000). Six out of 12 students with an ability estimate below 0.52 failed the KMLE. The results showed that CAT could replace P&P in medical school. The ability of CAT to predict whether students would pass the KMLE was 0.5 when the criterion of the theta value was set at -0.52 that was chosen arbitrarily for the prediction of pass or failure.

Citations

Citations to this article as recorded by

Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory
Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee
Keimyung Medical Journal.2023; 42(1): 7. CrossRef
Application of Computerized Adaptive Testing in Medical Education
Sun Huh
Korean Journal of Medical Education.2009; 21(2): 97. CrossRef
Estimation of an Examinee's Ability in the Web-Based Computerized Adaptive Testing Program IRT-CAT
Yoon-Hwan Lee, Jung-Ho Park, In-Yong Park
Journal of Educational Evaluation for Health Professions.2006; 3: 4. CrossRef

Students' Attitude toward and Acceptability of Computerized Adaptive Testing in Medical School and their Effect on the Examinees' Ability: Mee Young Kim, Sun Huh; J Educ Eval Health Prof. 2005;2(1):105-111. Published online June 30, 2005; DOI: https://doi.org/10.3352/jeehp.2005.2.1.105

31,523 View
172 Download
3 Crossref

Abstract PDF

An examinee's ability can be evaluated precisely using computerized adaptive testing (CAT), which is shorter than written tests and more efficient in terms of the duration of the examination. We used CAT for the second General Examination of 98 senior students in medical college on November 27, 2004. We prepared 1,050 pre-calibrated test items according to item response theory, which had been used for the General Examination administered to senior students in 2003. The computer was programmed to pose questions until the standard error of the ability estimate was smaller than 0.01. To determine the students' attitude toward and evaluation of CAT, we conducted surveys before and after the examination, via the Web. The mean of the students' ability estimates was 0.3513 and its standard deviation was 0.9097 (range -2.4680 to +2.5310). There was no significant difference in the ability estimates according to the responses of students to items concerning their experience with CAT, their ability to use a computer, or their anxiety before and after the examination (p>0.05). Many students were unhappy that they could not recheck their responses (49%), and some stated that there were too few examination items (24%). Of the students, 79 % had no complaints concerning using a computer and 63% wanted to expand the use of CAT. These results indicate that CAT can be implemented in medical schools without causing difficulties for users.

Citations

Citations to this article as recorded by

Computer‐Based Testing and Construction of an Item Bank Database for Medical Education in Korea
Sun Huh
Korean Medical Education Review.2014; 16(1): 11. CrossRef
Can computerized tests be introduced to the Korean Medical Licensing Examination?
Sun Huh
Journal of the Korean Medical Association.2012; 55(2): 124. CrossRef
Application of Computerized Adaptive Testing in Medical Education
Sun Huh
Korean Journal of Medical Education.2009; 21(2): 97. CrossRef

First
Prev
Page of 1
Next
Last

Search