Application of computer-based testing in the Korean Medical Licensing Examination, the emergence of the metaverse in medical education, journal metrics and statistics, and appreciation to reviewers and volunteers

Article information

J Educ Eval Health Prof. 2022;19.2
Publication date (electronic) : 2022 January 13
doi : https://doi.org/10.3352/jeehp.2022.19.2
Department of Parasitology and Institute of Medical Education, College of Medicine, Hallym University, Chuncheon, Korea
Editor: A Ra Cho, The Catholic University of Korea, Korea
*Corresponding email: shuh@hallym.ac.kr
Received 2022 January 12; Accepted 2022 January 13.

Computer-based testing adopted for the Korean Medical Licensing Examination on January 6–7, 2022

On January 6–7, 2022, computer-based testing (CBT) was adopted for the Korean Medical Licensing Examination (KMLE) [1] (Fig. 1). This is the first instance of CBT adoption for any of the 26 health professional licensing examinations in Korea. Of course, smart device-based testing (SBT), a variant of CBT, was already adopted for the Korea Emergency Medicine Technician Licensing Examination in 2017 [2]. Examinees’ demographic characteristics and perceived acceptability of SBT did not affect the SBT mock test scores of emergency medicine technician students in Korea [3]. After implementing CBT in the KMLE, it will be expanded to all other 24 professions, including dentists, oriental medicine doctors, and care workers, by 2025 [1].

Fig. 1.

Screenshot of the mock test of computerized adaptive testing of the Korean Medical Licensing Examination in 2021, provided by the Korea Health Personnel Licensing Examination Institute.

On March 21, 2003, presenters first proposed CBT and computerized adaptive testing (CAT) for the KMLE at a seminar held by the Institute of Medical Education of Hallym University at the Chuncheon Sejong Hotel. At that time, the staff members of the Korea Health Personnel Licensing Examination Institute (KHPLEI) attended the seminar. Thus, 19 years have passed since CBT and CAT were suggested for licensing examinations in Korea.

In November 2011, the Standing Committee of the KMLE recommended introducing computerized testing to the KMLE, including CBT and CAT [4]. I also suggested improving the quality of the KMLE through the introduction of computerized testing and recommended that medical schools should prepare for the new testing environment of the KMLE [4]. The 6th president of the KHPLEI, Dr. Myung-Hyun Chung (August 1, 2012–July 31, 2015), announced that research and development on CBT and ubiquitous-based testing would be accelerated in January 2013 [5]. Research on CBT then became more active. For example, the linear programming method was suggested to construct equated-item sets that reflect each content area. The best choice was to divide the predicted correct answer rate into 2 or 3 difficulty boundaries regardless of common items [6]. Because SBT was already well implemented in the Korea Emergency Medicine Technician Licensing Examination [3], there was no need to repeat an evaluation of the examinees’ adaptability to CBT. The only step left was decision-making by the president of KHPLEI. Dr. Yoon-Seong Lee, the 8th president, finally decided to introduce CBT.

CBT has already been a very common testing platform at most medical schools. For example, at Hallym University, CBT was introduced in 1999 for clinical course testing. In my parasitology class, CBT was already introduced in 1993. My students have had no difficulties in taking CBT. A new task after adopting CBT is to guarantee item quality, including that of audio-visual materials. CBT items should be more clinically oriented than paper-and-pencil materials to test examinees’ clinical competency. Furthermore, the number of items to measure the examinee’s latent traits should be re-considered. A correct distribution of core clinical content is required, and the balance between reliability and core clinical presentations should be guaranteed to ensure desirable item numbers. The standard setting for CBT, including the modified Angoff, modified Ebel, and Hofstee methods, should be implemented soon, instead of using a cut score to ensure a success rate of 60% [7,8]. It is necessary to maintain a consistent passing rate regardless of item difficulties.

Study size estimation and arrival of the metaverse in medical training in Korea

Two invited reviews were published last year: “Sample size determination and power analysis using the G*Power software” by Kang [9] and “Educational applications of metaverse: possibilities and limitations” by Kye et al. [10]. There was also an invited editorial: “Training in lung cancer surgery through the metaverse, including extended reality (XR), in the smart operating room of Seoul National University Bundang Hospital, Korea” by Koo [11].

Although Dr. Kang is an anesthesiologist, he studied statistics after graduating from Korea National Open University with a major in Statistics. I invited him to serve as a statistical editor in 2021. I asked him to submit a review on determining the study size for authors and reviewers of the journal. When I read manuscripts submitted to the journal, the most challenging task is to check the appropriateness of study size estimation, especially since many manuscripts do not describe it. If estimation before the study was not possible, it is recommended to conduct a post-hoc analysis. Dr. Kang stated that “the null and alternative hypothesis, effect size, power, alpha, type I error, and type II error should be described when calculating the sample size or power” [9]. I hope that this invited methodological review will be helpful for authors and researchers.

The year 2021 was the year of the metaverse [12]. Although the term “metaverse” was known beforehand, and it had already been developed as a technology, its use exploded in the coronavirus disease 2019 (COVID-19) pandemic era since many people could not meet face to face. This also occurred in health education. My university also trained faculty members on how to use Gather, one of the virtual educational spaces of metaverse platforms, called the virtual world. Unfortunately, I could not apply this platform for my medical students in 2021 due to my unpreparedness. I will use this platform or another virtual educational space for this year’s parasitology class. Several types of the metaverse, including augmented reality (AR), mirror world, lifelogging, and virtual reality (VR), have already been introduced in medical and health education.

I did a PubMed search with the keyword “metaverse,” and only 11 articles were found on January 8, 2022. However, this does not mean that the number of articles on metaverse is small. Before that, articles on the 4 types of the metaverse have been published frequently. These publications soared in 2021 (Fig. 2, Supplement 1) and the term “metaverse” began to appear in biomedical articles in 2020 and 2021. There are already 4 articles on this topic in the Journal of Educational Evaluation for Health Professions (JEEHP): VR endotracheal intubation training [13]; digital technologies introduced in medical and dental education [14]; a simulation-based blended training model for nurses [15], and VR training to decreases rates of needle stick/sharp injuries [16].

Fig. 2.

Number of articles on the 4 types of the metaverse in PubMed by year with the search term “(((augmented reality) OR (lifelogging)) OR (mirror world)) OR (virtual reality)” [cited 2022 Jan 8].

According to the usage trends of the term “metaverse,” a review article was invited as an abridged English translation of the issue report by the Korea Education and Research Information Service [10]. Dr. Bokyung Kye, Director of the Global Policy Research Section, kindly provided this report for the journal’s readers. This review article is a broad introduction to the educational application of the metaverse. The definition of the relevant terminology was explained, and current examples of metaverse use were introduced. It also included applications in health education. I hope that this accessible and comprehensive review article will be helpful for medical and health educators to understand the concept of the metaverse and the strengths and limitations of its application. Furthermore, Koo [11], a senior reporter from JTBC, one of the well-known broadcasting companies in Korea, wrote an editorial on the use of the metaverse to train chest surgeons at Seoul National University Bundang Hospital. This training course was an example of XR implemented by mixing AR and VR. XR refers to all VR technologies ranging from VR to mixed reality and AR.

Journal metrics and statistics

In 2021, JEEHP received its first Journal Citation Indicator (JCI) value, 0.51, in June 2021. This score means that the citation impact of articles from 2017 to 2019 was about half of the average citation impact of Web of Science Core Collection journals. JEEHP’s JCI ranking in the scientific education category was 9th out of 35 Emerging Sources Citation Index (ESCI) journals (74.3%) and 47th out of 78 SCIE (Science Citation Index Expanded) and ESCI journals (39.8%) [17]. To receive a higher JCI value, a higher citation frequency of citable articles will be essential. The 2021 JCI may be announced in June 2022 by Clarivate. I anticipate a higher 2021 JCI than that in 2020.

Bibliometric statistics for the author’s country and total cites are presented in Figs. 3 and 4 (Supplement 2), respectively. There were authors from 15 countries, mainly from Asia, which reflects the regional scope of JEEHP. The number of total cites continuously increased. In 2021, JEEHP was cited 525 times in the Crossref metadata, 555 times in Scopus, and 461 times in Web of Science Core Collection. The manually calculated impact factor in Web of Science increased to 1.846 in 2021 from 1.254 in 2020. The CiteScore Tracker 2021 calculated by Scopus was 2.5 (366 citations 2018 to date/145 documents 2018 to data), last updated on January 5, 2022, which is available from: https://www.scopus.com/sourceid/21100467423. This value is higher than that in 2020 (1.7)

Fig. 3.

Number of articles in Journal of Educational Evaluation for Health Professions according to the authors’ country in 2021.

Fig. 4.

Total cites of Journal of Educational Evaluation for Health Professions in Crossref metadata, Scopus, and the Web of Science Core Collection (WOS) from 2015 to 2021.

Table 1 presents journal statistics from 2021. The number of unsolicited submissions (n=296) increased from 2020 (n=275). However, the number of publications decreased from 44 in 2020 to 33 in 2021. This primarily originates from the low number of commissioned articles in 2021. The acceptance rate of unsolicited manuscripts (8.9%) was lower than that in 2002 (10.6%). In 2020, the first year of COVID-19, the number of submissions (n=286) was nearly double compared to 2019 (n=147). This sharp increase was a common phenomenon seen in other scholarly journals in Korea [18]. The amount of submissions to this journal was consistent in 2021.

Journal statistics of manuscripts submitted to Journal of Educational Evaluation for Health Professions from January 1 to December 31, 2021

I should be cautious when selecting manuscripts for review. The number of rejected or withdrawn manuscripts after review was 16 in 2020 and 10 in 2021. The number of accepted manuscripts after review was 26 out of 43 (70.3%) in 2021. In 2022, the editorial office will do its best to select acceptable manuscripts for review to save reviewers’ time.

Appreciation to reviewers and volunteers

I am deeply indebted to the reviewers who voluntarily devoted themselves to reviewing manuscripts. Their role is essential for maintaining the journal quality. I regret that I sent them some manuscripts that were finally rejected. Below are the reviewers’ names and affiliations by country.

  • Australia: Boaz Shulruf, University of New South Wales; Elio Stefan Arruzza, University of South Australia

  • Chile: Castillo Niño Manuel, University of Chile

  • Indonesia: Armyanti Ita, Tanjungpura University; Romiko, Muhammadiyah University of Palembang

  • India: Upreet Dhaliwal, University of Delhi; Kiran Goswami, All India Institute of Medical Sciences; T. S. Gugapriya, All India Institute of Medical Science, Manjiri Phansalkar, Pondicherry Institute of Medical Sciences

  • Israel: Colin Block, The Hebrew University of Jerusalem

  • Italy: Colaceci Sofia, Saint Camillus International University of Health Sciences

  • Korea: A Ra Cho, The Catholic University of Korea; Yera Hur, Hallym University; Junyong In, Dongguk University; Hyun Kang, Chung-Ang University; Jae Hyun Kim, Dankook University; Mi Kyoung Yim, Korea Health Personnel Licensing Examination Institute; Sun Kim, The Catholic University of Korea; Young Ju Kim, Ewha Womans University; Dong Kyu Lee, Dongguk University; Keumho Lee, Korea Institute for Research in the Behavioral Sciences; Younjae Oh, Hallym University; Janghee Park, Soonchunhyang University; Jeong Yun Park, University of Ulsan; Jungchul Park, Dankook University; Jungkyu Park, Kyungpook National University; Kyung Hye Park, Yonsei University; Song Yi Park, Dong-A University; Won Kyun Park, Keimyung University; Dong Gi Seo, Hallym University; Ji-Hyun Seo, Gyeongsang National University; Aeree Son, Samyook University; Sanghee Yeo, Kyungpook National University; Hyun Bae Yoon, Seoul National University; Dong-Mi Yoo, The Catholic University of Korea

  • Malaysia: Fata Nahas Abdulrahman, International Islamic University; Jamshid Shazia, Sultan Zainal Abidin University

  • Morocco: Hassouni Kenza, Mohammed VI University of Health Sciences; El Hajji Mohamed, Ibn Zohr University

  • Pakistan: Rano Mal Piryani, Liaquat University of Medical and Health Sciences

  • Portugal: José Miguel Padilha, Oporto Nursing School

  • South Africa: Richard Cooke, University of the Witwatersrand

  • Thailand: Poonpong Boonbrahm, Walailak University; Marisa Krairiksh, Khon Kaen University; Manoch Namfu, Rajamangala University of Technology Lanna

  • United Kingdom: Oliver Kemp, Gloucestershire NHS Foundation Trust; Marios Nicolaides, Queen Mary University of London; Cesar A. Orsini, Norwich Medical School, University of East Anglia; Emily Suckling, University Hospitals Bristol and Weston NHS Foundation Trust

  • United States of America: Chad Cook, Duke University; Robert Cunningham, Maryville University; Mariana D’amico, Nova Southeastern University; Jessica Fino, Midwestern State University; Sandra Groth, Midwestern State University; Junguk Hur, University of North Dakota; Myunghee Jun, University of Wisconsin-Green Bay; Hyekyung Kim, University of Wisconsin-Green Bay; Seock-Ho Kim, The University of Georgia; Amteshwar Singh, Johns Hopkins University; Jessyca Wagner, Midwestern State University

  • Vietnam: Bach Xuan Tran, Ha Noi Medical University

Tom Huh, a graduate student in the Division of Life Sciences, College of Life Sciences and Biotechnology, Korea University, Seoul, Korea, voluntarily made audio recordings of some abstracts.

Which journal policies will be emphasized in 2022

In the screening process of a submitted manuscript, the fitness of the manuscript in terms of both aims and scope and style and format will be meticulously checked before dispatching it for review. Reviewers or editors cannot determine originality; this determination is dependent on readers’ experience. Therefore, the review has been focused on scientific integrity. A description according to reporting guidelines will be more strictly required. It should be demonstrated that the study design follows specific reporting guidelines. JEEHP is a non-commercial scholarly journal, so the number of published articles is not critical—but scientific integrity is essential. The editorial team and I will do our best to maintain these policies in 2022 to save reviewers’ time.

Notes

Authors’ contributions

All the work was done by Sun Huh.

Conflict of interest

Sun Huh has been the editor of the Journal of Educational Evaluation for Health Professions since 2005. Otherwise, no potential conflict of interest relevant to this article was reported.

Funding

None.

Data availability

None.

Acknowledgements

None.

Supplementary materials

Supplement 1. Number of articles on the 4 types of the metaverse in PubMed by year with the search term “(((augmented reality) OR (lifelogging)) OR (mirror world)) OR (virtual reality)” [cited 2022 Jan 8], presented in Fig. 1.

Supplement 2. Total cites of Journal of Educational Evaluation for Health Professions in Crossref metadata, Scopus, and the Web of Science Core Collection (WOS) from 2015 to 2021, presented in Fig. 4.

References

1. Lee YS. Presidential address: Adoption of clinical skills examination for dental licensing, execution of computer-based testing for the medical licensing examination, and the 30th anniversary of the Korea Health Personnel Licensing Examination Institute. J Educ Eval Health Prof 2022;19:1. https://doi.org/10.3352/jeehp.2022.19.1.
2. Kim CH. Presidential address: introduction of smart device-based testing and item exposure policy for Korea Health Personnel Licensing Examination. J Educ Eval Health Prof 2017;14:5. https://doi.org/10.3352/jeehp.2017.14.5.
3. Lim EY, Yim MK, Huh S. The relationship of individual examinee characteristics and acceptability of smart device-based testing to test score in the practice test of the Korea Emergency Medicine Technician Licensing Examination. J Educ Eval Health Prof 2018;15:33. https://doi.org/10.3352/jeehp.2018.15.33.
4. Huh S. Can computerized tests be introduced to the Korean Medical Licensing Examination? J Korean Med Assoc 2012;55:124–130. https://doi.org/10.5124/jkma.2012.55.2.124.
5. Chung MH. President’s address: Research and development on computer-based testing and extension of clinical skill examination to a variety of fields. J Educ Eval Health Prof 2013;10:1. https://doi.org/10.3352/jeehp.2013.10.1.
6. Seo DG, Kim MG, Kim NH, Shin HS, Kim HJ. Liner programming method to construct equated item sets for implementation of periodical computer-based testing of Korean medical licensing examination. J Educ Eval Health Prof 2018;15:26. https://doi.org/10.3352/jeehp.2018.15.26.
7. Park J, Yim MK, Kim NJ, Ahn DS, Kim YM. Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination. J Educ Eval Health Prof 2020;17:28. https://doi.org/10.3352/jeehp.2020.17.28.
8. Yim MK, Shin S. Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination. J Educ Eval Health Prof 2020;17:14. https://doi.org/10.3352/jeehp.2020.17.14.
9. Kang H. Sample size determination and power analysis using the G*Power software. J Educ Eval Health Prof 2021;18:17. https://doi.org/10.3352/jeehp.2021.18.17.
10. Kye B, Han N, Kim E, Park Y, Jo S. Educational applications of metaverse: possibilities and limitations. J Educ Eval Health Prof 2021;18:32. https://doi.org/10.3352/jeehp.2021.18.32.
11. Koo H. Training in lung cancer surgery through the metaverse, including extended reality, in the smart operating room of Seoul National University Bundang Hospital, Korea. J Educ Eval Health Prof 2021;18:33. https://doi.org/10.3352/jeehp.2021.18.33.
12. Waters R. Year in a word: metaverse. Financial Times [Internet]. 2021. Dec. 22. [cited 2022 Jan 11]. Available from: https://www.ft.com/content/8903afb9-5206-4994-8bfd-e74b8f20899e.
13. Khundam C, Sukkriang N, Noel F. No difference in learning outcomes and usability between using controllers and hand tracking during a virtual reality endotracheal intubation training for medical students in Thailand. J Educ Eval Health Prof 2021;18:22. https://doi.org/10.3352/jeehp.2021.18.22.
14. Park JC, Kwon HE, Chung CW. Innovative digital tools for new trends in teaching and assessment methods in medical and dental education. J Educ Eval Health Prof 2021;18:13. https://doi.org/10.3352/jeehp.2021.18.13.
15. Huang CC, Huang SS, Yang YY, Kao SY. Effects of a simulation-based blended training model on nurses’ treatment decision-related knowledge about oral cancer in Taiwan: a pilot survey. J Educ Eval Health Prof 2021;18:10. https://doi.org/10.3352/jeehp.2021.18.10.
16. Wu SH, Huang CC, Huang SS, Yang YY, Liu CW, Shulruf B, Chen CH. Effect of virtual reality training to decreases rates of needle stick/sharp injuries in new-coming medical and nursing interns in Taiwan. J Educ Eval Health Prof 2020;17:1. https://doi.org/10.3352/jeehp.2020.17.1.
17. Huh S. The Journal Citation Indicator has arrived for Emerging Sources Citation Index journals, including the Journal of Educational Evaluation for Health Professions, in June 2021. J Educ Eval Health Prof 2021;18:20. https://doi.org/10.3352/jeehp.2021.18.20.
18. Huh S. Was the number of submissions to scholarly journals in Korea affected by the COVID-19 pandemic? Sci Ed 2021;8:117–122. https://doi.org/10.6087/kcse.239.

Article information Continued

Fig. 1.

Screenshot of the mock test of computerized adaptive testing of the Korean Medical Licensing Examination in 2021, provided by the Korea Health Personnel Licensing Examination Institute.

Fig. 2.

Number of articles on the 4 types of the metaverse in PubMed by year with the search term “(((augmented reality) OR (lifelogging)) OR (mirror world)) OR (virtual reality)” [cited 2022 Jan 8].

Fig. 3.

Number of articles in Journal of Educational Evaluation for Health Professions according to the authors’ country in 2021.

Fig. 4.

Total cites of Journal of Educational Evaluation for Health Professions in Crossref metadata, Scopus, and the Web of Science Core Collection (WOS) from 2015 to 2021.

Table 1.

Journal statistics of manuscripts submitted to Journal of Educational Evaluation for Health Professions from January 1 to December 31, 2021

No. of manuscripts Content
Manuscripts submitted 302
No. of commissioned manuscripts 6 Editorial, 4; review, 2
No. of unsolicited manuscripts 296
Manuscripts under re-submission, review, or revision 6 Under re-submission, 4; under review, 2
Manuscripts rejected without review 253 Unsuitable, 250; other reasons, 3
Manuscripts reviewed out of 290 unsolicited manuscripts 37 Accepted and published, 26; rejected 10; withdrawn, 1
No. of publications out of 296 submitted manuscripts in 2021 32 One article published in 2021 was submitted in 2020
No. of publications out of 290 unsolicited manuscripts 26 Opinion, 1; review, 4; research article, 14; brief report, 5; educational/faculty development material, 2
Acceptance rate overall (%) 10.8 32/296=0.108
Acceptance rate of unsolicited manuscripts (%) 8.9 26/290=0.089
Median time from submission to the first decision (day) 15
Median time from submission to publication (day) 43
Median time from acceptance to publication (day) 1