Graphical abstract

Introduction

Background/rationale

To reduce the gap between theory and practice in nursing education and support newly licensed nurses’ transition into clinical practice, it is essential to enhance clinical judgment and practical competence [1]. However, these competencies are difficult to assess adequately using written examinations alone. For example, memorizing abnormal breath sounds does not ensure the ability to auscultate lung sounds accurately or make clinical judgments in real situations. Accordingly, simulation-based and technology-enhanced educational approaches have been increasingly adopted in nursing education [2,3]. In contrast, systematic approaches to developing and validating assessment tools that reflect these competencies, particularly those incorporating technology-based item formats, remain limited.

In South Korea, the Korean Health Personnel Licensing Examination Institute (KHPLEI) has established task forces, plans to revise relevant regulations, and intends to implement a phased rollout with the goal of introducing a computer-based testing (CBT) format for the Korean Nursing Licensing Examination (KNLE) by 2028 [4]. The transition to CBT represents not only a change in test delivery but also an opportunity to use diverse item formats that better reflect real clinical situations. This transition may enhance job-related validity and improve item discrimination and reliability through the development of more valid test items. In addition, computer-based assessment allows the use of multimedia, such as videos, audio, and high-resolution images, that are difficult to implement in paper-based tests, thereby diversifying item presentation formats [5]. Multimedia-assisted items (MAIs) can represent authentic problem situations more effectively and convey meaning more efficiently than text alone, enabling more accurate measurement of learning outcomes [6]. Previous studies have also reported improvements in knowledge and performance when multimedia materials were used in education and assessment, supporting the effectiveness of computer-based assessment [7].

The United States nursing licensure examination (National Council Licensure Examination for Registered Nurses) uses computerized adaptive testing (CAT), in which item selection is based on examinees’ responses. Recognizing clinical judgment as a core competency for newly licensed nurses, the National Council of State Boards of Nursing launched the Next Generation NCLEX project and developed the Clinical Judgment Measurement Model as a systematic assessment framework [8]. In South Korea, CBT has also been introduced into national health professional licensure examinations, beginning with the medical licensing examination in 2022 and expanding to other health professions in 2023 [9]. These developments highlight the growing need for the KNLE to adopt assessment formats that can evaluate job-related competencies using MAIs.

High-quality items assessing clinical judgment can be selected from existing nursing licensure examination items and used to develop prototypes of case-based MAIs. In a study by Lee and Shin [10], job-based MAIs were developed and validated, and examinees in a mock examination rated these items as more closely related to actual job performance than traditional paper-based items. Similarly, Jang and Suh [11] developed a mobile-based, multimedia-utilized Nursing Competency Evaluation (NCE) system, and the experimental group reported significantly greater effectiveness in terms of convenience, benefits, usefulness, and satisfaction than the control group in a mock examination. These findings suggest that MAIs may enhance job relevance and assessment authenticity in nursing competency evaluation. They also indicate the value of examining the development and preliminary application of MAIs that reflect authentic clinical contexts within the KNLE.

In summary, the transition to CBT provides an assessment environment in which MAIs can be implemented, whereas MAIs offer an item-level approach to representing complex clinical information relevant to nursing practice. MAIs can be conceptualized as items designed to reflect clinical situations and practice-relevant information when assessing clinical judgment competency. Furthermore, estimating psychometric parameters through item response theory (IRT)-based analysis of recent examination items, alongside the development and pilot testing of multimedia items, is essential for evaluating the feasibility of CAT.

Therefore, this study aimed to administer a mock examination including MAIs and evaluate their validity by analyzing the examination results. Ultimately, the findings are intended to provide foundational evidence to support the transition to a computer-based KNLE.

Objectives

The purpose of this study was to administer a mock examination including MAIs and paired text-based items with identical content to nursing students and to evaluate the validity of the MAIs by analyzing the examination results in preparation for the transition to a computer-based KNLE.

Methods

Ethics statement

This study was conducted in accordance with research ethics principles and was approved by the Institutional Review Board (IRB) of the authors’ affiliated institution (IRB no., ewha-202511-0004-01). Informed consent was obtained from all participants before participation. Focus group interviews were audio-recorded with participants’ consent and transcribed verbatim, and the audio files were permanently deleted immediately after transcription. The study was conducted without compromising participant safety, and all collected data were used solely for research purposes and anonymized to protect participants’ privacy.

Study design

This study used a methodological design with a within-group counterbalanced approach. Two mock examination formats were administered to nursing students, and psychometric item characteristics, survey responses, and focus group interview data were collected and analyzed. A mixed-methods approach was adopted to obtain both psychometric evidence and response-process validity evidence. Quantitative analyses were used to examine the measurement properties of MAIs compared with text-based items, whereas survey and focus group interview data were used to explore examinees’ perceptions, cognitive responses, and problem-solving experiences. The findings were integrated during interpretation to evaluate MAIs in terms of psychometric performance, clinical authenticity, and examinee response processes.

Development of multimedia items

The multimedia items used in this study were developed through a stepwise development and validation process before the mock examination. First, focus group interviews and an item development workshop were conducted with 15 nursing faculty members to review items from the 61st to 63rd KNLEs (2021–2023) and identify items suitable for transformation into multimedia formats. Next, a priority evaluation was conducted with 10 nursing faculty members, resulting in the selection of 29 items identified as high priorities for MAI development. Five similar items, defined as those sharing identical or highly similar materials and overlapping core concepts or correct answers, were excluded after consultation with a subject matter expert. Five new items were additionally developed to include newly created MAIs beyond transformed existing KNLE items, yielding a final set of 29 items. To examine content validity, 10 newly licensed nurses evaluated each item using a 4-point Likert scale on 2 dimensions: (1) whether the MAI better assessed job-related competence compared with the text-based item and (2) whether the MAI better reflected clinical practice. The content validity index (CVI) was calculated as the proportion of responses rated 3 or 4, with a CVI of ≥0.78 considered acceptable. All items achieved a CVI of 1.00 for clinical practice reflection. For job-related competence assessment, 27 of the 29 items achieved a CVI of ≥0.78; the remaining 2 items yielded a CVI of 0.70. After an intensive research team meeting, these 2 items were retained in the final item set because their inclusion was not expected to substantially affect overall examination feasibility. A pilot test was conducted with 10 nursing students to assess item clarity. Each item was rated on a 4-point Likert scale for difficulty in understanding or solving the item. All 29 items had mean scores ranging from 1.10 to 1.80 (mean=1.34±0.62), with no item exceeding 2.00, indicating that examinees experienced no substantial difficulty. Based on qualitative feedback, improvements were made to multimedia elements, including image resolution, subtitles, and audio quality.

Setting

This study was conducted in an online mock examination environment with nursing students in South Korea. Participants were recruited after IRB approval beginning on October 29, 2025, and the mock examination was administered from November 20 to 25, 2025. All examinees participated in the mock examination once using a personal computer during the designated testing period. A post-examination survey was conducted immediately after the mock examination, and focus group interviews were conducted online with 2 groups on November 28, 2025.

Participants

The sample size was determined based on IRT requirements, with at least 500 participants recommended for 2-parameter logistic (2PL) models [12]. Accounting for a 10% attrition rate, 556 fourth-year nursing students in South Korea were recruited. Eligible participants were those able to complete an online examination. Data from 515 participants were included in the final analysis after excluding 24 non-examinees and 17 cases with invalid responses, defined as a completion time of less than 5 minutes (n=5) or an incomplete examination (n=12). For response-process validity, focus group interviews were conducted with 15 participants, consistent with prior recommendations for achieving data saturation [13]. Among participants who agreed to participate in the focus group interviews, 15 were randomly selected for the qualitative component using a random number table generated in Microsoft Excel (Microsoft Corp.). The 15 participants were divided into 2 focus groups, with 7 participants in Group A and 8 in Group B.

Study procedure and data collection

The flow diagram of participant enrollment and data collection is shown in Fig. 1. Participants were randomly assigned to either the MT group (MAIs first, then text-based items) or the TM group (text-based items first, then MAIs). Each format consisted of 29 paired items with identical content, for a total of 58 items. Binary item response data from the mock examination are provided in Dataset 1. Item difficulty and discrimination were analyzed using classical test theory (CTT) and IRT. A post-examination survey assessing perceptions of MAIs, consisting of 6 items on a 5-point Likert scale, was administered after the examination. The survey items examined participants’ perceptions of MAIs in terms of (1) realistic reflection of clinical practice, (2) helpfulness of the presented materials for problem solving, (3) perceived accuracy in assessing examinee ability compared with text-based items, (4) difficulty maintaining concentration while solving MAIs, (5) sufficiency of the allotted time for interpreting and responding to items, and (6) satisfaction with the screen layout. These items were developed by the research team based on the study purpose and the key characteristics of MAIs, including clinical authenticity, use of multimedia materials, perceived assessment accuracy, cognitive burden, response time, and interface usability. The survey questionnaire is provided in Supplement 1, and participant characteristics and post-survey data are provided in Dataset 2. Focus group interviews were conducted with consenting participants to examine response-process validity using a semi-structured interview guide. The focus group interview guide is provided in Supplement 2.

Bias

To minimize testing-environment bias, all participants completed the mock examination using the same online platform under identical conditions. Item order was randomized within each test format to reduce potential memory and order effects. During qualitative data collection, the researcher maintained a neutral stance and ensured that participants’ experiences were adequately reflected.

Data analysis

Data analyses were conducted using IBM SPSS Statistics ver. 30.0 (IBM Corp.), Microsoft Excel, and R ver. 4.4.1 (The R Foundation for Statistical Computing). Descriptive statistics were used to analyze participants’ characteristics and perceptions of MAIs. Item difficulty and discrimination were estimated based on CTT and IRT, with the 2PL model applied for IRT analyses using the irtQ package in R. CTT-based discrimination was calculated using the 27% upper–lower group method and corrected item–total correlations. Pearson correlation analyses were conducted to examine the consistency of the 5 item parameters between paired MAIs and text-based items. MAIs and text-based items were also analyzed as 2 separate tests. Total scores were calculated for the 29 MAIs (M01–M29) and the 29 text-based items (T01–T29), respectively, and IRT ability parameters were estimated using the 2PL model. Pearson correlations were then computed between the total scores and between the ability parameters derived from the 2 formats. Total scores and ability parameters were compared between the MT and TM groups using independent-samples t-tests. Item parameters estimated separately for each group were compared using paired-samples t-tests and Pearson correlation analyses. Quantitative and qualitative findings were integrated during interpretation to relate psychometric item characteristics to participants’ experiences and response processes.

Qualitative data were analyzed using inductive content analysis. Audio-recorded interviews were transcribed verbatim and repeatedly read to obtain an overall understanding of the data. Two researchers independently coded the transcripts, compared their codes through discussion, and resolved discrepancies by consensus. Codes with similar meanings were grouped into subcategories, which were then organized into broader categories and themes. To enhance trustworthiness, researcher triangulation, repeated transcript review, and consensus meetings were used throughout the analytic process. The focus group interviews were facilitated by nursing faculty members on the research team, none of whom had a prior relationship with the participants in their assigned group. To minimize potential power imbalances, interviewers were deliberately assigned based on participants’ regional affiliations, ensuring that no interviewer conducted interviews at an institution where the participants were enrolled. Member checking was not performed because of logistical constraints and the exploratory nature of the qualitative component. Transcripts were not returned to participants for review; however, credibility was supported through clarification questions during the interviews and team-based review of the transcripts and themes.

Results

Participants

Among the 515 participants, 256 (49.71%) were assigned to the MT group and 259 (50.29%) to the TM group. Most participants were female (85.44%), and the mean age was 24.68±5.51 years. The largest proportion of participants was from the Busan–Ulsan–Daegu–Gyeongsang region (24.08%). Before the mock examination, 31.26% of participants had experience with examinations including MAIs. Images were the most commonly used media type (90.06%), followed by audio (50.93%) and video (24.84%) (Table 1).

Characteristics of the developed MAIs

Of the 29 developed MAIs, adult nursing accounted for the largest proportion (n=10; 34.48%), followed by pediatric nursing (n=9; 31.03%), maternal nursing (n=6; 20.69%), fundamental nursing (n=3; 10.34%), and psychiatric nursing (n=1; 3.45%). By knowledge level, problem-solving items were most common (n=15; 51.72%), followed by interpretation (n=12; 41.38%) and recall (n=2; 6.90%). By nursing process, assessment items were most frequent (n=13; 44.83%). Images were used to present visual patient-related information, such as pressure ulcer status, patient positioning, and diagnostic test results. Audio materials provided auditory clinical cues, including lung sounds and statements from patients or nurses. The videos depicted dynamic clinical situations, such as electrocardiogram rhythms, chest drainage bottle status, and patients’ physical symptoms or clinical signs. These materials were designed to provide clinical cues and contextual information that are difficult to convey through text alone. The characteristics of all items are summarized in Table 2, and an example of the examinee test screen is shown in Fig. 2.

Psychometric properties of the items

Psychometric analyses were conducted based on paired items with identical content, and all participants (N=515) were pooled to minimize the effects of test-format presentation order.

Classical test theory

In the CTT-based item analysis, the correct response rate for MAIs ranged from 20.58% to 98.06%, with a mean of 63.37%±21.57%. Discrimination index 1, calculated using the upper–lower group method, ranged from 0.04 to 0.63 (mean=0.29±0.14), and discrimination index 2, based on item–total correlation, ranged from 0.03 to 0.40 (mean=0.21±0.08). For text-based items, the correct response rate ranged from 26.41% to 99.22%, with a mean of 64.67%±19.84%. Discrimination index 1 ranged from 0.01 to 0.63 (mean=0.30±0.15), and discrimination index 2 ranged from 0.07 to 0.41 (mean=0.22±0.08). The CTT-based item analysis results for all items are presented in Table 3.

Item response theory

In the IRT-based item analysis, the difficulty parameters of MAIs ranged from –4.59 to 27.35 (mean=–0.18±5.63), and discrimination parameters ranged from 0.05 to 1.49 (mean=0.67±0.34). For text-based items, difficulty parameters ranged from –5.01 to 6.38 (mean=–1.12±2.24), and discrimination parameters ranged from 0.14 to 1.62 (mean=0.69±0.34). However, several items showed extreme difficulty estimates (|b|>4) with large standard errors (>2), indicating some instability in parameter estimation. This instability may be related to the sample size for items with extreme values. The IRT-based item analysis results for all items are presented in Table 4.

Pearson correlation analyses were conducted to evaluate the consistency of item parameters between MAIs and their corresponding text-based items (Supplement 3). Strong positive correlations were observed across all 5 item parameters. Correlation coefficients were 0.94 for CTT difficulty, 0.92 for CTT discrimination using the upper–lower discrimination index, 0.83 for corrected item–total correlation, 0.84 for the IRT difficulty parameter, and 0.90 for the IRT discrimination parameter; all were statistically significant (P<0.001). Scatter plots of item parameter correlations between paired multimedia-assisted and text-based items are presented in Fig. 3.

Correlation between multimedia-assisted and text-based item formats

MAIs and text-based items were analyzed as separate tests to examine the correlations between total scores and ability parameters derived from each format. Total scores were strongly correlated between the 2 formats (r=0.73, P<0.001), and ability parameters also showed a strong correlation (r=0.72, P<0.001). Scatter plots illustrating these correlations are presented in Fig. 4.

Group comparison between the MT and TM groups

Independent t-tests showed no statistically significant differences between the MT and TM groups in total scores or ability parameters (Supplement 4). The mean total score was 36.76±6.87 in the MT group and 37.49±7.10 in the TM group (t=–1.19, P=0.235). The mean ability parameter was –0.07±0.88 in the MT group and 0.07±0.92 in the TM group (t=–1.83, P=0.068). Item parameters were estimated separately for the MT and TM groups (Supplement 5). No statistically significant differences were observed in CTT difficulty, CTT discrimination, corrected item–total correlation, or IRT difficulty. However, the IRT discrimination parameter was significantly higher in the TM group than in the MT group (t=–2.39, P=0.020). Pearson correlations between the MT and TM groups ranged from –0.02 to 0.90 across item parameters. These correlations are illustrated in Fig. 5. R script for item analysis using the irtQ package and data analysis outputs are presented in Supplement 6.

Post-test survey results

Participants rated MAIs as realistically reflecting clinical practice (4.13±0.74) and reported that the presented materials were helpful for problem solving (4.01±0.84). They also perceived MAIs as more accurately assessing examinee ability than text-based items (3.85±0.95). Participants reported no major difficulty maintaining concentration while solving MAIs (3.74±1.10) and indicated that the allotted time for interpreting and responding to items was sufficient (4.34±0.68). Satisfaction with the screen layout was relatively high (4.09±0.94). A total of 67 participants (13.01%) reported that some items included content they had not previously learned (Table 5).

Qualitative findings from focus group interviews

Fifteen participants were divided into 2 groups, and 2 focus group interviews were conducted. All participants were female, with a mean age of 25.5 years. Four participants (26.7%) had prior experience with MAIs. Group A consisted of 7 participants aged 22–39 years from the Gyeongsang, Gyeonggi, Gangwon, and Jeolla regions, including 2 participants with prior MAI experience. Group B consisted of 8 participants aged 22–30 years from the Jeolla, Seoul, Chungcheong, Gyeongsang, and Gangwon regions, including 2 participants with prior MAI experience. Each group was homogeneous in that all participants were fourth-year female nursing students, whereas heterogeneity between and within groups was considered by including participants from different regions and with varying prior experience with MAIs. Qualitative analysis yielded 3 themes, 8 categories, and 32 subcategories (Table 6). The 3 themes were (1) learners’ initial experiences and cognitive responses to MAIs, (2) experiences of assessing practical competence reflecting real clinical settings, and (3) the educational value and appropriate use of multimedia-based assessment.

Learners’ initial experiences and cognitive responses to MAIs

Participants experienced unfamiliarity, confusion, and tension when they first encountered MAIs, while also perceiving them as a novel item format. They tended to find MAIs difficult because they differed from conventional text-based items and required them to interpret patient information resembling real clinical situations. Participants also reported difficulty applying theoretical knowledge to materials that reflected real clinical contexts. In addition, they felt burdened by multi-source information processing because they had to observe several elements on the screen simultaneously, replay materials to confirm missed information, and spend more time solving items. Low-realism materials, including some AI-generated videos, were also reported to disrupt concentration.

Experiences of assessing practical competence reflecting real clinical settings

Participants perceived that MAIs required different problem-solving strategies and higher-order thinking processes compared with text-based items. Rather than relying on characteristic textual cues or fragmentary knowledge, they had to integrate various types of clinical information, such as the patient’s condition, sounds, and vital signs, and determine priorities. Participants viewed MAIs as suitable for practice-based assessment because they clearly represented clinical situations and required assessment and judgment processes similar to those used in actual clinical practice. They also reported that simulation and clinical training experiences facilitated problem solving in MAIs.

The educational value and appropriate use of multimedia-based assessment

Participants recognized the educational value and learning effects of MAIs, including intuitive comprehension of complex concepts, faster problem solving, and increased immersion during assessment. They emphasized the need to expand multimedia-based learning and suggested developing standardized multimedia educational materials through schools or professional associations. However, they also noted that multimedia materials are not necessary for all items and that text-based items and MAIs should be used in a complementary manner. Some participants pointed out that the advantages of MAIs were not fully realized when items could be solved without engaging with the multimedia materials.

Discussion

Key results

This study administered a mock examination presenting identical item content in both text-based and multimedia formats to examine psychometric characteristics by item type and assess learners’ perceptions of MAIs, thereby evaluating the validity of MAIs. The study used a within-group counterbalanced approach to examine the characteristics of MAIs rather than determine their superiority over conventional items.

CTT-based item analyses showed that the overall mean correct response rate was comparable between MAIs and text-based items; however, the direction of the difference varied across individual items. For items in which MAIs showed lower correct response rates, this may reflect the additional difficulty of connecting learned theoretical knowledge with multimedia-presented information. Lee and Shin [10] reported that participants found it difficult to combine theoretical content with multimedia-presented information and were not accustomed to integrating practical experience with theoretical knowledge when solving MAIs. However, several items demonstrated higher correct response rates in the multimedia format. For example, items presenting fetal/newborn reflexes and fundal height using images or videos (Items 3, 8, and 16) showed good discrimination (all>0.30), with higher correct response rates for multimedia than for text-based versions. These findings suggest that the mode of information presentation, text versus multimedia, can influence item difficulty and that multimedia, particularly audiovisual materials that convey critical information, may reduce extraneous cognitive load and facilitate problem understanding. This aligns with prior research indicating that MAIs support situational understanding and problem solving [10]. Overall, discrimination indices were similarly distributed across multimedia and text-based items, with mean values of approximately 0.3 for both formats. Compared with the KNLE, the upper–lower index of MAIs in this study (0.29) was higher than that of text-based items in the 61st–65th KNLEs (0.18–0.21), whereas the item–total correlation of MAIs (0.21) was slightly lower than that of KNLE text-based items (0.23–0.24). Taken together, these findings suggest that MAIs provided acceptable levels of discrimination and could be used alongside text-based items to differentiate examinee ability.

In the IRT-based item analysis, the mean discrimination parameters were similar between MAIs and text-based items, indicating that both formats had a comparable capacity to differentiate examinees. MAIs, however, tended to be estimated as more difficult than text-based items, and several items showed extreme difficulty estimates (e.g., |b|>4) accompanied by large standard errors, indicating instability in parameter estimates for these items. This instability suggests that the sample size may have been insufficient for stable parameter estimation of items with extreme difficulty, highlighting the need for further validation using larger samples. Moreover, items with extremely low difficulty estimates (Items 9, 11, and 20) had correct response rates exceeding 92%, likely because the multimedia materials directly conveyed visual or auditory cues that facilitated correct responses; this may have limited the extent to which these items could discriminate across the ability range. In contrast, Item 27, which presented a nurse rounding scenario via video, showed a correct response rate of 20.58%, approximating chance level. This may reflect excessive extraneous cognitive load imposed by the simultaneous presentation of multiple clinical cues, compounded by the fact that this item represented a novel format that departed from conventional KNLE item structures.

Across CTT- and IRT-based indices, paired MAIs and text-based items showed strong positive correlations, suggesting that converting items from text to multimedia format did not substantially alter their relative measurement properties within this mock examination. Although absolute difficulty levels differed between formats for some items, the relative ordering of item difficulty and discrimination was largely preserved across the 2 formats. Items that were relatively easy or difficult in text format tended to remain so when presented as MAIs, and items that showed higher discrimination in text format generally retained their discriminative capacity in multimedia format. These findings indicate that introducing MAIs is unlikely to markedly compromise the measurement characteristics established for existing text-based items. When MAIs and text-based items were analyzed as separate tests, total scores and ability parameters were also strongly correlated between the 2 formats. Total scores derived from MAI and text-based item sets were highly correlated (r=0.73), and the corresponding ability parameters showed a similarly high correlation (r=0.72). This suggests that examinee performance was estimated similarly across the MAI and text-based item sets and supports the consistency of ability estimation across the 2 formats.

Regarding the order in which MAIs and text-based items were administered, there were no statistically significant differences between the MT and TM groups in total scores or ability estimates, indicating that format order did not meaningfully affect overall performance. However, the higher 2PL discrimination parameter in the TM group suggests that format order may still have had a small effect on the measurement precision of certain items. Specifically, completing text-based items first may have influenced how some items functioned when participants subsequently answered MAIs. When item parameters were estimated separately for each group, discrepancies in IRT difficulty estimates were mainly observed for items with very low discrimination and extreme b values. These discrepancies are more likely to reflect instability due to limited information for items at the extremes of the ability distribution than true differences in item difficulty between groups.

Post-examination survey results indicated that participants perceived MAIs as realistic reflections of clinical settings and as helpful for problem solving. Participants also reported that MAIs could assess examinee ability more accurately, suggesting that multimedia-based assessment may be an appropriate tool for evaluating clinical judgment and integrative thinking beyond rote knowledge. This finding is consistent with the focus group theme concerning experiences of assessing practical competence reflecting real clinical settings. These findings suggest that the value of MAIs extends beyond psychometric performance to include clinical authenticity, examinee perceptions, and response-process evidence for clinical judgment.

Qualitative analysis further showed that learners perceived MAIs as facilitating a clearer understanding of clinical situations and requiring cognitive processes such as integrating complex information and prioritizing problems. However, some participants noted that excessive multimedia content could increase cognitive load, aligning with cognitive load theory, which emphasizes that the amount and presentation of information influence learning effectiveness [14]. Prior studies have likewise reported that audiovisual materials not directly aligned with learning objectives may increase learners’ cognitive burden [15]. Therefore, rather than being used indiscriminately, MAIs should be applied selectively and used in a complementary manner with text-based items.

These findings suggest that the implementation of MAIs in the computer-based KNLE should follow a progressive approach rather than immediate large-scale adoption. The rationale for introducing MAIs is not that they are psychometrically superior to conventional text-based items but that they can provide comparable measurement performance while enhancing the authenticity of clinical scenarios and supporting the transition to CBT-based examinations. Therefore, MAIs should be used selectively when multimedia presentation adds value to the assessment of clinical judgment, rather than being applied to all item types. At the national examination level, detailed operational standards, item review procedures, and trained item developers are needed to apply standardized MAI development guidelines within the existing item development system. In addition, preparatory systems, such as practice items and tutorials, should be provided to reduce examinees’ cognitive burden and improve their familiarity with MAI formats. Rather than predetermining a fixed proportion of MAIs, it may be more appropriate to first expand data-based items and then progressively introduce a small number of MAIs for items in which multimedia presentation is essential. Future MAI development should also expand beyond static images and figures to include video- and audio-based items that reflect dynamic clinical situations, such as changes in a patient’s condition over time. Further evidence should also be accumulated regarding examinees’ acceptance, cognitive load, response time, item difficulty, discrimination, and test efficiency according to the proportion and types of MAIs.

Limitations

This study has several limitations. First, instability in parameter estimation was observed for some items with extreme difficulty in the IRT analysis, warranting caution in interpreting the results. Second, because the study was conducted in an online mock examination environment, control over testing time and location was limited, and the conditions may not fully reflect those of an actual examination.

Despite these limitations, the findings provide important implications for applying MAIs to national examinations. Standardized guidelines and review procedures for MAI development are needed. In addition, preparatory strategies to reduce examinees’ initial cognitive burden should be implemented, and a phased introduction with an appropriate balance between multimedia and text-based items is recommended to ensure stable test administration. Future research should conduct item-level analyses to further explore specific items for which text-based items or MAIs yielded statistically higher correct response rates, providing deeper insights into the relative strengths of each item type.

Conclusion

This study examined the validity of MAIs by analyzing psychometric characteristics and learner perceptions using a mock examination that presented identical item content in text-based and multimedia formats. The findings showed that multimedia items demonstrated overall difficulty and discrimination comparable to those of text-based items, while some items facilitated problem understanding and problem solving by realistically representing clinical situations. Learners perceived multimedia items as reflective of clinical practice and useful for assessing practical competence. These results suggest that MAIs can serve as effective assessment tools for evaluating clinical judgment and integrative thinking in nursing education and provide foundational evidence for improving the quality of assessment in future health professions education.

Supplementary materials

Fig. 1.

Flow diagram of participant enrollment and data collection

Fig. 2.

Example of multimedia-assisted item used in the mock examination

Fig. 3.

(A–E) Scatter plots of item parameter correlations between paired multimedia-assisted and text-based items

Fig. 4.

Scatter plots of total score correlation (A) and latent ability correlation (B) between multimedia-assisted and text-based item formats.

Fig. 5.

(A–G) Comparison of total scores, latent abilities, and item parameters between the 2 groups. MT group: multimedia-assisted items (MAIs) first, then text-based items; TM group: text-based items first, then MAIs. CTT, classical test theory; IRT, item response theory.

References

• 1. Benner PE, Sutphen M, Leonard V, Day L. Educating nurses: a call for radical transformation. Jossey-Bass; 2010.
• 2. Jung SJ, Park J. Development of a patient safety simulation program for new nurses in the intensive care unit. J Korean Acad Fundam Nurs 2024;31:100-111. https://doi.org/10.7739/jkafn.2024.31.1.100 Article
• 3. Cho NH, Lee H. The effects of virtual reality simulation-based blood transfusion education on newly graduated nurses’ performance confidence, learning presence, and learning satisfaction. J Korean Acad Soc Nurs Educ 2025;31:5-16. https://doi.org/10.5977/jkasne.2025.31.1.5 Article
• 4. Korea Health Personnel Licensing Examination Institute. 2025 Opening ceremony: CBT transition will proceed without disruption [Internet]. Korea Health Personnel Licensing Examination Institute; 2025 [cited 2025 Mar 31]. Available from: https://www.kuksiwon.or.kr/news/brd/m_54/view.do?seq=608
• 5. Sung TJ. Modern educational evaluation. 5th ed. Hakjisa; 2019.
• 6. Park EA, Lee MB, Lee SH, Choi HJ, Seo MH, Kwon YB, Kim SS. Development and field test of a computer-based assessment tool for problem solving competence. Korea Institute for Curriculum and Evaluation; 2018.
• 7. Wang T, Kao C, Dai Y. Developing a web‐based multimedia assessment system for facilitating science laboratory instruction. Comput Assist Learn 2019;35:529-539. https://doi.org/10.1111/jcal.12357 Article
• 8. National Council of State Boards of Nursing. Next Generation NCLEX [Internet]. National Council of State Boards of Nursing; 2023 [cited 2025 May 14]. Available from: https://www.ncsbn.org/exams/next-generation-nclex.page
• 9. Korea Health Personnel Licensing Examination Institute. Computer-based testing [Internet]. Korea Health Personnel Licensing Examination Institute; 2025 [cited 2025 Mar 31]. Available from: https://www.kuksiwon.or.kr/cnt/c_2033/view.do?seq=10
• 10. Lee I, Shin S. Development and validation of practice-based multimedia assisted item: a mixed-method study. BMC Nurs 2025;24:241. https://doi.org/10.1186/s12912-025-02790-x ArticlePubMedPMC
• 11. Jang S, Suh EE. Development and application of a mobile-based multimedia nursing competency evaluation system for nursing students: a mixed-method randomized controlled study. Nurse Educ Pract 2022;64:103458. https://doi.org/10.1016/j.nepr.2022.103458 ArticlePubMed
• 12. Baker FB, Kim SH. The basics of item response theory using R. Springer; 2017.
• 13. Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Public Opin Q 2007;71:287-311. https://doi.org/10.1093/poq/nfm006 Article
• 14. Mayer RE. Multimedia learning. 3rd ed. Cambridge University Press; 2020.
• 15. Sweller J. Cognitive load theory and educational technology. Educ Technol Res Dev 2020;68:1-16. https://doi.org/10.1007/s11423-019-09701-3 Article

Figure & Data

References

Citations

Citations to this article as recorded by