Journal of Educational Evaluation for Health Professions

Research articles

The effect of strengthening nurse practitioners’ competency in occupational health services for agricultural workers exposed to pesticides in primary care units, Thailand: a before-and-after study: Napamon Pumsopa, Ann Jirapongsuwan, Surintorn Kalampakorn, Sukhontha Siri; J Educ Eval Health Prof. 2025;22:14. Published online April 21, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.14 [Epub ahead of print]

395 View
123 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to evaluate the effect of the Strengthening Nurse Practitioners’ Competency in Occupational Health Service (SNPCOHS) program. It was hypothesized that nurse practitioners (NPs) participating in the program would demonstrate increased competency in providing occupational health services to agricultural workers exposed to pesticides in primary care units (PCUs) compared to their baseline competency and to a comparison group.
Methods
A quasi-experimental study was conducted between August and December 2023. The 4-week intervention included 5 hours of an e-learning program, 3 hours of online discussion, and 2 hours dedicated to completing an assignment. The program was evaluated at 3 time points: pre-intervention, post-intervention (week 4), and follow-up (week 8). Sixty NPs volunteered to participate, with 30 in the experimental group and 30 in the comparison group. Data on demographics, professional attributes, knowledge, skills, and perceived self-efficacy were collected using self-administered questionnaires via Google Forms. Data analysis involved descriptive statistics, independent t-tests, and repeated measures analysis of variance.
Results
The experimental group demonstrated significantly higher mean scores in professional attributes, knowledge, skills, and perceived self-efficacy in providing occupational health services to agricultural workers exposed to pesticides compared to the comparison group at both week 4 and week 8 post-intervention.
Conclusion
The SNPCOHS program is well-suited for self-directed learning for nurses in PCUs, supporting effective occupational health service delivery. It should be disseminated and supported as an e-learning resource for NPs in PCUs (Thai Clinical Trials Registry: TCTR20250115004).

Assessing genetic and genomic literacy concepts among Albanian nursing and midwifery students: a cross-sectional study: Elona Gaxhja, Mitilda Gugu, Angelo Dante, Armelda Teta, Armela Kapaj, Liljana Ramasaco; J Educ Eval Health Prof. 2025;22:13. Published online April 21, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.13 [Epub ahead of print]

386 View
84 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to adapt and validate the Albanian version of the Genomic Nursing Concept Inventory (GNCI) and to assess the level of genomic literacy among nursing and midwifery students.
Methods
Data were collected via a monocentric online cross-sectional study using the Albanian version of the GNCI. Participants included first-, second-, and third-year nursing and midwifery students. Demographic data such as age, sex, year level, and prior exposure to genetics were collected. The Kruskal-Wallis, Mann-Whitney U, and chi-square tests were used to compare demographic characteristics and GNCI scores between groups.
Results
Among the 715 participants, most were female (88.5%) with a median age of 19 years. Most respondents (65%) had not taken a genetics course, and 83.5% had not attended any related training. The mean score was 7.49, corresponding to a scale difficulty of 24.38% correct responses.
Conclusion
The findings reveal a low foundational knowledge of genetics/genomics among future nurses and midwives. It is essential to enhance learning strategies and update curricula to prepare a competent healthcare workforce in precision health.

Evaluation of a virtual objective structured clinical examination in the metaverse (Second Life) to assess the clinical skills in emergency radiology of medical students in Spain: a cross-sectional study: Alba Virtudes. Perez-Baena, Teodoro Rudolphi-Solero, Rocio Lorenzo-Alvarez, Dolores Dominguez-Pinos, Miguel Jose Ruiz-Gomez, Francisco Sendra-Portero; J Educ Eval Health Prof. 2025;22:12. Published online April 21, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.12 [Epub ahead of print]

372 View
76 Download

Abstract PDF: Purpose
The objective structured clinical examination (OSCE) is an effective but resource-intensive tool for assessing clinical competence. This study hypothesized that implementing a virtual OSCE in the Second Life (SL) platform in the metaverse as a cost-effective alternative will effectively assess and enhance clinical skills in emergency radiology while being feasible and well-received. The aim was to evaluate a virtual radiology OSCE in SL as a formative assessment, focusing on feasibility, educational impact, and students’ perceptions.
Methods
Two virtual 6-station OSCE rooms dedicated to emergency radiology were developed in SL. Sixth-year medical students completed the OSCE during a 1-hour session in 2022–2023, followed by feedback including a correction checklist, individual scores, and group comparisons. Students completed a questionnaire with Likert-scale questions, a 10-point rating, and open-ended comments. Quantitative data were analyzed using the Student t-test and the Mann-Whitney U test, and qualitative data through thematic analysis.
Results
In total, 163 students participated, achieving mean scores of 5.1±1.4 and 4.9±1.3 (out of 10) in the 2 virtual OSCE rooms, respectively (P=0.287). One hundred seventeen students evaluated the OSCE, praising the teaching staff (9.3±1.0), project organization (8.8±1.2), OSCE environment (8.7±1.5), training usefulness (8.6±1.5), and formative self-assessment (8.5±1.4). Likert-scale questions and students’ open-ended comments highlighted the virtual environment’s attractiveness, case selection, self-evaluation usefulness, project excellence, and training impact. Technical difficulties were reported by 13 students (8%).
Conclusion
This study demonstrated the feasibility of incorporating formative OSCEs in SL as a useful teaching tool for undergraduate radiology education, which was cost-effective and highly valued by students.

A nationwide survey on the curriculum and educational resources related to the Clinical Skills Test of the Korean Medical Licensing Examination: a cross-sectional descriptive study: Eun-Kyung Chung, Seok Hoon Kang, Do-Hoon Kim, MinJeong Kim, Ji-Hyun Seo, Keunmi Lee, Eui-Ryoung Han; J Educ Eval Health Prof. 2025;22:11. Published online March 13, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.11

884 View
159 Download

Abstract PDF Supplementary Material: Purpose
The revised Clinical Skills Test (CST) of the Korean Medical Licensing Exam aims to provide a better assessment of physicians’ clinical competence and ability to interact with patients. This study examined the impact of the revised CST on medical education curricula and resources nationwide, while also identifying areas for improvement within the revised CST.
Methods
This study surveyed faculty responsible for clinical clerkships at 40 medical schools throughout Korea to evaluate the status and changes in clinical skills education, assessment, and resources related to the CST. The researchers distributed the survey via email through regional consortia between December 7, 2023 and January 19, 2024.
Results
Nearly all schools implemented preliminary student–patient encounters during core clinical rotations. Schools primarily conducted clinical skills assessments in the third and fourth years, with a simplified form introduced in the first and second years. Remedial education was conducted through various methods, including one-on-one feedback from faculty after the assessment. All schools established clinical skills centers and made ongoing improvements. Faculty members did not perceive the CST revisions as significantly altering clinical clerkship or skills assessments. They suggested several improvements, including assessing patient records to improve accuracy and increasing the objectivity of standardized patient assessments to ensure fairness.
Conclusion
During the CST, students’ involvement in patient encounters and clinical skills education increased, improving the assessment and feedback processes for clinical skills within the curriculum. To enhance students’ clinical competencies and readiness, strengthening the validity and reliability of the CST is essential.

Correlation between a motion analysis method and Global Operative Assessment of Laparoscopic Skills for assessing interns’ performance in a simulated peg transfer task in Jordan: a validation study: Esraa Saleh Abdelall, Shadi Mohammad Hamouri, Abdallah Fawaz Al Dwairi, Omar Mefleh Al- Araidah; J Educ Eval Health Prof. 2025;22:10. Published online March 6, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.10

886 View
168 Download

Abstract PDF Supplementary Material: Purpose
This study aims to validate the use of ProAnalyst (Xcitex Inc.), a software for professional motion analysts to assess the performance of surgical interns while performing the peg transfer task in a simulator box for safe practice in real minimally invasive surgery.
Methods
A correlation study was conducted in a multidisciplinary skills simulation lab at the Faculty of Medicine, Jordan University of Science and Technology from October 2019 to February 2020. Forty-one interns (i.e., novices and intermediates) were recruited and an expert surgeon participated as a reference benchmark. Videos of participants’ performance were analyzed through the ProAnalyst and Global Operative Assessment of Laparoscopic Skills (GOALS). Two results were s analyzed for correlation.
Results
The motion analysis scores by Proanalyst were correlated with those by GOALS for novices (r=–0.62925, P=0.009), and Intermediates (r= –0.53422, P=0.033). Both assessment methods differentiated the participants’ performance based on their experience level.
Conclusion
The motion analysis scoring method with Proanalyst provides an objective, time-efficient, and reproducible assessment of interns’ performance, and comparable to GOALS. It may require initial training and set-up; however, it eliminates the need for expert surgeon judgment.

Correspondence

Accuracy of ChatGPT in answering cardiology board-style questions: Albert Andrew; J Educ Eval Health Prof. 2025;22:9. Published online February 27, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.9

1,147 View
129 Download

PDF Supplementary Material

Research articles

Empirical effect of the Dr LEE Jong-wook Fellowship Program to empower sustainable change for the health workforce in Tanzania: a mixed-methods study: Masoud Dauda, Swabaha Aidarus Yusuph, Harouni Yasini, Issa Mmbaga, Perpetua Mwambinngu, Hansol Park, Gyeongbae Seo, Kyoung Kyun Oh; J Educ Eval Health Prof. 2025;22:6. Published online January 20, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.6

1,337 View
207 Download

Abstract PDF Supplementary Material: Purpose
This study evaluated the Dr LEE Jong-wook Fellowship Program’s impact on Tanzania’s health workforce, focusing on relevance, effectiveness, efficiency, impact, and sustainability in addressing healthcare gaps.
Methods
A mixed-methods research design was employed. Data were collected from 97 out of 140 alumni through an online survey, 35 in-depth interviews, and one focus group discussion. The study was conducted from November to December 2023 and included alumni from 2009 to 2022. Measurement instruments included structured questionnaires for quantitative data and semi-structured guides for qualitative data. Quantitative analysis involved descriptive and inferential statistics (Spearman’s rank correlation, non-parametric tests) using Python ver. 3.11.0 and Stata ver. 14.0. Thematic analysis was employed to analyze qualitative data using NVivo ver. 12.0.
Results
Findings indicated high relevance (mean=91.6, standard deviation [SD]=8.6), effectiveness (mean=86.1, SD=11.2), efficiency (mean=82.7, SD=10.2), and impact (mean=87.7, SD=9.9), with improved skills, confidence, and institutional service quality. However, sustainability had a lower score (mean=58.0, SD=11.1), reflecting challenges in follow-up support and resource allocation. Effectiveness strongly correlated with impact (ρ=0.746, P<0.001). The qualitative findings revealed that participants valued tailored training but highlighted barriers, such as language challenges and insufficient practical components. Alumni-led initiatives contributed to knowledge sharing, but limited resources constrained sustainability.
Conclusion
The Fellowship Program enhanced Tanzania’s health workforce capacity, but it requires localized curricula and strengthened alumni networks for sustainability. These findings provide actionable insights for improving similar programs globally, confirming the hypothesis that tailored training positively influences workforce and institutional outcomes.

Reliability and construct validation of the Blended Learning Usability Evaluation–Questionnaire with interprofessional clinicians in Canada: a methodological study: Anish Kumar Arora, Jeff Myers, Tavis Apramian, Kulamakan Kulasegaram, Daryl Bainbridge, Hsien Seow; J Educ Eval Health Prof. 2025;22:5. Published online January 16, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.5

934 View
179 Download

Abstract PDF Supplementary Material: Purpose
To generate Cronbach’s alpha and further mixed methods construct validity evidence for the Blended Learning Usability Evaluation–Questionnaire (BLUE-Q).
Methods
Forty interprofessional clinicians completed the BLUE-Q after finishing a 3-month long blended learning professional development program in Ontario, Canada. Reliability was assessed with Cronbach’s α for each of the 3 sections of the BLUE-Q and for all quantitative items together. Construct validity was evaluated through the Grand-Guillaume-Perrenoud et al. framework, which consists of 3 elements: congruence, convergence, and credibility. To compare quantitative and qualitative results, descriptive statistics, including means and standard deviations for each Likert scale item of the BLUE-Q were calculated.
Results
Cronbach’s α was 0.95 for the pedagogical usability section, 0.85 for the synchronous modality section, 0.93 for the asynchronous modality section, and 0.96 for all quantitative items together. Mean ratings (with standard deviations) were 4.77 (0.506) for pedagogy, 4.64 (0.654) for synchronous learning, and 4.75 (0.536) for asynchronous learning. Of the 239 qualitative comments received, 178 were identified as substantive, of which 88% were considered congruent and 79% were considered convergent with the high means. Among all congruent responses, 69% were considered confirming statements and 31% were considered clarifying statements, suggesting appropriate credibility. Analysis of the clarifying statements assisted in identifying 5 categories of suggestions for program improvement.
Conclusion
The BLUE-Q demonstrates high reliability and appropriate construct validity in the context of a blended learning program with interprofessional clinicians, making it a valuable tool for comprehensive program evaluation, quality improvement, and evaluative research in health professions education.

Educational/Faculty development material

The role of large language models in the peer-review process: opportunities and challenges for medical journal reviewers and editors: Jisoo Lee, Jieun Lee, Jeong-Ju Yoo; J Educ Eval Health Prof. 2025;22:4. Published online January 16, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.4

2,412 View
214 Download

Abstract PDF Supplementary Material: The peer review process ensures the integrity of scientific research. This is particularly important in the medical field, where research findings directly impact patient care. However, the rapid growth of publications has strained reviewers, causing delays and potential declines in quality. Generative artificial intelligence, especially large language models (LLMs) such as ChatGPT, may assist researchers with efficient, high-quality reviews. This review explores the integration of LLMs into peer review, highlighting their strengths in linguistic tasks and challenges in assessing scientific validity, particularly in clinical medicine. Key points for integration include initial screening, reviewer matching, feedback support, and language review. However, implementing LLMs for these purposes will necessitate addressing biases, privacy concerns, and data confidentiality. We recommend using LLMs as complementary tools under clear guidelines to support, not replace, human expertise in maintaining rigorous peer review standards.

Research articles

Empathy and tolerance of ambiguity in medical students and doctors participating in art-based observational training at the Rijksmuseum in Amsterdam, the Netherlands: a before-and-after study: Stella Anna Bult, Thomas van Gulik; J Educ Eval Health Prof. 2025;22:3. Published online January 14, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.3

1,275 View
162 Download

Abstract PDF Supplementary Material: Purpose
This research presents an experimental study using validated questionnaires to quantitatively assess the outcomes of art-based observational training in medical students, residents, and specialists. The study tested the hypothesis that art-based observational training would lead to measurable effects on judgement skills (tolerance of ambiguity) and empathy in medical students and doctors.
Methods
An experimental cohort study with pre- and post-intervention assessments was conducted using validated questionnaires and qualitative evaluation forms to examine the outcomes of art-based observational training in medical students and doctors. Between December 2023 and June 2024, 15 art courses were conducted in the Rijksmuseum in Amsterdam. Participants were assessed on empathy using the Jefferson Scale of Empathy (JSE) and tolerance of ambiguity using the Tolerance of Ambiguity in Medical Students and Doctors (TAMSAD) scale.
Results
In total, 91 participants were included; 29 participants completed the JSE and 62 completed the TAMSAD scales. The results showed statistically significant post-test increases for mean JSE and TAMSAD scores (3.71 points for the JSE, ranging from 20 to 140, and 1.86 points for the TAMSAD, ranging from 0 to 100). The qualitative findings were predominantly positive.
Conclusion
The results suggest that incorporating art-based observational training in medical education improves empathy and tolerance of ambiguity. This study highlights the importance of art-based observational training in medical education in the professional development of medical students and doctors.

Pharmacy students’ perspective on remote flipped classrooms in Malaysia: a qualitative study: Wei Jin Wong, Shaun Wen Huey Lee, Ronald Fook Seng Lee; J Educ Eval Health Prof. 2025;22:2. Published online January 14, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.2

787 View
173 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to explore pharmacy students’ perceptions of remote flipped classrooms in Malaysia, focusing on their learning experiences and identifying areas for potential improvement to inform future educational strategies.
Methods
A qualitative approach was employed, utilizing inductive thematic analysis. Twenty Bachelor of Pharmacy students (18 women, 2 men; age range, 19–24 years) from Monash University participated in 8 focus group discussions over 2 rounds during the coronavirus disease 2019 pandemic. Participants were recruited via convenience sampling. The focus group discussions, led by experienced academics, were conducted in English via Zoom, recorded, and transcribed for analysis using NVivo. Themes were identified through emergent coding and iterative discussions to ensure thematic saturation.
Results
Five major themes emerged: flexibility, communication, technological challenges, skill-based learning challenges, and time-based effects. Students appreciated the flexibility of accessing and reviewing pre-class materials at their convenience. Increased engagement through anonymous question submission was noted, yet communication difficulties and lack of non-verbal cues in remote workshops were significant drawbacks. Technological issues, such as internet connectivity problems, hindered learning, especially during assessments. Skill-based learning faced challenges in remote settings, including lab activities and clinical examinations. Additionally, prolonged remote learning led to feelings of isolation, fatigue, and a desire to return to in-person interactions.
Conclusion
Remote flipped classrooms offer flexibility and engagement benefits but present notable challenges related to communication, technology, and skill-based learning. To improve remote education, institutions should integrate robust technological support, enhance communication strategies, and incorporate virtual simulations for practical skills. Balancing asynchronous and synchronous methods while addressing academic success and socioemotional wellness is essential for effective remote learning environments.

Correspondence

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2023, Korea: a psychometric study: Yoon Hee Kim, Bo Hyun Kim, Joonki Kim, Bokyoung Jung, Sangyoung Bae; J Educ Eval Health Prof. 2024;21:40. Published online December 11, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.40

812 View
100 Download

PDF Supplementary Material

Research article

Inter-rater reliability and content validity of the measurement tool for portfolio assessments used in the Introduction to Clinical Medicine course at Ewha Womans University College of Medicine: a methodological study: Dong-Mi Yoo, Jae Jin Han; J Educ Eval Health Prof. 2024;21:39. Published online December 10, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.39

1,108 View
172 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to examine the reliability and validity of a measurement tool for portfolio assessments in medical education. Specifically, it investigated scoring consistency among raters and assessment criteria appropriateness according to an expert panel.
Methods
A cross-sectional observational study was conducted from September to December 2018 for the Introduction to Clinical Medicine course at the Ewha Womans University College of Medicine. Data were collected for 5 randomly selected portfolios scored by a gold-standard rater and 6 trained raters. An expert panel assessed the validity of 12 assessment items using the content validity index (CVI). Statistical analysis included Pearson correlation coefficients for rater alignment, the intraclass correlation coefficient (ICC) for inter-rater reliability, and the CVI for item-level validity.
Results
Rater 1 had the highest Pearson correlation (0.8916) with the gold-standard rater, while Rater 5 had the lowest (0.4203). The ICC for all raters was 0.3821, improving to 0.4415 after excluding Raters 1 and 5, indicating a 15.6% reliability increase. All assessment items met the CVI threshold of ≥0.75, with some achieving a perfect score (CVI=1.0). However, items like “sources” and “level and degree of performance” showed lower validity (CVI=0.72).
Conclusion
The present measurement tool for portfolio assessments demonstrated moderate reliability and strong validity, supporting its use as a credible tool. For a more reliable portfolio assessment, more faculty training is needed.

Case Study

Feasibility of utilizing functional near-infrared spectroscopy to measure the cognitive load of paramedicine students undertaking high-acuity clinical simulations in Australia: a case study: Jason Betson, Erich Christian Fein, David Long, Peter Horrocks; J Educ Eval Health Prof. 2024;21:38. Published online December 10, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.38

980 View
130 Download

Abstract PDF Supplementary Material: Purpose
Paramedicine education often uses high-fidelity simulations that mimic real-life emergencies. These experiences can trigger stress responses characterized by physiological changes, including alterations in cerebral blood flow and oxygenation. Functional near-infrared spectroscopy (fNIRS) is emerging as a promising tool for assessing cognitive stress in educational settings.
Methods
Eight final-year undergraduate paramedicine students completed 2 high-acuity scenarios 7 days apart. Real-time continuous recording of cerebral blood flow and oxygenation levels in the prefrontal cortex was undertaken via fNIRS as a means of assessing neural activity during stressful scenarios.
Results
fNIRS accurately determined periods of increased cerebral oxygenation when participants were undertaking highly technical skills or making significant clinical decisions.
Conclusion
fNIRS holds potential for objectively measuring the cognitive load in undergraduate paramedicine students. By providing real-time insights into neurophysiological responses, fNIRS may enhance training outcomes in paramedicine programs and improve student well-being (Australian New Zealand Clinical Trials Registry: ACTRN12623001214628).

Research article

Validation of the 21st Century Skills Assessment Scale for public health students in Thailand: a methodological study: Suphawadee Panthumas, Kaung Zaw, Wirin Kittipichai; J Educ Eval Health Prof. 2024;21:37. Published online December 10, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.37

1,796 View
203 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to develop and validate the 21st Century Skills Assessment Scale (21CSAS) for Thai public health (PH) undergraduate students using the Partnership for 21st Century Skills framework.
Methods
A cross-sectional survey was conducted among 727 first- to fourth-year PH undergraduate students from 4 autonomous universities in Thailand. Data were collected using self-administered questionnaires between January and March 2023. Exploratory factor analysis (EFA) was used to explore the underlying dimensions of 21CSAS, while confirmatory factor analysis (CFA) was conducted to test the hypothesized factor structure using Mplus software (Muthén & Muthén). Reliability and item discrimination were assessed using Cronbach’s α and the corrected item-total correlation, respectively.
Results
EFA performed on a dataset of 300 students revealed a 20-item scale with a 6-factor structure: (1) creativity and innovation; (2) critical thinking and problem-solving; (3) information, media, and technology; (4) communication and collaboration; (5) initiative and self-direction; and (6) social and cross-cultural skills. The rotated eigenvalues ranged from 2.12 to 1.73. CFA performed on another dataset of 427 students confirmed a good model fit (χ2/degrees of freedom=2.67, comparative fit index=0.93, Tucker-Lewis index=0.91, root mean square error of approximation=0.06, standardized root mean square residual=0.06), explaining 34%–71% of variance in the items. Item loadings ranged from 0.58 to 0.84. The 21CSAS had a Cronbach’s α of 0.92.
Conclusion
The 21CSAS proved be a valid and reliable tool for assessing 21st century skills among Thai PH undergraduate students. These findings provide insights for educational system to inform policy, practice, and research regarding 21st-century skills among undergraduate students.

First
Prev
Page of 7
Next
Last

Video clip