Journal of Educational Evaluation for Health Professions

Research articles

Refusal-aware evaluation of frontier AI models available in June 2026 using the Japanese National License Examination for Pharmacists: a comparative study: Hiroyasu Sato, Katsuhiko Ogasawara, Hidehiko Sakurai; J Educ Eval Health Prof. 2026;23:29. Published online September 3, 2026; DOI: https://doi.org/10.3352/jeehp.2026.23.29 [Epub ahead of print]

720 View
74 Download

Abstract PDF: Purpose
Conventional single-run accuracy may be insufficient for evaluating frontier generative artificial intelligence (AI) models when safety-related refusals occur. This 4-model benchmark examined the need for repeated, refusal-aware evaluation using the Japanese National License Examination for Pharmacists (JNLEP).
Methods
ChatGPT GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and Claude Fable 5 were evaluated using all 345 questions from the 107th JNLEP. The original Japanese questions, including image-containing items, were submitted through application programming interfaces (APIs) in 3 independent runs. Refusals were treated as incorrect when overall accuracy was calculated. For Fable 5, accuracy excluding refusals, refusal rate, refusal consistency across runs, the subject-wise distribution of refusals, and system-assigned refusal categories were also evaluated.
Results
The mean overall accuracies were 98.7% for GPT-5.5, 98.3% for Gemini 3.5 Flash, 96.1% for Claude Opus 4.8, and 70.3% for Claude Fable 5. Fable 5 had a mean refusal rate of 29.0%, whereas its mean accuracy excluding refusals was 99.0%. Among the 345 items, 93 were refused in all 3 runs, 14 were refused inconsistently across runs, and 238 were never refused. All 300 refusal responses were assigned to the bio category. Refusals were most frequent in Biology (90.0%) and Pharmacology (69.2%) but uncommon in Practice (2.5%).
Conclusion
Near-saturation benchmark performance coexisted with frequent and partly run-dependent refusals in a safeguard-equipped model. Overall accuracy, accuracy excluding refusals, refusal rate, and refusal consistency describe complementary aspects of performance; therefore, repeated, refusal-aware evaluation is needed to interpret frontier AI models in pharmacy education.

Events related to medication errors and related factors involving nurses’ behavior to reduce medication errors in Japan: a Bayesian network modeling-based factor analysis and scenario analysis: Naotaka Sugimura, Katsuhiko Ogasawara; J Educ Eval Health Prof. 2024;21:12. Published online June 11, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.12

6,596 View
351 Download
3 Web of Science
5 Crossref

Purpose
This study aimed to identify the relationships between medication errors and the factors affecting nurses’ knowledge and behavior in Japan using Bayesian network modeling. It also aimed to identify important factors through scenario analysis with consideration of nursing students’ and nurses’ education regarding patient safety and medications.
Methods
We used mixed methods. First, error events related to medications and related factors were qualitatively extracted from 119 actual incident reports in 2022 from the database of the Japan Council for Quality Health Care. These events and factors were then quantitatively evaluated in a flow model using Bayesian network, and a scenario analysis was conducted to estimate the posterior probabilities of events when the prior probabilities of some factors were 0%.
Results
There were 10 types of events related to medication errors. A 5-layer flow model was created using Bayesian network analysis. The scenario analysis revealed that “failure to confirm the 5 rights,” “unfamiliarity with operations of medications,” “insufficient knowledge of medications,” and “assumptions and forgetfulness” were factors that were significantly associated with the occurrence of medical errors.
Conclusion
This study provided an estimate of the effects of mitigating nurses’ behavioral factors that trigger medication errors. The flow model itself can also be used as an educational tool to reflect on behavior when incidents occur. It is expected that patient safety education will be recognized as a major element of nursing education worldwide and that an integrated curriculum will be developed.

Citations

Citations to this article as recorded by

Investigating stress dynamics triggers and sleep disturbances among Indian medical students
Anuj Singh, Anjani Kumar Srivastava, Neha Singh Chauhan, Anjali Singh, Mritunjay Singh Thakur, Vineeta Gupta
Discover Public Health.2026;[Epub] CrossRef
Explaining the educational needs of nursing students in preventing medication errors: a qualitative study
Nayereh Baghcheghi, Hamid Reza Koohestani, Linda Graudins
International Journal For Quality In Health Care.2026;[Epub] CrossRef
Identifying contributing-factor configurations in reported bed falls: a Bayesian network–based exploratory study
Mingxuan Wang, Yanhui Zhang, Jiuqun Li, Peitao Li, Tianyi Huo, Yuting Jiang, Quanying Qu, Baohua Li
Frontiers in Public Health.2026;[Epub] CrossRef
Medication-Related Errors Among Nurses by Unit Adaptation Levels: Bayesian Network–Based Exploratory Study
Naotaka Sugimura, Katsuhiko Ogasawara
JMIR Nursing.2026; 9: e87436. CrossRef
Öğrenci Hemşirelerin İlaç Güvenliği Yeterliliklerinin Belirlenmesi
Ayşe Aydınlı
Uluborlu Mesleki Bilimler Dergisi.2025; 8(2): 16. CrossRef

ChatGPT (GPT-4) passed the Japanese National License Examination for Pharmacists in 2022, answering all items including those with diagrams: a descriptive study: Hiroyasu Sato, Katsuhiko Ogasawara; J Educ Eval Health Prof. 2024;21:4. Published online February 28, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.4

12,471 View
362 Download
13 Web of Science
15 Crossref

Abstract PDF Supplementary Material

Purpose
The objective of this study was to assess the performance of ChatGPT (GPT-4) on all items, including those with diagrams, in the Japanese National License Examination for Pharmacists (JNLEP) and compare it with the previous GPT-3.5 model’s performance.
Methods
The 107th JNLEP, conducted in 2022, with 344 items input into the GPT-4 model, was targeted for this study. Separately, 284 items, excluding those with diagrams, were entered into the GPT-3.5 model. The answers were categorized and analyzed to determine accuracy rates based on categories, subjects, and presence or absence of diagrams. The accuracy rates were compared to the main passing criteria (overall accuracy rate ≥62.9%).
Results
The overall accuracy rate for all items in the 107th JNLEP in GPT-4 was 72.5%, successfully meeting all the passing criteria. For the set of items without diagrams, the accuracy rate was 80.0%, which was significantly higher than that of the GPT-3.5 model (43.5%). The GPT-4 model demonstrated an accuracy rate of 36.1% for items that included diagrams.
Conclusion
Advancements that allow GPT-4 to process images have made it possible for LLMs to answer all items in medical-related license examinations. This study’s findings confirm that ChatGPT (GPT-4) possesses sufficient knowledge to meet the passing criteria.

Citations

Citations to this article as recorded by

Applications and potential of ChatGPT in dentistry: Scoping review of research perspectives
Masakazu Hamada, Sumire Kikuchi, Tatsuya Akitomo, Satoru Kusaka, Yuko Iwamoto, Ryota Nomura
Journal of Dental Sciences.2026; 21(1): 1. CrossRef
Performance of ChatGPT‐3.5 and ChatGPT‐4o in the Japanese National Dental Examination
Osamu Uehara, Tetsuro Morikawa, Fumiya Harada, Nodoka Sugiyama, Yuko Matsuki, Daichi Hiraki, Hinako Sakurai, Takashi Kado, Koki Yoshida, Yukie Murata, Hirofumi Matsuoka, Toshiyuki Nagasawa, Yasushi Furuichi, Yoshihiro Abiko, Hiroko Miura
Journal of Dental Education.2025; 89(4): 459. CrossRef
Qwen-2.5 Outperforms Other Large Language Models in the Chinese National Nursing Licensing Examination: Retrospective Cross-Sectional Comparative Study
Shiben Zhu, Wanqin Hu, Zhi Yang, Jiani Yan, Fang Zhang
JMIR Medical Informatics.2025; 13: e63731. CrossRef
ChatGPT (GPT-4V) Performance on the Healthcare Information Technologist Examination in Japan
Kai Ishida, Eisuke Hanada
Cureus.2025;[Epub] CrossRef
Medication counseling for OTC drugs using customized ChatGPT-4: Comparison with ChatGPT-3.5 and ChatGPT-4o
Keisuke Kiyomiya, Tohru Aomori, Hisakazu Ohtani
DIGITAL HEALTH.2025;[Epub] CrossRef
Current Use of Generative Artificial Intelligence in Pharmacy Practice: A Literature Mini-review
Keisuke Kiyomiya, Tohru Aomori, Hitoshi Kawazoe, Hisakazu Ohtani
Iryo Yakugaku (Japanese Journal of Pharmaceutical Health Care and Sciences).2025; 51(4): 177. CrossRef
Performance evaluation of large language models for the national nursing examination in Japan
Tomoki Kuribara, Kengo Hirayama, Kenji Hirata
DIGITAL HEALTH.2025;[Epub] CrossRef
Harnessing ChatGPT for digital tools in pharmacy practice
Reginald Amin Yakob, Adeola Bamgboje-Ayodele, Jack C. Collins, Parisa Aslani
Research in Social and Administrative Pharmacy.2025; 21(11): 943. CrossRef
Performance Evaluation of 18 Generative AI Models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese Pharmacist Licensing Examination: Comparative Study
Hiroyasu Sato, Katsuhiko Ogasawara, Hidehiko Sakurai
JMIR Medical Education.2025; 11: e76925. CrossRef
Evaluation of the Accuracy and Reliability of Responses Generated by Artificial Intelligence Related to Clinical Pharmacology
Michal Ordak, Julia Adamczyk, Agata Oskroba, Michal Majewski, Tadeusz Nasierowski
Journal of Clinical Medicine.2025; 14(21): 7563. CrossRef
Performance of ChatGPT-4 on the French Board of Plastic Reconstructive and Aesthetic Surgery written exam: a descriptive study
Emma Dejean-Bouyer, Anoujat Kanlagna, François Thuau, Pierre Perrot, Ugo Lancien
Journal of Educational Evaluation for Health Professions.2025; 22: 27. CrossRef
Potential of ChatGPT to Pass the Japanese Medical and Healthcare Professional National Licenses: A Literature Review
Kai Ishida, Eisuke Hanada
Cureus.2024;[Epub] CrossRef
Performance of Generative Pre-trained Transformer (GPT)-4 and Gemini Advanced on the First-Class Radiation Protection Supervisor Examination in Japan
Hiroki Goto, Yoshioki Shiraishi, Seiji Okada
Cureus.2024;[Epub] CrossRef
An exploratory assessment of GPT-4o and GPT-4 performance on the Japanese National Dental Examination
Masaki Morishita, Hikaru Fukuda, Shino Yamaguchi, Kosuke Muraoka, Taiji Nakamura, Masanari Hayashi, Izumi Yoshioka, Kentaro Ono, Shuji Awano
The Saudi Dental Journal.2024; 36(12): 1577. CrossRef
Evaluating the Accuracy of ChatGPT in the Japanese Board-Certified Physiatrist Examination
Yuki Kato, Kenta Ushida, Ryo Momosaki
Cureus.2024;[Epub] CrossRef

First
Prev
Page of 1
Next
Last

Search