Journal of Educational Evaluation for Health Professions

Research article

ChatGPT (GPT-4) passed the Japanese National License Examination for Pharmacists in 2022, answering all items including those with diagrams: a descriptive study: Hiroyasu Sato, Katsuhiko Ogasawara; J Educ Eval Health Prof. 2024;21:4. Published online February 28, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.4

6,782 View
330 Download
12 Web of Science
15 Crossref

Purpose
The objective of this study was to assess the performance of ChatGPT (GPT-4) on all items, including those with diagrams, in the Japanese National License Examination for Pharmacists (JNLEP) and compare it with the previous GPT-3.5 model’s performance.
Methods
The 107th JNLEP, conducted in 2022, with 344 items input into the GPT-4 model, was targeted for this study. Separately, 284 items, excluding those with diagrams, were entered into the GPT-3.5 model. The answers were categorized and analyzed to determine accuracy rates based on categories, subjects, and presence or absence of diagrams. The accuracy rates were compared to the main passing criteria (overall accuracy rate ≥62.9%).
Results
The overall accuracy rate for all items in the 107th JNLEP in GPT-4 was 72.5%, successfully meeting all the passing criteria. For the set of items without diagrams, the accuracy rate was 80.0%, which was significantly higher than that of the GPT-3.5 model (43.5%). The GPT-4 model demonstrated an accuracy rate of 36.1% for items that included diagrams.
Conclusion
Advancements that allow GPT-4 to process images have made it possible for LLMs to answer all items in medical-related license examinations. This study’s findings confirm that ChatGPT (GPT-4) possesses sufficient knowledge to meet the passing criteria.

Citations

Citations to this article as recorded by

Performance of ChatGPT‐3.5 and ChatGPT‐4o in the Japanese National Dental Examination
Osamu Uehara, Tetsuro Morikawa, Fumiya Harada, Nodoka Sugiyama, Yuko Matsuki, Daichi Hiraki, Hinako Sakurai, Takashi Kado, Koki Yoshida, Yukie Murata, Hirofumi Matsuoka, Toshiyuki Nagasawa, Yasushi Furuichi, Yoshihiro Abiko, Hiroko Miura
Journal of Dental Education.2025; 89(4): 459. CrossRef
Qwen-2.5 Outperforms Other Large Language Models in the Chinese National Nursing Licensing Examination: Retrospective Cross-Sectional Comparative Study
Shiben Zhu, Wanqin Hu, Zhi Yang, Jiani Yan, Fang Zhang
JMIR Medical Informatics.2025; 13: e63731. CrossRef
ChatGPT (GPT-4V) Performance on the Healthcare Information Technologist Examination in Japan
Kai Ishida, Eisuke Hanada
Cureus.2025;[Epub] CrossRef
Medication counseling for OTC drugs using customized ChatGPT-4: Comparison with ChatGPT-3.5 and ChatGPT-4o
Keisuke Kiyomiya, Tohru Aomori, Hisakazu Ohtani
DIGITAL HEALTH.2025;[Epub] CrossRef
Current Use of Generative Artificial Intelligence in Pharmacy Practice: A Literature Mini-review
Keisuke Kiyomiya, Tohru Aomori, Hitoshi Kawazoe, Hisakazu Ohtani
Iryo Yakugaku (Japanese Journal of Pharmaceutical Health Care and Sciences).2025; 51(4): 177. CrossRef
Performance evaluation of large language models for the national nursing examination in Japan
Tomoki Kuribara, Kengo Hirayama, Kenji Hirata
DIGITAL HEALTH.2025;[Epub] CrossRef
Harnessing ChatGPT for digital tools in pharmacy practice
Reginald Amin Yakob, Adeola Bamgboje-Ayodele, Jack C. Collins, Parisa Aslani
Research in Social and Administrative Pharmacy.2025; 21(11): 943. CrossRef
Performance Evaluation of 18 Generative AI Models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese Pharmacist Licensing Examination: Comparative Study
Hiroyasu Sato, Katsuhiko Ogasawara, Hidehiko Sakurai
JMIR Medical Education.2025; 11: e76925. CrossRef
Applications and potential of ChatGPT in dentistry: Scoping review of research perspectives
Masakazu Hamada, Sumire Kikuchi, Tatsuya Akitomo, Satoru Kusaka, Yuko Iwamoto, Ryota Nomura
Journal of Dental Sciences.2025;[Epub] CrossRef
Evaluation of the Accuracy and Reliability of Responses Generated by Artificial Intelligence Related to Clinical Pharmacology
Michal Ordak, Julia Adamczyk, Agata Oskroba, Michal Majewski, Tadeusz Nasierowski
Journal of Clinical Medicine.2025; 14(21): 7563. CrossRef
Performance of ChatGPT-4 on the French Board of Plastic Reconstructive and Aesthetic Surgery written exam: a descriptive study
Emma Dejean-Bouyer, Anoujat Kanlagna, François Thuau, Pierre Perrot, Ugo Lancien
Journal of Educational Evaluation for Health Professions.2025; 22: 27. CrossRef
Potential of ChatGPT to Pass the Japanese Medical and Healthcare Professional National Licenses: A Literature Review
Kai Ishida, Eisuke Hanada
Cureus.2024;[Epub] CrossRef
Performance of Generative Pre-trained Transformer (GPT)-4 and Gemini Advanced on the First-Class Radiation Protection Supervisor Examination in Japan
Hiroki Goto, Yoshioki Shiraishi, Seiji Okada
Cureus.2024;[Epub] CrossRef
An exploratory assessment of GPT-4o and GPT-4 performance on the Japanese National Dental Examination
Masaki Morishita, Hikaru Fukuda, Shino Yamaguchi, Kosuke Muraoka, Taiji Nakamura, Masanari Hayashi, Izumi Yoshioka, Kentaro Ono, Shuji Awano
The Saudi Dental Journal.2024; 36(12): 1577. CrossRef
Evaluating the Accuracy of ChatGPT in the Japanese Board-Certified Physiatrist Examination
Yuki Kato, Kenta Ushida, Ryo Momosaki
Cureus.2024;[Epub] CrossRef

First
Prev
Page of 1
Next
Last

Search