Graphical abstract

Background/rationale

Constructing objective structured clinical examination (OSCE) stations is time-consuming for medical teachers, as each station must be carefully written, reviewed, and tested to ensure its quality. Any means of reducing the time required to produce valid OSCE stations could increase the number of stations available to students, facilitate the creation of station databases, and ultimately enhance training. Large language models (LLMs) are increasingly used in medical education and have demonstrated the ability to perform OSCEs, simulate standardized participants, and assess conversational OSCEs [1-3]. In contrast, their application to the generation of highly complex educational content, such as OSCE stations, remains largely unexplored [4].

Objectives

We hypothesized that the GPT-4o LLM could support medical teachers in producing ready-to-use OSCE stations. The primary objective of this study was to assess the quality of GPT-4o-generated OSCE stations compared with similar stations written by trained medical teachers.

Ethics statement

No institutional review board approval was required.

Study design and setting

The study used the METRICS checklist (Supplement 1, Table S1) to guide the design and reporting of the use of generative artificial intelligence in medical education [5]. Five GPT-4o-generated OSCE stations were blindly compared with 5 “mirror” stations written by a medical teacher and targeting the same learning objectives. The study followed 4 sequential steps: prompt engineering, station writing by the medical teacher, station generation by GPT-4o (up to 3 attempts), and blinded assessment of the stations. Writing and generation times were recorded. The study involved 1 assistant professor responsible for prompt engineering and for writing and generating the OSCE stations, as well as 7 academic physicians (1 full professor and 6 assistant professors). To ensure consistency in evaluation, each teacher was selected based on training and experience in OSCE development and evaluation; all were certified after completing the French national OSCE evaluator training course (https://formation.uness.fr/formation/course/view.php?id=20081). The paid version of GPT-4o was selected because of its availability and because it outperformed other LLMs in prior evaluations [6].

Data sources/measurement

All data were collected prospectively. Compliance with quality standards for OSCE stations was evaluated using 5-point Likert scales (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree) (Supplement 1, Table S2) based on the French guidelines for medical OSCE assessors (OSCE vademecum: https://formation.uness.fr/formation/pluginfile.php/332080/mod_resource/content/5/Vademecum%20ECOS.pdf) and a published assessment grid [7]. Each teacher independently evaluated each station using the Likert-scale–based assessment grid. Individual item ratings were collected at the teacher level and analyzed across teachers for each item, with results presented as the distribution of Likert-scale responses by station type (GPT-generated vs. teacher-written). The primary endpoint was the ready-to-use quality of the OSCE stations, which was defined at the station level based on a majority rule: a station was considered ready-to-use if at least 4 of the 7 teachers responded “agree” or “strongly agree” to the statement “The station is good enough to be used by students.” This threshold was defined pragmatically to reflect consensus among expert assessors, consistent with common practices in educational evaluation based on aggregated expert judgment. Assessment of OSCE station quality was blinded to the origin of the stations (GPT-4o-generated vs. teacher-written).

Statistical methods

Categorical variables are presented as frequencies and percentages, and continuous variables are presented as medians with interquartile ranges (IQRs). Fisher exact tests and paired Wilcoxon signed-rank tests were used as appropriate. Inter-rater agreement was assessed using the intraclass correlation coefficient (ICC). Analyses were performed using R software ver. 4.4.1 (The R Foundation for Statistical Computing).

Main results

Prompt engineering

The large language model used was GPT-4o (OpenAI), accessed in August 2024 via the ChatGPT application. Default model parameters were applied. Prompt engineering was performed by a single medical teacher (M.D.) using a progressive increase in prompt complexity, ranging from a simple instruction (“write an OSCE station about septic shock for medical students”) to a 3,000-word prompt. The final prompt was structured into 4 components: instructions regarding tags, context and general objectives, review of the reference course, and step-by-step generation of each section (general information, clinical situation, instructions for the standardized participant, iconography, and assessment grid). It assigned GPT-4o the role of a medical teacher and included reference teaching material on sepsis and septic shock, ordered tasks with chained instructions, examples, and structured tags (“context,” “task,” “instruction,” and “refine”) designed to optimize the LLM’s output. A translation of the prompt is available in Supplement 1. The final version of the prompt consistently generated OSCE stations on septic shock that included all expected sections (clinical situation, learning objectives, standardized participant script, laboratory results, and evaluation grid), regardless of content relevance. This prompt was validated on other medical topics, which were not included in the final evaluation, to ensure reliability. Prompt calibration required a total of 24 hours of work.

Generation of the stations

The prespecified characteristics of the 5 OSCE stations were based on the national second-cycle knowledge item list and are presented in Table 1. Writing and generation times are also reported in Table 1. The total time required to generate an OSCE station using GPT-4o comprised model output time, formatting according to the OSCE template (GPT-4o was unable to generate content directly in .docx format during prompt engineering, despite being provided with a corresponding .docx template), and proofreading without major content modification. For paired OSCE stations, the median time required for manual writing was 54 minutes (IQR, 53–55 minutes), compared with 24 minutes (IQR, 22–31 minutes) using GPT-4o, yielding a median paired time saving of 29 minutes (IQR, 20–33 minutes) (P=0.06). GPT-4o successfully generated relevant laboratory results, including antibiotic susceptibility testing and arterial blood gas analyses, but failed to produce appropriate electrocardiogram tracings or chest X-ray images (Figs. 1, 2). Electrocardiogram tracings were replaced with real tracings for the evaluation, whereas the chest X-ray was retained. In addition, GPT-4o failed to generate an OSCE station involving a standardized patient for OSCE station 3 and consistently produced a station involving a standardized healthcare professional instead across all 3 attempts.

Key results

One of the 5 GPT-4o-generated OSCE stations was considered ready-to-use, compared with all 5 teacher-written stations (Table 2). Discrepancies between learning objectives and evaluation grids and lack of accuracy were the main issues identified by teachers in the GPT-4o-generated OSCE stations. GPT-4o occasionally failed to appropriately align the assessment grid with the clinical data or the instructions provided to standardized patients. No concerns were raised regarding the generated laboratory results. All assessors noted the inappropriateness of the chest X-ray. Hallucinations generated by GPT-4o may have occurred, as some assessment grid items or learning objectives were not aligned with the OSCE vademecum. Details of compliance with the OSCE vademecum for each station are presented in Fig. 3. Inter-rater agreement was moderate, with an ICC of 0.65 (95% confidence interval, 0.55–0.73).

Interpretation

Our study compared 5 GPT-4o-generated OSCE stations with 5 teacher-written OSCE stations targeting identical learning objectives. Only one GPT-4o-generated OSCE station was deemed ready-to-use for student training, compared with all 5 teacher-written stations. Although most clinical situations were considered realistic and the instructions provided to students and standardized participants were considered clear, lack of realism in some stations, the presence of hallucinations, and low-quality assessment grids prevented most GPT-4o-generated stations from being considered ready-to-use. Interestingly, station 3, for which GPT-4o failed to generate a scenario involving a standardized patient, was paradoxically rated higher than both the corresponding teacher-written station and the other GPT-4o-generated stations. This finding may be explained by the fact that the teacher-written station was considered too difficult for students, particularly with regard to interpretation of the antibiogram, and lacked realism because it required explanation of an antibiogram to a standardized patient. Furthermore, because this station was the most difficult to generate due to repeated non-compliance with the instructions, 3 attempts were required, which may partly explain why it performed slightly better than the other 4 GPT-4o-generated cases. The inability to generate accurate medical imaging limited the range of clinical situations that could be realistically produced. The time saved during station generation was not significantly different from that required for manual OSCE writing and may be offset by the extensive revision needed for GPT-4o-generated stations.

Comparison with previous study

Another study compared 3 prompting strategies for generating OSCE stations in the field of digital health education [4]. Their simulated-agent GPT strategy—similar to our prompting approach but relying on sequential generation with provision of intermediate outputs before each subsequent step—achieved high compliance with the prompt, except for the assessment grid. Consistent with our findings, they reported difficulties for GPT-4o in consistently producing observable, distinct, and sufficiently detailed assessment items that would allow reproducible grid completion. Although they did not report hallucinations, they identified inconsistencies such as inaccurate or missing information and implicit answers. Their decision to split tasks between GPT agents resembles our strategy of adding tags to different tasks to distinguish contextual explanations, task structure, specific outputs, and refinement of previous generations [8]. Another study focusing on artificial intelligence-generated medical cases—which are comparable to OSCE stations given their complexity—also highlighted the strong potential of such tools in medical education while emphasizing the need to improve case realism, consistency, and relevance [9]. Neither of these studies estimated time savings or compared the quality of generated cases with standard teacher-written cases.

Limitations/generalizability

Our study has several limitations. The success rate for generating ready-to-use OSCE stations was lower in our study (1 out of 5) than in previous studies [4,9]. This difference may be explained by our prompting strategy, which aimed to generate a complete OSCE station in a single output, whereas previous studies relied on sequential, step-by-step case generation. As inputs become longer, LLMs tend to use the available information less consistently [10]. Furthermore, prompt engineering, station generation, and station writing were all performed by the same teacher, which may have introduced bias. The performance of LLMs is rapidly evolving, and future improvements may affect the reproducibility and comparability of results obtained using identical prompts. Our a priori objective was not to modify the generated content and to obtain a fully “ready-to-use” OSCE station in each case, which may have been overly ambitious. Rather than acting as a complete substitute for OSCE writing, LLMs may be better suited as tools for saving time on specific components of OSCE construction, such as drafting clinical scenarios or standardized patient scripts. However, GPT-4o outputs are not reproducible, even when the same prompt is used repeatedly.

Suggestions

GPT-4o and future versions may warrant further exploration as supportive tools, for example by generating clinical scenarios aligned with predefined learning objectives while using assessment grids established beforehand by expert medical teachers. Such an approach may reduce the risk of inconsistency between the clinical case and the assessment grid while also saving time.

Conclusion

GPT-4o failed to consistently produce fully ready-to-use OSCE stations. However, it may warrant further evaluation as a supportive and time-saving tool for generating specific components of OSCE stations under the supervision of medical teachers.

Supplementary materials

Fig. 1.

(A, B) Electrocardiogram tracings generated by GPT-4o for objective structured clinical examination station number 1. (A) Generated using the prompt: “generate a typical electrocardiogram tracing of an anterior myocardial infarction.” (B) Generated using the prompt: “generate an electrocardiogram tracing with ST-segment elevation in leads V1 to V6, I, and aVL.” Both tracings failed to produce a clinically correct electrocardiogram.

Fig. 2.

Chest X-ray generated by GPT-4o for objective structured clinical examination station number 2. Generated using the prompt: “generate a chest X-ray showing a right lower lobe pneumonia.” GPT-4o produced an image resembling a chest X-ray but failed to depict alveolar consolidation in the right lower lobe. Notably, several anatomical abnormalities can be observed, including an additional right rib, vertebral compression, and apparent absence of spinous processes.

Fig. 3.

Compliance of objective structured clinical examination (OSCE) stations with OSCE vademecum. Distribution of Likert-scale responses of teachers by station type (GPT-4o-generated vs. teacher-written). “Clarity of instructions” corresponded to questions 1 to 3 of the station assessment grid. “Accuracy of information” included questions 6 and 8. “Realism of the clinical situation” was assessed by question 4, while “Relevance of the objective” was assessed by question 5. “Feasibility for actors and students” encompassed questions 7 and 15. Finally, “Pedagogical validity of the assessment grid” included questions 9 to 14. Details of the assessment grid are provided in Supplement 1.

References

• 1. Luordo D, Torres Arrese M, Tristán Calvo C, Shani Shani KD, Rodríguez Cruz LM, García Sánchez FJ, Lagares Gómez-Abascal A, Rubio García R, Delgado Jiménez J, Pérez Carreras M, Diez Lobato R, Granizo Martínez JJ, Tung-Chen Y, Villena Garrido MV. Application of artificial intelligence as an aid for the correction of the objective structured clinical examination (OSCE). Appl Sci 2025;15:1153. https://doi.org/10.3390/app15031153 Article
• 2. Li SW, Kemp MW, Logan SJ, Dimri PS, Singh N, Mattar CN, Dashraath P, Ramlal H, Mahyuddin AP, Kanayan S, Carter SW, Thain SP, Fee EL, Illanes SE, Choolani MA; National University of Singapore Obstetrics and Gynecology Artificial Intelligence (NUS OBGYN-AI) Collaborative Group. ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology. Am J Obstet Gynecol 2023;229:172. https://doi.org/10.1016/j.ajog.2023.04.020 Article
• 3. Pereira DS, Falcão F, Nunes A, Santos N, Costa P, Pêgo JM. Designing and building OSCEBot^® for virtual OSCE: performance evaluation. Med Educ Online 2023;28:2228550. https://doi.org/10.1080/10872981.2023.2228550 ArticlePubMedPMC
• 4. Zouakia Z, Logak E, Szymczak A, Jais JP, Burgun A, Tsopra R. AI-driven objective structured clinical examination generation in digital health education: comparative analysis of three GPT-4o configurations. JMIR Med Educ 2026;12:e82116. https://doi.org/10.2196/82116 ArticlePubMedPMC
• 5. Sallam M, Barakat M, Sallam M. A preliminary checklist (METRICS) to standardize the design and reporting of studies on generative artificial intelligence-based models in health care education and practice: development study involving a literature review. Interact J Med Res 2024;13:e54704. https://doi.org/10.2196/54704 ArticlePubMedPMC
• 6. Indran IR, Paranthaman P, Gupta N, Mustafa N. Twelve tips to leverage AI for efficient and effective medical question generation: a guide for educators using Chat GPT. Med Teach 2024;46:1021-1026. https://doi.org/10.1080/0142159X.2023.2294703 ArticlePubMed
• 7. Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration. Med Teach 2013;35:e1447-e1463. https://doi.org/10.3109/0142159X.2013.818635 ArticlePubMed
• 8. Jowsey T, Stokes-Parish J, Singleton R, Todorovic M. Medical education empowered by generative artificial intelligence large language models. Trends Mol Med 2023;29:971-973. https://doi.org/10.1016/j.molmed.2023.08.012 ArticlePubMed
• 9. Takahashi H, Shikino K, Kondo T, Komori A, Yamada Y, Saita M, Naito T. Educational utility of clinical vignettes generated in Japanese by ChatGPT-4: mixed methods study. JMIR Med Educ 2024;10:e59133. https://doi.org/10.2196/59133 ArticlePubMedPMC
• 10. Liu NF, Lin K, Hewitt J, Paranjape A, Bevilacqua M, Petroni F, Liang P. Lost in the Middle: how language models use long contexts. Trans Assoc Comput Linguist 2024;12:157-173. https://doi.org/10.1162/tacl_a_00638 Article

Figure & Data

References

Citations

Citations to this article as recorded by  

• El papel emergente de la inteligencia artificial en los exámenes clínicos objetivos estructurados
  Nelson Luis Cahuapaza-Gutierrez, Cielo Cinthya Calderon-Hernandez
  Medical Education & Clinical Practice.2026; 1(3): e032.     CrossRef