Graphical abstract

Introduction

Background and rationale

Chat Generative Pre-Trained Transformer (ChatGPT) is a widely used large language model that has gained worldwide popularity since its release in November 2022. ChatGPT consists of a chatbot interface that generates verbal responses to a wide variety of prompts by producing a sequence of words based on previously input text. There are 2 available versions at the time of this manuscript: GPT-3.5 and GPT-4. The former is freely available, while the latter is restricted to a subscription model with time-gated limitations on its use. GPT-4 is thought to be a significant improvement over GPT-3.5 with respect to the speed and accuracy of responses. At the time of our present study, both were trained with a substantial dataset of information available on the internet through September 2021. Practical applications of the model have been made in numerous fields, including medicine. Within urology, ChatGPT has been used to manage clinical scenarios and answer items [1].

The Self-Assessment Study Program (SASP) is an educational tool jointly created and delivered by the American Board of Urology (ABU) Examination committee and the American Urological Association (AUA). This educational product is widely used by urology residents to prepare for oral board examinations and the annual urology in-service examination. Additionally, the SASP serves in part as an item bank for the ABU Continuing Urologic Certification (CUC) knowledge reinforcement activity [2].

Annually, item sets consisting of 150 multiple-choice items are released. Previous studies involving ChatGPT investigated its accuracy with the 2021 and 2022 SASP item sets. One study showed that GPT-3 (an earlier version of ChatGPT than those used in the present study) answered 42% correctly for SASP 2021 and 30% correctly for SASP 2022, suggesting a potential trend of increased accuracy of the language models for earlier iterations of SASP [1].

Objectives

The aim of our study was to characterize the accuracy of GPT-3.5 and GPT-4 over a larger sample size of standardized urology items and to determine whether there is a significant benefit to the use of GPT-4 over GPT-3.5 for answering these items. Such observations may have implications for the proposed CUC program knowledge reinforcement activity, which is administered as a self-paced, open resource examination.

Methods

Ethics statement

No human subjects or healthcare data were used in this investigation, and ethical approval was not needed.

Study design

This descriptive study investigated the performance of GPT-3.5 and GPT-4 on urology board exam-style items.

Setting

We created 700 unique multiple-choice board exam items that were in the style of—but wholly distinct from—standardized SASP program items. This approach was specifically used to avoid any degree of copyright infringement of content produced either by the AUA or the ABU. The item content was reviewed and sorted into the following categories: urologic oncology, endourology or kidney stone disease, sexual medicine or infertility, pediatric urology, reconstruction or trauma, and neurourology. Each item was standardized to resemble the format of an item stem followed by 5 letter-labeled answer choices. Representative questions from each content category and level of complexity are provided in Supplement 1.

Variables

The primary outcome measured was the accuracy of GPT-3.5 and GPT-4 with respect to answering multiple-choice items.

Data sources and measurement

The accuracy of GPT-3.5 and GPT-4 was evaluated for every multiple-choice item in February 2024. For each item, a new session of GPT-3.5 was opened. One item per session was copied and pasted into the chat box followed by the prompt: “Provide only an answer choice without explanation” (Fig. 1). The answer choice provided by ChatGPT was then compared to the correct answer determined by the study authors, and a binary result was recorded to reflect whether the answer choice did or did not coincide with the correct answer. No items required additional clarification to retrieve an answer. The same process was repeated with GPT-4. Item complexity was determined using additional prompting after the answer choices were provided by ChatGPT. This was requested using the prompt: “Is this a first, second, or third order item. Provide the answer without an explanation.” First-order items were defined to be recall of factual information. Second-order items involved interpretation of data for diagnostic purposes. Third-order items required the interpretation and integration of information for problem-solving, management, or prognostic purposes.

Bias

Each multiple-choice item was submitted to ChatGPT using a new chat session in order to avoid answer contamination across items.

Study size

Two generative artificial intelligence platforms, GPT-3.5 and GPT-4 were used to assess a total of 700 urology items.

Statistical methods

Statistical analysis was performed using the R software suite, and the chi-square test was used to compare the performance of GPT-3.5 and GPT-4 according to item complexity and item category. A 5% significance level was used.

Results

A total of 700 unique items were independently created by the study authors in the style and taxonomy of SASP items with respect to accuracy and complexity. The distribution of items according to topic domains was as follows: 192, urologic oncology; 71, endourology or kidney stone disease; 79, sexual medicine or infertility; 85, pediatric urology; 62, reconstruction or trauma; and 51, neurourology items. A total of 340 recall, 305 interpretation, and 55 problem-solving items were evaluated (Dataset 1).

GPT-3.5 answered 29.7% and GPT-4 answered 45.5% of items correctly (P<0.00001). GPT-4 performed more accurately than GPT-3.5 on items across all 3 complexity levels, rising to the level of statistical significance for recall (P<0.0001) and interpretation (P=0.005) items (Table 1). GPT-4 answered more items correctly across all six content categories; however, this was only statistically significant in the categories of sexual medicine (P=0.046), urologic oncology (P=0.028), and pediatric urology (P=0.012). These findings are graphically represented in Fig. 2.

We found that GPT-3.5 and GPT-4 both performed inconsistently with respect to each other. The proportion of items correctly answered by GPT-3.5 and incorrectly by GPT-4 was 9.1%. Twenty-five percent of items were answered correctly by only GPT-4. Forty-five percent of items were answered incorrectly by both versions, and 20.6% were answered correctly by both versions.

Discussion

Key results

GPT-4 outperformed GPT-3.5 when answering multiple-choice urology items; however, the accuracy was below passing rates for urology certification testing.

Interpretation

In this study, we show that GPT-4 and GPT-3.5 are capable of answering board exam-style urology items, although their accuracy remains poor. Deebel and Terlecki investigated the accuracy of GPT-3 with respect to SASP items from 2021 and 2022 [1]. Their study found the percentage correct to be 42.3% and 30.0% for the 2021 and 2022 respectively. The accuracy for 2022 was similar to our overall accuracy of 29.7% for GPT-3.5; however, this was achieved using a slightly more updated version of ChatGPT.

Later generations of ChatGPT have been touted as improving with respect to item accuracy for other standardized exams, such as the MCAT and USMLE [3,4]. GPT-4 has specifically been studied for significant improvements in accuracy. Out study found that GPT-4 answered significantly more items correctly than GPT-3.5. GPT-4 also did not consistently correctly answer items that were correctly answered by ChatGPT 3.5, as approximately one-tenth of all items were answered correctly by ChatGPT 3.5 and incorrectly by GPT-3.5. Overall, the higher-performing GPT-4 would still not achieve the proposed minimum passing standards for the ABU CUC knowledge reinforcement activity of 60% [2]. Prior work involving ChatGPT and urology assessment items detailed inconsistencies in answers or failures to produce a specific answer choice. Cocci et al. [5] discussed the importance of wording prompts to produce a definitive answer. In our study of 700 items, there were no instances in which ChatGPT yielded ambivalent answers. We attribute this finding to the direct wording of the prompt, which forces ChatGPT to provide a single answer choice.

Comparison with previous studies

There has been significant variability among surgical subspecialties with respect to standardized written board exam items. Ali et al. demonstrated passing rates for mock neurosurgical board exam items and accuracy that were similar between GPT-4 and item bank users [6]. ChatGPT was able to exceed trainee performance on thoracic surgery items and achieve passing rates [7]. In orthopedic surgery, plastic surgery, and ophthalmology, ChatGPT performed poorly on practice board exam items [8-10]. One study investigating the South Korean general surgery board exams found a significant improvement in accuracy in GPT-4 compared to GPT-3.5, which was not consistently replicated by our study [11].

Limitations

One limitation of this study involves changes to ChatGPT’s accuracy over time. Chen et al. [12] demonstrated that GPT-3.5 and GPT-4 have variable accuracy over time. When asked the same items in March 2023 and June 2023, GPT-4 was less accurate at identifying prime numbers in June 2023 than in March 2023; however, the contrary was true for GPT-3.5. The quality of responses also worsened in both models for code generation, and GPT-4 was less willing to answer sensitive items in June 2023 than in March 2023 [12]. Our data were collected in February 2024 and may reflect a decline in the accuracy of GPT-4. Language models such as ChatGPT are believed to be trainable from the prompts provided to them. It is therefore possible that recent studies involving the 2021 and 2022 SASP item banks may have inflated the accuracy of the responses.

Generalizability

A significant implication of the ability of artificial intelligence to accurately answer multiple-choice items is related to the recertification process of board-certified practicing urologists. The ABU is spearheading a transition of recertification to a program known as the CUC [2]. This will involve formative assessments in which at least 30% of items will be derived from the last 5 years of the SASP in an open-resource fashion. Additional items will be derived from clinically beneficial journal articles. The benefit of these assessments is that candidates can research topics and learn from the literature in a form of knowledge reinforcement. Candidates enrolled in CUC will be expected to achieve accuracy thresholds of at least 60% or 80% depending on the stage of certification, neither of which is presently achievable using ChatGPT. It is possible that as ChatGPT improves, these items may be answerable by artificial intelligence without significant knowledge gain on the part of the user, bypassing the learning experience.

Suggestions

Our methodology excluded items that involved the interpretation of images, urodynamics, charts, or other visual data. These items tend to have higher complexity. Therefore, the omission of these items reduces the total number of available interpretation and problem-solving items available in the study. With the recent release of image interpretation in GPT-4, further study of these items may be possible.

Conclusion

Our study demonstrates that although GPT-3.5 and GPT-4 are capable of answering multiple-choice standardized urology board exam items, their accuracy is inconsistent both overall and across categories. GPT-4 generally performs better than GPT-3.5; however, the accuracy of GPT-4 is poor compared to the minimum passing standard of the ABU CUC. These results show that although ChatGPT can confidently produce answers, those answers should be scrutinized by the reader for accuracy and cannot be wholly relied upon for the purposes of education or board certification.

Supplementary materials

Fig. 1.

Example of a multiple-choice item and prompt submitted to ChatGPT, along with the response.

Fig. 2.

Percentage of items correct by GPT-3.5 and GPT-4 by item category.

^*P<0.05 (statistically significant).

References

• 1. Deebel NA, Terlecki R. ChatGPT performance on the American Urological Association self-assessment study program and the potential influence of artificial intelligence in urologic training. Urology 2023;177:29-33. https://doi.org/10.1016/j.urology.2023.05.010 ArticlePubMed
• 2. Continuing Urologic Certification (CUC) [Internet]. American Board of Urology; 2024 [cited 2024 Feb 15]. Available from: https://www.abu.org/curologicc/overview
• 3. Bommineni VL, Bhagwagar S, Balcarcel D, Bommineni V, Davazitkos C, Boyer D. Performance of ChatGPT on the MCAT: the road to personalized and equitable premedical learning. MedRxiv [Preprint] 2023 Jun 6 https://doi.org/10.1101/2023.03.05.23286533 Article
• 4. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2:e0000198. https://doi.org/10.1371/journal.pdig.0000198 ArticlePubMedPMC
• 5. Cocci A, Pezzoli M, Lo Re M, Russo GI, Asmundo MG, Fode M, Cacciamani G, Cimino S, Minervini A, Durukan E. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer Prostatic Dis 2024;27:103-108. https://doi.org/10.1038/s41391-023-00705-y ArticlePubMed
• 6. Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS, Asaad WF, Cielo D, Oyelese AA, Doberstein CE, Gokaslan ZL, Telfeian AE. Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations. Neurosurgery 2023;93:1353-1365. https://doi.org/10.1227/neu.0000000000002632 ArticlePubMed
• 7. Gencer A, Aydin S. Can ChatGPT pass the thoracic surgery exam? Am J Med Sci 2023;366:291-295. https://doi.org/10.1016/j.amjms.2023.08.001 ArticlePubMed
• 8. Lum ZC. Can artificial intelligence pass the American Board of Orthopaedic Surgery Examination?: orthopaedic residents versus ChatGPT. Clin Orthop Relat Res 2023;481:1623-1630. https://doi.org/10.1097/CORR.0000000000002704 ArticlePubMedPMC
• 9. Gupta R, Herzog I, Park JB, Weisberger J, Firouzbakht P, Ocon V, Chao J, Lee ES, Mailey BA. Performance of ChatGPT on the Plastic Surgery Inservice Training Examination. Aesthet Surg J 2023;43:NP1078-NP1082. https://doi.org/10.1093/asj/sjad128 ArticlePubMed
• 10. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. Ophthalmol Sci 2023;3:100324. https://doi.org/10.1016/j.xops.2023.100324 ArticlePubMedPMC
• 11. Oh N, Choi GS, Lee WY. ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models. Ann Surg Treat Res 2023;104:269-273. https://doi.org/10.4174/astr.2023.104.5.269 ArticlePubMedPMC
• 12. Chen L, Zaharia M, Zou J. How is ChatGPT’s behavior changing over time? arXiv [Preprint] 2023 Oct 31 https://doi.org/10.48550/arXiv.2307.09009 Article

Figure & Data

References

Citations

Citations to this article as recorded by  

• From GPT-3.5 to GPT-4.o: A Leap in AI’s Medical Exam Performance
  Markus Kipp
  Information.2024; 15(9): 543.     CrossRef