ChatGPT an Evidence Based Neurosurgical Patient Resource? Assessing the accuracy of ChatGPT in Regard to Patient Questions Compared to Findings of Published Neurosurgical Literature

Introduction: Artificial intelligence (AI) has the potential to provide accurate information and improve the accessibility of medical literature. As AI platforms are increasingly utilized, evaluating the accuracy of health-related guidance provided to patients is vital. This study aims to evaluate the accuracy of information publicly accessible that AI platforms provide compared to recently published literature regarding spondylolisthesis, movement disorders, glioblastoma multiforme (GBM), and endovascular conditions.

Methods: Four AI platforms were assessed: ChatGPT-3.5, ChatGPT-4.0, Perplexity.AI, and Bing AI. Sets of patient-centered questions regarding treatment of spondylolisthesis, movement disorders, GBM, and endovascular elective intracranial aneurysm repair were posed to these platforms based on neurosurgical research findings. Responses were assessed for accuracy in comparison to published literature.

Results: Model accuracy was quantified as follows: GPT3.5 (58.11%), GPT4 (56.76%), Perplexity (66.22%), and BingAI (56.76%). Friedman’s test indicated no significant difference among the models in accuracy (χ2=1.599, p=0.660). Notably, the Wilcoxon signed-rank test revealed that GPT4, Perplexity, and Bing were all significantly better than GPT3.5 at identifying the correct article (p < 0.001). Further, Perplexity.AI was significantly better than GPT4 (p=0.040), but not BingAI. No difference was found in regards to AI platforms citing articles that were open access compared to subscription based (59% v. 60%, p=0.934).

Conclusion : Popular AI platforms currently lack the ability to consistently and accurately answer common neurosurgical patient questions with evidence based answers. Patients should be cautioned on their use of Popular AI platforms as a supplementary source with caution. As neurosurgical focused AI is being developed, it is imperative that the development of AI capable of answering neurosurgical patients questions with accurate evidenced based answers not be overlooked. Future efforts to develop patient facing AI platforms will aid in improving patient health literacy, potentially improving healthcare utilization and outcomes.