Usefulness and Accuracy of Artificial Intelligence Chatbot Responses to Patient Questions for Neurosurgical Procedures

Introduction: The internet has become a primary source of health information, leading patients to seek answers online before consulting healthcare providers. This study aims to evaluate the implementation of ChatGPT in neurosurgery by assessing the accuracy and helpfulness of AI-generated responses to common post-surgical questions.

Methods: A list of 60 commonly asked questions regarding neurosurgical procedures were developed. ChatGPT-3.0 and ChatGPT-3.5 respones to these qestions were recorded and graded by numerous practioners for accuracy and helpfulness. Understandability and actionability of the answers were assessed using the Patient Education Materials Assessment Tool (PEMAT). Readability analysis was conducted using established scales.

Results: In total, 720 questions were graded, with 360 being ChatGPT-3.0 responses and 360 being ChatGPT-3.5 responses. The mean helpfulness score across the three subsections was 3.511 ± 0.647, while the accuracy score was 4.165 ± 0.567. The PEMAT analysis revealed that the AI-generated responses had higher scores for actionability compared to understandability. This indicates that the responses provided practical guidance and recommendations that patients could apply effectively. On the other hand, the mean Flesch Reading Ease (FRE) score was 33.5, suggesting that the readability level of the responses was relatively difficult. The Raygor Readability Estimate (RRE) scores ranged within the graduate level, with an average score of the 15th grade.

Conclusion : The AI chatbot's responses were factually accurate but not perceived as highly beneficial by raters. There were no significant differences in accuracy or helpfulness between ChatGPT-3.0 and ChatGPT-3.5. AI generated responses had improved readability compared to existing materials.