Using artificial intelligence models to improve the readability of neurosurgical literature for patients

Introduction: This study assessed the ability of Chat GPT3.5 and GPT4 to generate readable and accurate summaries of published neurosurgical literature.

Methods: Random abstracts published in journal issues released between June - August 2023 (n=150) were selected from the top 5 ranked neurosurgical journals. ChatGPT models were instructed to generate a readable summary of the original abstract from a statistically validated prompt. Readability scores and grade-level indicators (RR-GLIs) were calculated for GPT3.5 and GPT4 summaries and original abstracts. Two physicians independently rated the accuracy of ChatGPT-generated layperson summaries to ensure scientific validity. ANOVA and chi-squared tests were performed to compare readability scores. Cohen’s Kappa was used to assess interrater reliability among the 2 rater physicians.

Results: Analysis of 150 original abstracts showed a statistically significant difference for all RR-GLIs between the ChatGPT-generated summaries and original abstracts. The following readability scores are formatted (original abstract mean, GPT3.5 summary mean, GPT4 summary mean, P value): Flesh Kincade Reading Grade (12.55, 7.80, 7.70, p< 0.0001); Gunning Fog Score (15.46, 10.00, 9.00, p< 0.0001); Smog Index (11.30, 7.13, 6.60, p< 0.0001); Coleman Liau Index (14.67, 11.32, 10.26, p< 0.0001); Automated Readability Index (10.87, 8.50, 7.75 p< 0.0001); Flesh Kincade Reading Ease (33.29, 68.45, 69.55 p< 0.0001). GPT4-generated summaries demonstrated higher RR-GLIs than GPT3.5-generated summaries in the following categories: Gunning Fog Score (0.0003); Smog Index (0.027); Coleman Liau Index ( <.0001); Sentences ( < 0.0001); Complex Words ( < 0.0001); % Complex Words (.0035). Respectively, GPT3.5 and GPT4-generated summaries maintained an overall scientific validity of 68.4% and 84.2%.

Conclusion : Both ChatGPT models generated clear scientific summaries of neurosurgical abstracts. GPT4 demonstrated stronger potential for providing readable and accurate neurosurgical literature accessible to the public.