Large language models (LLMs) are artificial intelligence (AI) models trained through self-supervised learning to generate natural-language text. They can respond to unstructured content on specific subjects [1], and their applications are becoming increasingly versatile. LLMs can rapidly organize and present large volumes of information accumulated online, functioning as user-friendly tools for information synthesis. Over the years, researchers have sought to integrate AI into medicine because the field is broad and highly specialized, with extensive use of technical terminology. Accordingly, studies have assessed whether LLM-generated responses are accurate and presented in an easily understandable manner [2–5]. Several studies have demonstrated that LLMs can assist physicians with diagnoses, suggest possible treatment options, and translate discharge summaries into language that is easier for patients to understand [6–10].

Meanwhile, ChatGPT 4.0, the state-of-the-art model at that time, reportedly achieved passing grades on neurology board-style questions [11]. It also answered questions about multiple sclerosis with readability, satisfaction, and empathy scores comparable to those of a neurologist [12]. However, its decision-making performance was less accurate, showing only 54.3% agreement with neurologists in deciding whether to perform mechanical thrombectomy. Errors occurred in 8.8% of cases because of mathematical or logical mistakes and misinterpretation of clinical information [6]. Therefore, validation is needed before LLMs can be directly applied in medical practice, particularly with respect to the accuracy and safety of aggregated information, the appropriateness of clinical interpretation, readability, and conflicts of interest [13–16].

Managing cardiovascular risk factors has been shown to reduce mortality among patients with stroke [17]. Because stroke has various etiologies and treatment approaches differ according to cause, effective patient education requires individualized guidance based on each patient’s pathophysiology. This study aimed to evaluate the natural language processing capabilities of LLMs in providing personalized patient education and to assess their user-friendliness. To this end, we compared LLM-generated materials with materials written by a human neurologist in terms of accuracy, legality, ethics, comprehensiveness, and information delivery, using a newly developed evaluation questionnaire.

Of the 136 participants contacted, 132 responded (response rate, 97.1%), whereas four non-medical personnel did not respond because they had difficulty understanding the survey questions. Fifty-two participants (39.4%) were female, and most participants were in their 40s (59%). A substantial proportion of participants agreed with the use of AI in healthcare (79.5%), and a slightly smaller proportion agreed with the use of LLMs in healthcare (72.7%). The characteristics of the study population are summarized in Table 3.

This study indicates that both ChatGPT and Claude can generate patient education materials comparable to those written by a neurologist. ChatGPT received high ratings in both scenarios, particularly for accuracy and comprehensiveness. No significant differences in perceptions of LLM performance were observed among healthcare professionals, patients/caregivers, research workers, and the general public. Furthermore, participants who were willing to accept LLMs in healthcare rated the explanations generated by ChatGPT and Claude more favorably than those who were not.

To our knowledge, this is among the first studies to assess the utility of LLMs for individualized education in clinical settings by comparing LLM-generated materials with specialist-authored materials based on realistic clinical scenarios. Previous studies have mainly assessed the accuracy and comprehensiveness of responses to frequently asked questions about various diseases [2,12,18,19]. However, these assessments focused primarily on whether LLMs could gather fundamental knowledge that is easily accessible through standard online searches. In contrast, this study demonstrated the capability of LLMs to organize specialized knowledge, such as stroke-related information, and convey it clearly and comprehensibly. The LLMs interpreted the provided clinical information and synthesized it into an organized format for users, offering guidance on health management strategies to prevent recurrence and important precautions for daily life (Supplementary Table S2). Most previous studies evaluated readability and comprehension solely through healthcare professionals [14,19] or scoring tools [12,18]. Because comprehensibility and readability ultimately depend on how patients perceive the information, those studies had limitations in evaluating whether LLMs could genuinely convey information in a manner comprehensible to non-experts. By involving both patients with stroke and members of the general public, the present study was able to evaluate this capability more directly. This study also provides insight into the potential applicability of LLMs in real-world settings.

The terminology used by LLMs was assessed across five domains: accuracy, legality, ethics, comprehensiveness, and information delivery. Participants, including both patients and healthcare professionals, indicated that explanations written by LLMs were easier to understand than those written by the neurologist. This may be because LLMs organized the content with subheadings and list formatting, enabling readers to grasp complex information more readily. However, subtle content differences were observed between the explanations provided by the neurologist and those generated by the LLMs. Unlike the neurologist, the LLMs did not explain the bleeding risks of medications that patients needed to take. In addition, although the neurologist explained the rationale for medication selection based on stroke etiology, the LLMs focused primarily on the efficacy of taking the medications without providing this contextual explanation. This finding is consistent with previous research. A study conducted by Yeo et al. indicated that LLMs may identify diagnostic algorithms but omit details such as cutoff values for diagnostic tests and may fail to provide appropriate information relevant to patients with liver cirrhosis and hepatocellular carcinoma [4]. Similarly, the LLMs successfully obtained information on certain medications used for stroke treatment but failed to explain when those medications should be used. Although LLMs may find it challenging to truly understand the mechanisms of diagnosis and treatment, the survey results indicated that the generated texts were medically appropriate compared with neurologist-authored materials.

LLMs received lower legality and ethics scores than accuracy and comprehensiveness scores. ChatGPT received an average score above 4 for accuracy and comprehensiveness but scored in the mid-3-point range for legality and ethics, with most participants selecting “neither.” Claude showed no significant difference in these domains compared with the neurologist. This may be because most participants were uncertain about using LLMs in medical settings and lacked sufficient knowledge about AI, as 52.3% responded “neither” when asked whether they were familiar with AI.

No significant differences in ratings were observed among the subgroups (Supplemental Table 3). This suggests that perceptions of LLM performance were consistent regardless of occupational background. Instead, score differences were more apparent according to whether participants agreed with the use of LLMs. Although other factors, including technology education status, may have influenced individual attitudes toward LLMs [20], the present findings suggest that positive changes in patients’ views of LLMs may promote the seamless integration of LLMs into healthcare.

Although LLMs were shown to handle complex information, such as medical information, several issues must be addressed before they can be applied in medical settings. First, consistency needs to be improved. Pugh et al. showed that the consistency of LLMs can be influenced by the model used and the temperature parameter when estimating ratings of speech samples for coherence, content, and tangentiality among patients diagnosed with schizophrenia [21]. Second, LLMs are vulnerable to hallucinations, which could mislead users by presenting incorrect information with confidence. In research using LLMs for outpatient rehabilitation, Google Bard confidently referenced websites that did not exist [14]. Thus, materials created by LLMs must be reviewed by specialists before use to prevent patient harm. In addition, implementing LLMs to process medical data on overseas servers presents substantial regulatory challenges, particularly regarding compliance with cross-border data transfer requirements. Even temporary data transmission requires adherence to stringent legal requirements, including explicit consent from data subjects, as mandated by domestic personal information protection laws and the European Union General Data Protection Regulation. These regulatory frameworks remain applicable regardless of temporary storage provisions or automatic deletion protocols. Beyond immediate compliance concerns, the use of LLMs in healthcare raises additional privacy considerations, including the risk of unintentional retention of personal information in online spaces and the need for clear guidelines on source attribution and copyright compliance in LLM outputs [22]. Given the highly sensitive nature of medical information and these multifaceted regulatory challenges, implementing isolated models may be a viable technical solution to ensure both data protection and regulatory compliance [20].

Considering that this was an online survey, most individuals who actively participated may have been familiar with digital environments and potentially more favorable toward LLMs. However, as familiarity with LLMs increases over time, similar results may be expected among future patients with stroke. Several limitations should be considered when interpreting the findings of this study. First, the use of fictional patient scenarios, although valuable for controlled assessment, may only partially capture the complexity and unpredictability of real patient cases. This limits the generalizability of the results to actual clinical settings. Second, because the study participants were Korean, the LLMs may have received relatively high scores. In Korea, where public Wi-Fi is widely available, even older adults, including patients with stroke, tend to be relatively digitally connected, which may make them more familiar with AI than older populations in other countries. Third, the scope of this study was restricted to evaluating accuracy, legal compliance, ethics, comprehensiveness, and information delivery; it did not explore other critical dimensions, such as patients’ psychological responses, trust, or the longitudinal impact on health outcomes. Moreover, the study attempted to minimize hallucinations—instances in which models generate incorrect or misleading information with high confidence—rather than addressing them directly. However, evaluation is essential before clinical application because hallucinations could substantially affect patient safety. Finally, the LLMs evaluated in this study, ChatGPT 4.0 and Claude 3, are continuously evolving, which means that their performance may vary with subsequent updates and require regular reassessment in clinical applications. Therefore, updated versions could be repeatedly assessed using the evaluation instrument developed in this study.

In Conclusion, This study showed that LLMs can convey complex content, such as stroke-related information, in an accessible and understandable manner for non-experts. Patients with stroke and individuals at risk of stroke favored educational texts generated by LLMs, indicating the potential role of LLMs in creating patient-tailored education materials in real clinical settings. However, several prerequisites must be addressed before such implementation. LLM-generated materials should be regularly reviewed by experts for output consistency and hallucination-related inaccuracies, and the evaluation instrument developed in this study could support that process. In addition, explicit ethical guidelines concerning privacy and commercial sponsorship should be established before implementation. If LLMs are implemented in clinical settings for patient education, they could help foster a more patient-friendly healthcare environment.