AI-driven Medical Care: Evaluation of Large Language Models in Generating Personalized Stroke Education Materials

Article information

Healthc Inform Res. 2026;32(2):179-189

Publication date (electronic) : 2026 April 30

doi : https://doi.org/10.4258/hir.2026.32.2.179

Surim Yoon^1,*, Woo-Keun Seo^1,2,*, Kyungseo Kim², Seongvin Ju², Hyun Kyung Kim¹, Hyung Jun Kim¹, Jong-Won Chung¹, Oh Young Bang¹, Gyeong-Moon Kim¹, Eun Young Lee³, Youngrak Choi⁴, Soyoung Yoo⁵

¹Department of Neurology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea

²Department of Digital Health, Samsung Advanced Institute for Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Korea

³Clinical Research Center, Asan Medical Center, Seoul, Korea

⁴Asan Institute for Life Science, Asan Medical Center, Seoul, Korea

⁵Department of Digital Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea

Corresponding Author: Soyoung Yoo, Department of Digital Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea. Tel: +82-2-3010-7161, E-mail: mesoyoung@gmail.com (https://orcid.org/0000-0002-2953-508X)

*These authors contributed equally to this work.

Received 2025 September 1; Revised 2026 April 17; Accepted 2026 April 26.

Abstract

I. Introduction

Large language models (LLMs) are artificial intelligence (AI) models trained through self-supervised learning to generate natural-language text. They can respond to unstructured content on specific subjects [1], and their applications are becoming increasingly versatile. LLMs can rapidly organize and present large volumes of information accumulated online, functioning as user-friendly tools for information synthesis. Over the years, researchers have sought to integrate AI into medicine because the field is broad and highly specialized, with extensive use of technical terminology. Accordingly, studies have assessed whether LLM-generated responses are accurate and presented in an easily understandable manner [2–5]. Several studies have demonstrated that LLMs can assist physicians with diagnoses, suggest possible treatment options, and translate discharge summaries into language that is easier for patients to understand [6–10].

Meanwhile, ChatGPT 4.0, the state-of-the-art model at that time, reportedly achieved passing grades on neurology board-style questions [11]. It also answered questions about multiple sclerosis with readability, satisfaction, and empathy scores comparable to those of a neurologist [12]. However, its decision-making performance was less accurate, showing only 54.3% agreement with neurologists in deciding whether to perform mechanical thrombectomy. Errors occurred in 8.8% of cases because of mathematical or logical mistakes and misinterpretation of clinical information [6]. Therefore, validation is needed before LLMs can be directly applied in medical practice, particularly with respect to the accuracy and safety of aggregated information, the appropriateness of clinical interpretation, readability, and conflicts of interest [13–16].

Managing cardiovascular risk factors has been shown to reduce mortality among patients with stroke [17]. Because stroke has various etiologies and treatment approaches differ according to cause, effective patient education requires individualized guidance based on each patient’s pathophysiology. This study aimed to evaluate the natural language processing capabilities of LLMs in providing personalized patient education and to assess their user-friendliness. To this end, we compared LLM-generated materials with materials written by a human neurologist in terms of accuracy, legality, ethics, comprehensiveness, and information delivery, using a newly developed evaluation questionnaire.

II. Methods

1. Study Design

An online survey was designed to ask participants to rate the feasibility of personalized patient education materials generated by LLMs. ChatGPT 4.0 from OpenAI and Claude 3 from Anthropic were used because they were state-of-the-art models that were easily accessible to the general public in Korea at the time of the study. The specific sizes, architectures, and training data of these models have not been officially disclosed; however, they were selected because of their ability to process Korean, which is less represented than English in many LLM training and evaluation contexts. Although their performance differs across functional domains, such as reasoning, coding, and mathematics, this study aimed to evaluate their ability to aggregate medical information and present it effectively to users.

The evaluation instrument was developed through a systematic process. First, the initial items were generated based on a literature review. Second, content validation was performed by an expert panel comprising three neurologists and two medical educators. Third, psychometric analyses, including internal consistency assessment (Cronbach’s α = 0.85) and test-retest reliability assessment, were conducted. Fourth, final refinement resulted in 10 questions across five domains: medical accuracy, legality, ethics, comprehensiveness, and information delivery.

A neurologist, OpenAI’s ChatGPT 4.0, and Anthropic’s Claude 3 Opus generated patient education materials for each scenario, resulting in six distinct sets of materials. These materials were rated using the final evaluation instrument on a 5-point Likert scale: 1, strongly disagree; 2, disagree; 3, neither; 4, agree; and 5, strongly agree (Table 1). Information about the authorship of each description was anonymized.

Table 1

Survey questionnaire

This cross-sectional survey study did not include any intervention. Personal information was not used in the analysis. The data and information provided to participants were based on fictional patient scenarios; therefore, a waiver of informed consent was obtained from the Institutional Review Board of Samsung Medical Center (waiver 2024-05-136).

2. Clinical Scenario Design

Two scenarios involving fictional patients with stroke were created because the medical information used in real-patient scenarios could identify patients, and disclosure of such information to study participants could violate personal privacy and raise ethical concerns. Clinical information included basic characteristics, National Institutes of Health Stroke Scale score, screening test results, suspected lesion area, and prescriptions or recommendations (Tables 2). The information was provided to the LLMs in two formats: sentences and lists (Supplementary Table S1). The scenarios were designed to represent distinct stroke etiologies and to determine whether LLMs could distinguish and explain different treatments according to the underlying cause. Specifically, the scenarios corresponded to different Trial of Org 10172 in Acute Stroke Treatment (TOAST) classifications: Scenario 1 involved cardioembolic stroke, and Scenario 2 involved stroke due to large-artery atherosclerosis.

Table 2

Survey scenarios

III. Results

Of the 136 participants contacted, 132 responded (response rate, 97.1%), whereas four non-medical personnel did not respond because they had difficulty understanding the survey questions. Fifty-two participants (39.4%) were female, and most participants were in their 40s (59%). A substantial proportion of participants agreed with the use of AI in healthcare (79.5%), and a slightly smaller proportion agreed with the use of LLMs in healthcare (72.7%). The characteristics of the study population are summarized in Table 3.

Table 3

Characteristics of the study population (n = 132)

1. Performance in Generating Patient-Specific Descriptions

ChatGPT 4.0 received the highest scores in both scenarios compared with the neurologist and Claude 3 (Figure 1). This preference for ChatGPT was particularly evident for accuracy, especially in the review of patient status (Q2). In Scenario 1, 64.08% of participants gave positive ratings for the medical accuracy of ChatGPT. The same proportion (64.08%) positively rated Claude, while only 53.73% gave positive ratings for the neurologist. For ChatGPT, positive ratings were given by 54.96% of participants for legality (Claude, 45.25%; neurologist, 40.68%), 53.85% for ethics (Claude, 40.31%; neurologist, 37.98%), 77.27% for comprehensiveness (Claude and neurologist, 59.85% each), and 76.52% for information delivery (Claude, 55.73%; neurologist, 54.55%). A similar trend was observed in Scenario 2.

Figure 1

Percentages of overall survey responses. (A), (B), and (C) are the scores of each entity in Scenario 1 given by survey participants. (D), (E), and (F) are the scores of each entity in Scenario 2. Q1–Q5 are about medical accuracy, Q6 and Q7 are about legality, Q8 is about ethics, Q9 is about comprehensiveness, Q10 is about information delivery. In Scenario 1, ChatGPT 4.0 (B) scored the most “agree” and “strongly agree” across all perspectives, followed by Claude 3 (C), then the neurologist (A). A similar trend was demonstrated in Scenario 2 (E>F>D).

In Scenario 1, ChatGPT received the highest scores across all domains (Table 4). For accuracy, the ranking of mean scores was ChatGPT, Claude, and the neurologist (Q1, η² = 0.140, p < 0.001). The difference was most prominent for summarizing clinical information (Q2; η² = 0.175, p < 0.001). ChatGPT performed markedly better in comprehensiveness (p < 0.001) and information delivery (p < 0.001) than Claude and the neurologist.

Table 4

Differences in comprehensiveness and information delivery in agreeing to utilize LLM in healthcare

In Scenario 2, ChatGPT achieved the highest scores in four of the five domains, except for legality (Table 4). The same hierarchy was observed for accuracy (Q1, η² = 0.077, p < 0.001), summarization of clinical information (Q2, η² = 0.194, p < 0.001), comprehensiveness (p < 0.001), and information delivery (p < 0.001). The clearest difference was observed for information delivery (Q10, η² = 0.214).

In Scenario 2, significant differences were observed between the neurologist and Claude in three domains: accuracy, comprehensiveness, and information delivery. In Scenario 2, Claude received higher scores than the neurologist. No significant differences were observed among the subgroups, including healthcare professionals, patients/caregivers, researchers, and the general public (Supplementary Table S3).

2. Differences in Comprehensiveness and Information Delivery according to Agreement with LLM Use in Healthcare

Participants were categorized into two groups: those who clearly agreed with the use of LLMs in healthcare, defined as those who responded “strongly agree” or “agree” to the question, “Do you agree with the use of LLMs in healthcare?”; and those who did not clearly agree, defined as those who responded “strongly disagree,” “disagree,” or “neutral” to the same question. Overall, participants who agreed with the use of LLMs in healthcare gave higher scores than those who did not clearly agree (Table 5). ChatGPT consistently received higher scores from participants who agreed with LLM use in healthcare in both scenarios for comprehensiveness (Scenario 1, p = 0.005; Scenario 2, p = 0.007) and information delivery (Scenario 1, p = 0.003; Scenario 2, p = 0.026).

Table 5

Difference in performance among the three entities (n = 132)

IV. Discussion

This study indicates that both ChatGPT and Claude can generate patient education materials comparable to those written by a neurologist. ChatGPT received high ratings in both scenarios, particularly for accuracy and comprehensiveness. No significant differences in perceptions of LLM performance were observed among healthcare professionals, patients/caregivers, research workers, and the general public. Furthermore, participants who were willing to accept LLMs in healthcare rated the explanations generated by ChatGPT and Claude more favorably than those who were not.

To our knowledge, this is among the first studies to assess the utility of LLMs for individualized education in clinical settings by comparing LLM-generated materials with specialist-authored materials based on realistic clinical scenarios. Previous studies have mainly assessed the accuracy and comprehensiveness of responses to frequently asked questions about various diseases [2,12,18,19]. However, these assessments focused primarily on whether LLMs could gather fundamental knowledge that is easily accessible through standard online searches. In contrast, this study demonstrated the capability of LLMs to organize specialized knowledge, such as stroke-related information, and convey it clearly and comprehensibly. The LLMs interpreted the provided clinical information and synthesized it into an organized format for users, offering guidance on health management strategies to prevent recurrence and important precautions for daily life (Supplementary Table S2). Most previous studies evaluated readability and comprehension solely through healthcare professionals [14,19] or scoring tools [12,18]. Because comprehensibility and readability ultimately depend on how patients perceive the information, those studies had limitations in evaluating whether LLMs could genuinely convey information in a manner comprehensible to non-experts. By involving both patients with stroke and members of the general public, the present study was able to evaluate this capability more directly. This study also provides insight into the potential applicability of LLMs in real-world settings.

The terminology used by LLMs was assessed across five domains: accuracy, legality, ethics, comprehensiveness, and information delivery. Participants, including both patients and healthcare professionals, indicated that explanations written by LLMs were easier to understand than those written by the neurologist. This may be because LLMs organized the content with subheadings and list formatting, enabling readers to grasp complex information more readily. However, subtle content differences were observed between the explanations provided by the neurologist and those generated by the LLMs. Unlike the neurologist, the LLMs did not explain the bleeding risks of medications that patients needed to take. In addition, although the neurologist explained the rationale for medication selection based on stroke etiology, the LLMs focused primarily on the efficacy of taking the medications without providing this contextual explanation. This finding is consistent with previous research. A study conducted by Yeo et al. indicated that LLMs may identify diagnostic algorithms but omit details such as cutoff values for diagnostic tests and may fail to provide appropriate information relevant to patients with liver cirrhosis and hepatocellular carcinoma [4]. Similarly, the LLMs successfully obtained information on certain medications used for stroke treatment but failed to explain when those medications should be used. Although LLMs may find it challenging to truly understand the mechanisms of diagnosis and treatment, the survey results indicated that the generated texts were medically appropriate compared with neurologist-authored materials.

LLMs received lower legality and ethics scores than accuracy and comprehensiveness scores. ChatGPT received an average score above 4 for accuracy and comprehensiveness but scored in the mid-3-point range for legality and ethics, with most participants selecting “neither.” Claude showed no significant difference in these domains compared with the neurologist. This may be because most participants were uncertain about using LLMs in medical settings and lacked sufficient knowledge about AI, as 52.3% responded “neither” when asked whether they were familiar with AI.

No significant differences in ratings were observed among the subgroups (Supplemental Table 3). This suggests that perceptions of LLM performance were consistent regardless of occupational background. Instead, score differences were more apparent according to whether participants agreed with the use of LLMs. Although other factors, including technology education status, may have influenced individual attitudes toward LLMs [20], the present findings suggest that positive changes in patients’ views of LLMs may promote the seamless integration of LLMs into healthcare.

Although LLMs were shown to handle complex information, such as medical information, several issues must be addressed before they can be applied in medical settings. First, consistency needs to be improved. Pugh et al. showed that the consistency of LLMs can be influenced by the model used and the temperature parameter when estimating ratings of speech samples for coherence, content, and tangentiality among patients diagnosed with schizophrenia [21]. Second, LLMs are vulnerable to hallucinations, which could mislead users by presenting incorrect information with confidence. In research using LLMs for outpatient rehabilitation, Google Bard confidently referenced websites that did not exist [14]. Thus, materials created by LLMs must be reviewed by specialists before use to prevent patient harm. In addition, implementing LLMs to process medical data on overseas servers presents substantial regulatory challenges, particularly regarding compliance with cross-border data transfer requirements. Even temporary data transmission requires adherence to stringent legal requirements, including explicit consent from data subjects, as mandated by domestic personal information protection laws and the European Union General Data Protection Regulation. These regulatory frameworks remain applicable regardless of temporary storage provisions or automatic deletion protocols. Beyond immediate compliance concerns, the use of LLMs in healthcare raises additional privacy considerations, including the risk of unintentional retention of personal information in online spaces and the need for clear guidelines on source attribution and copyright compliance in LLM outputs [22]. Given the highly sensitive nature of medical information and these multifaceted regulatory challenges, implementing isolated models may be a viable technical solution to ensure both data protection and regulatory compliance [20].

Considering that this was an online survey, most individuals who actively participated may have been familiar with digital environments and potentially more favorable toward LLMs. However, as familiarity with LLMs increases over time, similar results may be expected among future patients with stroke. Several limitations should be considered when interpreting the findings of this study. First, the use of fictional patient scenarios, although valuable for controlled assessment, may only partially capture the complexity and unpredictability of real patient cases. This limits the generalizability of the results to actual clinical settings. Second, because the study participants were Korean, the LLMs may have received relatively high scores. In Korea, where public Wi-Fi is widely available, even older adults, including patients with stroke, tend to be relatively digitally connected, which may make them more familiar with AI than older populations in other countries. Third, the scope of this study was restricted to evaluating accuracy, legal compliance, ethics, comprehensiveness, and information delivery; it did not explore other critical dimensions, such as patients’ psychological responses, trust, or the longitudinal impact on health outcomes. Moreover, the study attempted to minimize hallucinations—instances in which models generate incorrect or misleading information with high confidence—rather than addressing them directly. However, evaluation is essential before clinical application because hallucinations could substantially affect patient safety. Finally, the LLMs evaluated in this study, ChatGPT 4.0 and Claude 3, are continuously evolving, which means that their performance may vary with subsequent updates and require regular reassessment in clinical applications. Therefore, updated versions could be repeatedly assessed using the evaluation instrument developed in this study.

In Conclusion, This study showed that LLMs can convey complex content, such as stroke-related information, in an accessible and understandable manner for non-experts. Patients with stroke and individuals at risk of stroke favored educational texts generated by LLMs, indicating the potential role of LLMs in creating patient-tailored education materials in real clinical settings. However, several prerequisites must be addressed before such implementation. LLM-generated materials should be regularly reviewed by experts for output consistency and hallucination-related inaccuracies, and the evaluation instrument developed in this study could support that process. In addition, explicit ethical guidelines concerning privacy and commercial sponsorship should be established before implementation. If LLMs are implemented in clinical settings for patient education, they could help foster a more patient-friendly healthcare environment.

Supplementary Materials

Article information Continued