A large language models-assisted and expert-corrected workflow for preoperative anesthesia assessment drafts: A single-centre exploratory feasibility study

Abstract

Background

Large language models (LLMs) may help organize clinical information, but their use in perioperative settings requires careful evaluation because errors may have immediate safety implications. This study aimed to describe the feasibility and perceived usefulness of a single-centre, expert-corrected LLM workflow for preparing preoperative anesthesia assessment drafts for complex consultation cases. Secondary aims were to describe error patterns identified by anesthesiologists and to explore residents’ perceptions after reviewing expert-corrected materials.

Methods

We retrospectively selected 15 complex preoperative anesthesia consultation cases from The First Affiliated Hospital, Zhejiang University School of Medicine. Complex cases were defined as cases referred for specialized preoperative anesthesia consultation because of multiple comorbidities, clinically relevant organ dysfunction, implanted devices, major cardiopulmonary or vascular disease, or other features requiring individualized anesthetic assessment. A large language model-based artificial intelligence system, DeepSeek-R1, was used to generate structured draft assessments from de-identified case information and a standardized prompt. The original LLM-generated drafts were independently reviewed by three experienced anesthesiologists for information completeness, scientific plausibility, and focus on key risk factors. All resident-facing documents were then corrected by senior anesthesiologists before being distributed to first-year anesthesia residents, who completed a questionnaire on perceived usefulness, guidance value, educational support, and error recognition. No comparator group, blinding, or objective performance outcome was included.

Results

The LLM-generated drafts were structurally complete but contained clinically relevant errors, mainly in risk assessment and anesthetic plan formulation. Thirty-two error instances were identified across the 15 cases, including errors related to proposed plans, American Society of Anesthesiologists (ASA) physical status classification, and abnormal test interpretation. After expert correction, residents reported favorable perceptions of the materials, but their self-reported ability to identify expert-annotated errors remained limited. These findings should be interpreted as perception-based observations rather than evidence of educational effectiveness.

Conclusions

In this single-centre exploratory study, an LLM-assisted workflow was feasible for producing preliminary preoperative anesthesia assessment drafts, but expert correction was essential before resident-facing use. The findings do not establish standalone clinical or educational effectiveness of the LLM. Instead, they highlight the gap between structural completeness and clinical reliability, and suggest that any use of LLM-generated anesthesia assessment drafts should remain adjunctive, supervised, and explicitly framed to reduce automation bias.

Resumen

Introducción

Los modelos de lenguaje de gran tamaño pueden ayudar a organizar información clínica, pero su uso en el contexto perioperatorio exige una evaluación prudente, ya que los errores pueden tener implicaciones directas para la seguridad del paciente. Este estudio tuvo como objetivo describir la factibilidad y la utilidad percibida de un flujo de trabajo unicéntrico, asistido por un modelo de lenguaje y corregido por expertos, para elaborar borradores de evaluación anestésica preoperatoria en casos complejos de interconsulta. Los objetivos secundarios fueron describir los patrones de error identificados por anestesiólogos y explorar las percepciones de los residentes tras revisar materiales corregidos por expertos.

Métodos

Se seleccionaron retrospectivamente 15 casos complejos de interconsulta anestésica preoperatoria del First Affiliated Hospital, Zhejiang University School of Medicine. Se consideraron casos complejos aquellos remitidos para valoración anestésica especializada por presentar múltiples comorbilidades, disfunción orgánica clínicamente relevante, dispositivos implantados, enfermedad cardiopulmonar o vascular importante, u otras características que requerían una evaluación anestésica individualizada. Se utilizó DeepSeek-R1, un sistema de inteligencia artificial basado en un modelo de lenguaje de gran tamaño, para generar borradores estructurados a partir de información clínica desidentificada y una indicación estandarizada. Los borradores originales generados por el modelo fueron revisados de forma independiente por tres anestesiólogos con experiencia en cuanto a integridad de la información, plausibilidad científica y enfoque en factores de riesgo clave. Posteriormente, todos los documentos destinados a los residentes fueron corregidos por anestesiólogos sénior antes de su distribución a residentes de primer año de anestesiología, quienes completaron un cuestionario sobre utilidad percibida, valor orientador, apoyo educativo e identificación de errores. No se incluyeron grupo comparador, enmascaramiento ni resultados objetivos de desempeño.

Resultados

Los borradores generados por el modelo fueron estructuralmente completos, pero contenían errores clínicamente relevantes, principalmente en la evaluación del riesgo y en la formulación del plan anestésico. Se identificaron 32 errores en los 15 casos, incluidos errores relacionados con los planes propuestos, la clasificación ASA y la interpretación de pruebas anormales. Tras la corrección experta, los residentes comunicaron percepciones favorables de los materiales, aunque su capacidad autoinformada para identificar los errores señalados por los expertos fue limitada. Estos resultados deben interpretarse como observaciones basadas en percepciones, no como evidencia de eficacia educativa.

Conclusiones

En este estudio exploratorio unicéntrico, el flujo de trabajo asistido por un modelo de lenguaje fue factible para producir borradores preliminares de evaluación anestésica preoperatoria, pero la corrección por expertos fue esencial antes de su uso con residentes. Los hallazgos no demuestran eficacia clínica ni educativa independiente del modelo. Más bien, ponen de manifiesto la diferencia entre completitud estructural y fiabilidad clínica, y sugieren que el uso de borradores anestésicos generados por modelos de lenguaje debe mantenerse como una herramienta complementaria, supervisada y enmarcada explícitamente para reducir el sesgo de automatización.

Introduction

Preoperative anesthesia assessment is a critical component of perioperative medical management. This core preoperative procedure is essential for ensuring surgical safety, optimizing anesthesia protocols, and reducing perioperative complications. Furthermore, it helps patients and their families prepare adequately for surgery.¹ The “Basic Standards for Medical Record Documentation,” which was issued by the National Health Commission of the People's Republic of China, stipulates that preanesthetic risk assessments must be conducted and documented on the preoperative anesthesia assessment form before anesthesia is administered.² Traditional preoperative anesthesia assessment forms only cover the primary diagnosis, the name of the surgical procedure, major examinations, key laboratory tests, and the planned anesthesia technique. To complete these forms, anesthesiologists must review patient records, summarize medical histories, assess organ function, identify anesthetic risks, and formulate anesthesia plans within a limited period before completing these forms. They then conduct preoperative visits with patients and obtain signed informed consent, a process that is highly dependent on the anesthesiologist's clinical experience and knowledge base.³ Previous reports have estimated the density of anesthesiologists in China at approximately 4.8 per 100,000 population, close to the minimum World Federation of Societies of Anesthesiologists (WFSA) benchmark of 5 per 100,000 but still unevenly distributed across regions.⁴ Consequently, China has experienced a longstanding shortage of anesthesiologists, making training young anesthesiologists critically important.⁵

In China, anesthesiology residency training is conducted through a standardized residency training (SRT) program. In this study, junior anesthesiologists refer to physicians in the first year of the anesthesiology SRT program, whereas residents refer to physicians enrolled in the anesthesiology SRT program regardless of training year. Residents often shoulder heavy clinical workloads and are responsible for conducting preliminary preoperative anesthesia visits under supervision.⁶ However, junior anesthesiologists frequently lack the requisite skills to assess anesthesia for patients with complex medical conditions. In this context, helping residents organize complex preoperative information and recognize anesthesia-related risks is a practical challenge. A competency deemed indispensable for anesthesiologists in the early stages of their practice is the ability to perform comprehensive preoperative assessments, especially for cases that are inherently complex. Interest in the potential applications of artificial intelligence (AI) in medicine is increasing. The complexity and dynamic nature of perioperative patients, particularly the multitude of variables and constant changes in patient condition, make medicine one of the most promising fields for AI.⁷ Artificial intelligence applications have been reported to support junior physicians in selected diagnostic tasks.^8,9

LLMs have been explored for tasks such as information extraction, clinical text structuring, and medical education support; however, their role in high-stakes perioperative settings remains uncertain and requires cautious evaluation.¹⁰ Therefore, the primary objective of this exploratory study was to describe the feasibility of using an LLM-assisted, expert-corrected workflow to prepare structured preoperative anesthesia assessment drafts for complex consultation cases. The secondary objectives were to describe error patterns identified by experienced anesthesiologists and to explore first-year anesthesia residents’ perceptions after reviewing expert-corrected materials.

Materials and methods

Ethical approval for this study (IIT20250752B) was provided by the Ethical Committee of The First Affiliated Hospital, Zhejiang University School of Medicine on June 24, 2025. The institutional review board waived the requirement for informed consent.

Data collection and preparation

This single-centre retrospective exploratory feasibility study was conducted in the Department of Anesthesiology, The First Affiliated Hospital, Zhejiang University School of Medicine. The study consisted of eight steps:

• 1.

  Case pool of 113 consultation requests.

• 2.

  Interval sampling and eligibility screening.

• 3.

  Selection of 15 de-identified complex cases.

• 4.

  DeepSeek-R1 draft generation.

• 5.

  Independent expert evaluation.

• 6.

  Consensus correction with revision marks.

• 7.

  Distribution to residents.

• 8.

  Questionnaire collection and analysis.

Considering the pronounced deficiency among residents in proficiency regarding preoperative anesthesia consultations, particularly in the context of complex cases, this study focused on such cases to determine the viability and perceived educational support of the “LLM-assisted generation + expert refinement” workflow. Within the framework of Chinese hospitals, when inpatients necessitate specialized care from other departments, a formal consultation request is needed, with the consultation findings meticulously documented on the consultation request form. To identify patients with complex anesthesia requirements and multiple comorbidities, an anesthesiologist with more than 10 years of experience extracted 113 preoperative consultation requests received by the Anesthesiology Department between February 1 2025 and May 1 2025 from the electronic medical record system between May 20 and May 27. For this study, complex cases were defined as preoperative anesthesia consultation cases requiring specialist anesthesiology input because of multiple comorbidities, clinically relevant organ dysfunction, implanted cardiac devices, major cardiopulmonary or vascular disease, dialysis dependence, active pulmonary disease, or uncertainty requiring individualized perioperative risk assessment. These cases were sorted according to the consultation request date. With an interval of 113/15 ≈ 8, Excel's RANDBETWEEN() function is used to generate a random integer starting point between 1 and 8. Starting from this point, one patient was selected for every 8 cases. Cases were skipped if any of the following conditions applied: missing medical record information and missing critical test results (e.g., complete blood count, coagulation profile, electrocardiogram, etc.). This process continued until 15 patients were selected.

The admission records, all preoperative examination results, and all preoperative laboratory test results for the above patients were extracted. Medical records were merged with examination and laboratory test results, resulting in 15 case reports. Patient names, hospital ID numbers, home addresses, phone numbers, and examination/laboratory test serial numbers were removed before the data were categorized and stored (Table 1). Expert raters and residents were not blinded to the LLM-assisted nature of the materials.

Table 1.

Case key information.

Case ID	Primary diagnosis	Other diagnoses	Proposed surgery
1	Urethral tumor	Urethral stricture (postoperative)Personal history of malignant urethral tumorVaginal tumorType 2 diabetesChronic obstructive pulmonary diseaseHypertension (grade 3, very high risk)Coronary atherosclerotic heart diseaseEmphysema	Partial urethral resection
2	Patella fracture (right side)	Coronary atherosclerosis (coronary atherosclerosis)Coronary artery stenosis (left anterior descending artery)Deep vein thrombosis of the lower extremities	Patellar reduction with open internal fixation
3	Gallbladder stones	AsthmaHypertensionChronic obstructive pulmonary disease	Cholecystectomy
4	Thoracic aortic pseudoaneurysm	Cirrhosis (liver cirrhosis)Hepatic encephalopathyCryptococcal encephalitisAtrial flutter	Thoracic aortic aneurysm peritoneal stent placement
5	Gallbladder stones with cholecystitis	Coronary heart diseaseHyperlipidemiaDiabetesRight kidney stones	Cholecystectomy
6	Renal tumor (right side)	Bladder massBenign prostatic hyperplasiaRenal cystPulmonary nodules (bilateral)Elevated D-dimerChronic bronchitis	Partial nephrectomy
7	Inguinal hernia (left side)	Colonic malignant tumorBenign prostatic hyperplasiaPulmonary noduleBronchitisLeft anterior mediastinal nodule	Tension-free repair of left inguinal hernia
8	Breast nodule (left)	Asthma	Breast nodule excision
9	Cerebral infarction (syncope, cause to be determined)	Middle cerebral artery stenosis (left M1 segment, severe)Essential hypertensionType 2 diabetesCataract	Percutaneous middle cerebral artery stenting
10	Malignant breast tumor (right side)	CardiomyopathyImplantable cardioverter defibrillator (ICD) post-implantationDiabetes	Mammary mass excision
11	Chronic kidney disease stage 5	Hemodialysis status	Arterial-venous fistula thrombectomy of artificial vessel post-procedure
12	Renal mass lesion (left side)	Thyroid nodule (TIRADS 4a)Chronic hepatitis BDiabetesHypertension	Laparoscopic partial nephrectomy
13	Ovarian cyst (left side)	Status after hysterectomy (post-subtotal hysterectomy)Post-mitral valve replacementEssential hypertensionFatty liverGallbladder polypsGallbladder adenomyosis (considered)Splenic hemangioma (considered)History of drug allergy (amoxicillin)	Bilateral salpingo-oophorectomy (bilateral)
14	Carotid artery stenosis (right side)	Coronary atherosclerotic heart diseaseSleep apnea syndrome	Carotid artery stenting
15	Otitis (left ear)	Perforated eardrum (left ear)Mixed hearing loss (left ear)Sensorineural hearing loss (right ear)Tracheal diverticulumChronic sinusitisNonspecific STT changes	Tympanoplasty

Design of the standardized pre-anesthesia assessment form

To ensure that the generated assessment form better serves junior anesthesiologists, we considered highlighting key points and challenges during the patient's anesthesia process. Therefore, on the basis of the standard preoperative interview form, we generated a customized LLM preoperative interview form for each patient. Using standardized questions, the LLM highlighted the primary concerns, precautions, and additional information required during the preoperative interview for each specific case. A single standardized prompt was used for all cases to reduce inter-user variability.

Standard prompt: An experienced anesthesiologist conducts a preoperative assessment for the de-identified case information below and generates an anesthesia preoperative assessment checklist according to the following requirements:

• 1.

  Summary of medical history by system (cardiovascular, respiratory, digestive, neurological, hematological, urological, endocrine, psychiatric, obstetric, smoking/alcohol/drug dependence, allergies, surgical/anesthetic history, family history, and special medication history). Note any abnormal test results and determine the anesthesia classification and risk levels.

• 2.

  Identify the key perioperative monitoring priorities for this patient, and then provide a detailed anesthesia plan.

• 3.

  Identify any additional information required during the preoperative anesthesia assessment.

The medical records and test results are as follows:

LLM selection

DeepSeek-R1 was used as the large language model for draft generation. The model was accessed via its publicly available web interface (https://chat.deepseek.com/) at the time of the study. No additional customization or fine-tuning was performed.

LLM-based anesthesia visit note generation

We used DeepSeek-R1 (version 0528) to generate LLM-generated draft assessments. On July 1, 2025, we entered the standard questions and 15 case report datasets into the DeepSeek web interface (https://chat.deepseek.com/). We enabled deep thinking (R1) and an online search on the DeepSeek webpage. After each response, the dialogue restarted. The responses were saved as separate documents and merged with the corresponding case report content, and the document names were coded. Each case was generated once using a single-session, one-shot approach. Generation parameters were not configurable via the web interface, and output stability or variance across repeated runs was not assessed. Before input into the web interface, all direct patient identifiers were removed, including names, hospital identification numbers, addresses, telephone numbers, and examination serial numbers. No images or original identifying documents were uploaded. The investigators did not have access to a study-specific technical mechanism to verify whether the web-based system retained or used inputs for model improvement. This was therefore treated as a data-governance and reproducibility limitation.

Comparative analysis methodology

We distributed the above documents to the anesthesia teaching team, which comprised three anesthesiologists, each of whom had more than five years of independent anesthesia and teaching experience. None of the anesthesiologists specialized in a subspecialty of anesthesia. Each anesthesiologist independently evaluated the completeness and scientific plausibility of the LLM-generated drafts prior to expert correction. We summarized these expert ratings across three dimensions (Completeness of Assessment Forms Information, Scientific Rationality of Assessment Forms, and Focus of Assessment Forms) to describe common strengths and error patterns of the drafts and the extent of expert correction required. Errors in the assessment forms have been annotated and corrected (Table 2).

Table 2.

Expert scoring table.

Expert scoring table
Evaluation dimensions	Scoring criteria	Score	Specific errors/deficiencies (please correct or note in this column)
I. Completeness of Assessment Forms Information (35 points)
1. Abnormal test/examination value identification rate (15 points)	Correctly identify all abnormal indicators (including critical values). Deduct 1 point for each missed or misidentified item (minimum score: 0 points)
2. Coverage of surgical anesthesia focus areas (20 points)	15 points for covering 14 systems in the standard questions; 10 points for covering all risk points. Deduct 1 point for each omitted system, 2 points for each omitted core point (minimum 0 points)
II. Scientific Rationality of Assessment Forms (40 points)
1. Accuracy of anesthesia risk assessment (20 points)	ASA classification aligned with clinical assessment: 10 points; 2 points deducted per grade discrepancy (minimum 0 points); accurate postoperative outcome assessment: 10 points; 5 points deducted per incorrect item (minimum 0 points)
2. Alignment of proposed plan with clinical practice and guidelines (20 points)	Anesthesia method, drug selection, monitoring measures, etc., conforming to clinical practice and guidelines: 20 points; 2 points deducted for each nonconforming item (minimum 0 points)
III. Focus of Assessment Forms (25 points)
Accuracy of core risk factor annotation during bedside consultation (25 points)	10 points for identifying all core risks; 2 points deducted for each omission. 15 points for identifying medical history/medication requiring further inquiry; 3 points deducted for each omitted key question (minimum 0 points).

Expert revision of drafts

The teaching secretary in the anesthesiology department scored the assessment forms and compiled the results. Consensus revisions were highlighted in the original forms, along with subsequent corrections that did not cover the original errors. Any disagreements were consolidated for discussion by the internal teaching team, and the forms were modified according to the final consensus.

Questionnaire design and distribution

Residents received the de-identified original case information and the corresponding expert-corrected LLM draft with revision marks. They did not receive uncorrected LLM drafts for independent clinical or educational use. In this study, a Likert-scale questionnaire was used to evaluate junior residents’ understanding of and attitudes toward preoperative anesthesia assessment forms. Nine questions (including two reverse-scored items) covered five domains: usage intent (Q7 and Q9), risk alerts (Q1 and Q4), guidance value (Q2 and Q5), perceived educational support (Q3 and Q8), and error identification (Q6). Each question had five response options: 1. strongly disagree, 2. disagree, 3. undecided/neutral, 4. agree, and 5. strongly agree (Table 3). The assessment questionnaire was included in each expert-revised material. Fifteen original medical records, 15 modified individualized assessment forms, and 15 assessment questionnaires were distributed to 17 first-year standardized training residents in the department of anesthesiology. The questionnaires were collected 24 h later. To facilitate comparison with other scoring systems, the total score for each dimension was linearly converted to a 10-point scale. The average score for each item within that dimension was first calculated and then multiplied by 2.

Table 3.

Assessment scale.

Q1: I believe this assessment form clearly identifies the patient's primary anesthesia risk factors.

Q2: Based on this assessment form, I can clearly identify the specific preanesthetic preparations required for this patient.

Q3: Reading this form helped me better understand key points in anesthetic management for such cases.

Q4: This form highlights risk factors or considerations that I might have overlooked.

Q5: I believe the anesthesia recommendations provided in this assessment form are inappropriate.

Q6: I can identify every piece of incorrect information marked by the senior anesthesiologist on this preoperative anesthesia assessment form.

Q7: This preoperative anesthesia assessment form has made me less confident in providing anesthesia to this patient.

Q8: Overall, I believe this LLM-generated form provides valuable guidance for junior anesthesiologists like myself.

Q9: I would be willing to use such LLM-generated drafts in clinical practice under the guidance of senior physicians.

Statistical methods

The expert scoring sheet scores were analyzed using the intraclass correlation coefficient (ICC) in SPSS 26.0 (SPSS Inc., Chicago, IL, USA) to assess scoring consistency. A two-way mixed model with a 95% confidence interval and a test value of zero was employed. The LLM-generated visit form scores are represented by their mean values. After addressing the reverse-coded items in the questionnaire results, the Cronbach's alpha coefficient was calculated to assess the reliability of the questionnaire. The questionnaire scores are presented as the mean ± standard deviation.

Results

Expert evaluation scores

The inter-rater reliability among the three anesthesiologists was good, with an ICC of 0.82, indicating acceptable consistency in the expert evaluation of the original LLM-generated drafts. As shown in Table 4, the drafts scored highly for information completeness and identification of key risk factors, but scored lower for alignment of the proposed anesthetic plan with clinical practice and guidelines. This pattern suggests that the drafts were more reliable for structuring information than for making clinically sensitive anesthesia recommendations.

Table 4.

Scoring results.

Evaluation dimension	Subitem	Total score	Average score	Percentage score
Completeness of assessment forms information		35	33.4	95.4
	Abnormal test/examination value identification rate	15	14.2	94.7
	Coverage of surgical anesthesia focus areas	20	19.2	96.0
Scientific and rationality of assessment forms		40	32.4	81.0
	Accuracy of anesthesia risk assessment	20	17.6	88.0
	Alignment of proposed plan with clinical practice and guidelines	20	14.8	74.0
Focus of assessment forms	Accuracy of core risk factor annotation during bedside consultation	25	24.4	97.6
Overall score		100	90.2	90.2

Error analysis from the expert review

The expert anesthesiologists identified and annotated errors and deficiencies in the LLM-generated drafts across the evaluated dimensions. A total of 32 distinct error instances were documented across the 15 cases. Of these, 18 (56.3%) pertained to alignment of the proposed plan with clinical practice and guidelines and involved primarily inappropriate anesthetic technique or medication suggestions. Eight errors (25%) were related to accuracy of anesthesia risk assessment and included incorrect American Society of Anesthesiologists (ASA) physical status classifications. Six errors (18.7%) were noted in abnormal test/examination value identification and involved missed or misinterpreted values. Before submitting the forms to the residents, the teaching team corrected all the identified errors while retaining revision marks and the original mistakes.

Resident questionnaire results

Out of 255 distributed questionnaires, 202 were returned; thus, the response rate was 79%. The Cronbach's alpha coefficient for the questionnaire was 0.85.

The mean scores (on a 5-point Likert scale) for each question across all the cases are presented in Table 5. Following reverse-coding of items Q5 and Q7 and linear transformation to a 10-point scale for each domain, the mean domain scores were as follows (Table 6): intention to use, 8.40 ± 0.96; risk indication, 8.49 ± 1.21; guidance value, 8.27 ± 0.96; perceived educational support, 8.62 ± 1.18; and error recognition, 3.19 ± 0.79.

Table 5.

Mean scores for each questionnaire.

	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Overall Average
Case 1	4.47	4.41	4.29	4.47	4.18	3.18	3.88	4.35	4.41	4.18
Case 2	4.50	4.36	4.29	4.43	4.07	3.14	3.79	4.36	4.43	4.15
Case 3	4.38	4.15	4.23	4.08	4.08	3.31	4.15	4.54	4.31	4.14
Case 4	4.31	4.15	4.23	4.31	4.08	3.08	4.08	4.54	4.38	4.13
Case 5	4.23	4.31	4.23	4.15	4.00	3.08	3.85	4.38	4.46	4.08
Case 6	4.29	4.14	4.36	4.36	4.00	3.07	4.07	4.43	4.50	4.13
Case 7	4.36	4.21	4.29	4.00	3.93	3.29	4.14	4.29	4.50	4.11
Case 8	4.38	4.31	4.31	4.15	3.92	3.54	4.15	4.38	4.46	4.18
Case 9	4.31	4.46	4.23	4.00	3.92	3.08	3.92	4.38	4.46	4.09
Case 10	4.23	4.31	4.08	4.08	4.00	3.31	4.15	4.31	4.23	4.08
Case 11	4.15	4.23	4.46	4.08	4.08	3.08	3.92	4.23	4.38	4.07
Case 12	4.14	4.14	4.36	4.29	4.07	3.36	3.93	4.36	4.57	4.13
Case 13	4.33	4.00	4.08	4.25	4.08	3.25	4.17	4.33	4.25	4.08
Case 14	4.23	4.31	4.23	3.85	3.92	2.85	4.00	4.38	4.38	4.02
Case 15	4.31	4.15	4.15	4.08	4.00	3.23	3.77	4.31	4.31	4.03

Table 6.

Overall classification description.

	Minimum	Maximum	Mean ± standard deviation
Intention to use	5.00	10.00	8.40 ± 0.96
Risk indication	4.00	10.00	8.49 ± 1.21
Guidance value	6.00	10.00	8.27 ± 0.96
Perceived educational support	3.00	10.00	8.62 ± 1.18
Student error recognition	1.00	5.00	3.19 ± 0.79

Discussion

This study presents an exploratory examination of an LLM-assisted, expert-supervised workflow for generating structured preoperative anesthesia assessment materials in a real-world educational context. Importantly, this work does not aim to evaluate the independent clinical performance or educational effectiveness of a large language model. Rather, it seeks to describe the feasibility, perceived characteristics, and potential risks associated with introducing LLM-assisted materials into anesthesia training under conditions of explicit human oversight.

One important observation from this small series is that the LLM-generated drafts looked organized but were not consistently reliable for clinical judgment. The drafts usually summarized medical history and abnormal test results in a readable format. However, expert review found errors in ASA physical status classification, risk interpretation, and anesthetic plan formulation. This distinction is important: a document may appear complete and clinically fluent while still containing recommendations that require correction by an experienced anesthesiologist. This observation aligns with prior literature indicating that LLM performance often degrades in nuanced, real-world clinical scenarios requiring integrated judgment rather than pattern recognition alone. For example, in Case 1, a patient with mild COPD and moderate coronary artery stenosis was incorrectly classified as ASA IV, which was inconsistent with the clinical reality. In our observed outputs, the model did not provide reliable individualized numerical adjustments (e.g., anesthetic dosing or ventilator parameters), reinforcing the need for expert review for any clinically sensitive recommendations. For example, in Case 6, which involved a patient with pulmonary issues, showed that the LLM recommended a high positive end-expiratory pressure (PEEP), which could potentially cause harm in a patient with pulmonary bullae. Residents reported that the revised materials increased their perceived confidence under supervision; however, this reflects subjective perception rather than objectively measured performance. Chung et al. investigated the ability of the general-purpose LLM GPT-4 Turbo to predict perioperative risks and prognoses. Their findings suggest that LLMs may support certain perioperative classification and alerting tasks in specific settings, which aligns with our observation that structured drafting may be feasible only under expert oversight.¹¹ Previous studies have evaluated ChatGPT-4s ability to formulate anesthesia plans. These results indicate that ChatGPT-4 is currently unsuitable for assisting with preoperative anesthesia planning.¹² Çelik et al. compared Google's Gemini AI application with anesthesiologists in selecting anesthesia methods for 72 orthopedic surgery patients. They found that the AI and anesthesiologists agreed on the anesthesia method only 68.5% of the time.¹³

The necessity for extensive expert correction represents a critical feature – not a shortcoming – of the workflow described. The relatively high expert scores should therefore not be interpreted as evidence that the LLM could produce safe preoperative anesthesia assessments on its own. In our scoring system, information completeness and risk-factor annotation contributed substantially to the total score. These domains reflected the model's ability to structure clinical information. By contrast, the lower score for alignment with clinical practice and guidelines, together with the 32 expert-identified errors, shows that clinically sensitive recommendations remained a weak point. This is especially relevant in anesthesia, where an apparently small error in risk stratification, drug choice, airway planning, or ventilatory strategy may affect patient safety.

The performance of DeepSeek-R1 was uneven across different assessment domains. It demonstrated strong capability in identifying abnormal laboratory values and structuring patient information by organ systems, which was consistent with prior studies that have shown the proficiency of LLMs in information extraction and summarization.^14,15 Although studies have indicated that Deepseek-R1 is superior in terms of knowledge recall and evaluation when assessing the chain-of-thought (CoT) in the American Society of Anesthesiologists Physical Status (ASA-PS) classification system, its performance in formulating anesthesia plans for complex cases in this study was suboptimal.¹⁶ This aligns with recent evidence that indicates that while LLMs such as ChatGPT-4 can achieve reasonable accuracy in ASA status prediction in standardized settings, their performance may decrease in nuanced, real-world scenarios that require integrated clinical judgment.¹⁷

Our study design does not allow for the isolation of the LLM's standalone clinical utility. The materials that were evaluated by residents were hybrid products that were refined by experts. Therefore, the primary focus of our study was to investigate the potential and risks of the “LLM-assisted generation + expert supervision” workflow as a teaching tool, as opposed to providing an assessment of the standalone clinical reliability of LLMs. The anesthesia assessment form generated by the LLM was revised by subject matter experts, with revision traces retained. This approach was intended to prevent misleading residents while making them aware that LLMs can generate unrecognizable errors. In this way, the aforementioned approach served as a reminder to residents not to blindly trust LLM-generated results in future clinical practice. While trainees perceived high perceived educational support in the LLM-generated drafts (8.62/10), their ability to independently identify errors previously flagged by senior anesthesiologists was limited (error recognition score: 3.19/5). Residents reported favorable perceptions after reviewing the expert-corrected materials, but this does not demonstrate improved competence. No comparator group, randomization, blinded assessment, or objective test of learning was included. In addition, the residents’ limited self-reported ability to identify expert-annotated errors raises concern about automation bias. Junior trainees may accept a fluent LLM-generated assessment as reliable simply because it is well structured. For this reason, the educational value of such materials may depend less on the original LLM output and more on how the errors are exposed, corrected, and discussed by senior clinicians. Technical strategies such as retrieval-augmented generation, guideline-constrained prompting, and domain-specific fine-tuning may mitigate some of the errors observed in general-purpose LLM outputs.¹⁸ However, even with such advances, the necessity of expert oversight is unlikely to be eliminated in high-risk clinical domains.

The application of general-purpose LLMs in real clinical settings presents several challenges. We also identified several issues in this study. Real-world electronic medical records are rife with uncertainty and may contain erroneous information or incomplete patient histories.¹¹ However, studies also indicate that case summaries generated by GenAI can rival those compiled manually by physicians in terms of completeness, accuracy, and brevity.¹⁹ For preoperative anesthesia visits, patient medical records and test results are often incomplete, and the patient's actual physical condition is rarely documented. Therefore, anesthesiologists must conduct focused bedside visits after an initial review of medical records and test results to accurately assess the patient's condition and anesthesia risks. This requires substantial clinical expertise. Scenarios that demand extensive professional knowledge for dynamic decision-making may pose challenges for LLMs. Additionally, LLMs trained with online content datasets may generate incorrect or nonsensical information, so called “hallucinations,” thereby requiring careful evaluation of their outputs.^20,21

Several features of the present workflow limit reproducibility. The model was accessed through a public web interface, generation parameters were not configurable, online search was enabled, and each case was generated once. We therefore could not assess run-to-run stability or determine how output might change with model updates. Future studies should use fixed model versions where possible, report generation settings, repeat generation in a subset of cases, and compare LLM-assisted teaching with traditional teaching using objective outcomes such as risk-stratification accuracy, error detection, structured case assessments, or objective structured clinical examination. In our view, the safest role for this workflow is not autonomous assessment, but supervised teaching. If used in training, LLM-generated drafts should be treated as preliminary material. Senior anesthesiologists should review the draft against the original medical record, correct risk stratification and anesthesia plans, retain visible correction marks when appropriate, and explicitly discuss with trainees why the original output was wrong. This process may help residents learn both preoperative assessment and the limitations of AI-generated clinical text.

In summary, this study highlights both the potential and the pitfalls of introducing LLM-assisted materials into anesthesia education. The findings emphasize that while such tools may support information structuring and risk awareness under supervision, they also introduce significant cognitive and safety risks if used without appropriate safeguards. Any integration of LLMs into anesthesiology training or practice should therefore prioritize human oversight, transparency of errors, and preservation of independent clinical judgment.

Limitations

This study has several limitations. First, it was a single-centre exploratory study based on 15 selected complex consultation cases, and the findings are not generalizable to routine preoperative assessment or to other institutions. Second, the evaluated resident-facing documents were expert-corrected LLM drafts. Therefore, the study cannot isolate the independent contribution of the LLM or assess its standalone clinical or educational performance. Third, residents’ responses were perception-based and may have been influenced by expectation bias and novelty effects, especially because neither residents nor expert raters were blinded. Fourth, no comparator group, randomization, or objective performance outcome was included. Fifth, the LLM was accessed through a public web interface; generation parameters were not controllable, online search was enabled, and each case was generated once, so output variability and reproducibility could not be assessed. Finally, although direct identifiers were removed before LLM input, the investigators could not technically verify data retention or model-training policies of the web-based system.

Conclusions

In this single-centre exploratory study, an LLM-assisted workflow was feasible for producing structured preoperative anesthesia assessment drafts, but the original drafts contained clinically relevant errors and required expert correction before resident-facing use. The favorable resident responses should be interpreted as perceived usefulness after expert correction, not as evidence of educational effectiveness. The findings highlight a practical safety concern: structurally complete LLM-generated text may still be clinically unreliable, and junior trainees may have difficulty recognizing such errors. In this hospital-based workflow, senior anesthesiologist review, visible correction of errors, and explicit discussion of AI limitations were essential safeguards. Controlled studies with objective educational and clinical outcomes are needed before such workflows can be recommended more broadly.

Authors’ contributions

SG, DS and XK designed the research study. SG, JP, HL, HX and FF performed the research. SG, JP and FF analyzed the data. SG wrote the manuscript. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

This retrospective chart review study involving human participants was in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Since this study was retrospective and only used the medical history data stored in the hospital's electronic medical record system, after passing the ethical review, ethical approval for this study (IIT20250752B) was provided by the Ethical Committee of The First Affiliated Hospital, Zhejiang University School of Medicine on June 24, 2025. The institutional review board also waived the requirement for informed consent.

Funding

This research received no external funding.

Conflict of interest

The authors declare no conflict of interest.

Data availability

No datasets were generated or analysed during the current study.