A Pragmatic Trial of AI-Assisted Discharge Summary Generation

Keywords

• Discharge Summary
• AI Note Writing Tool
• Generative artificial intelligence
• Large language model
• Clinical documentation
• Electronic health record
• Epic
• Documentation burden
• Hospital medicine
• Physician workload
• Pragmatic randomized controlled trial

Lead sponsor

Unity Health Toronto is the lead sponsor of 435 studies on the registry; 77 are open to participants now.

Of its 5 completed or terminated interventional studies of FDA-regulated products, 1 (20%) have results posted.

Study arms

• Experimental
  Epic Hospital Course Generator

  Staff physicians randomized to this arm, and resident physicians working under them on teaching services, will have access to the Epic Hospital Course Generator within the Epic electronic health record during the study period. Clinicians in this arm will be notified that they have access to the tool and will be given instructions on how to use it. Use of the tool is optional. Clinicians remain responsible for reviewing any AI-generated content and deciding whether and how to incorporate it into the discharge summary.

  Other: Epic Discharge Summary Hospital Course Generator

• No intervention
  Control

  Staff physicians randomized to this arm, and resident physicians working under them on teaching services, will not have the Epic Hospital Course Generator available for use during the study period and will complete discharge summaries using usual documentation workflows.

Interventions

• OtherEpic Discharge Summary Hospital Course Generator

  A generative AI (large language model) tool embedded in Epic that drafts a narrative of the patient's hospital course from existing chart data, including clinician notes, investigations, and medications and treatments. A draft is generated only when a clinician requests it, can be generated at any time during the hospitalization, and can be regenerated an unlimited number of times. Clinicians can choose a narrative (chronological) format, a problem-based format with a numbered entry for each clinical problem managed during the hospitalization, or an organ system-based format. The draft appears in a separate tab and is not automatically inserted into the discharge summary; clinicians may copy, edit, or disregard it.

Primary outcomes

1. Time-in-note (minutes per discharge summary)

   Total active time (minutes) spent writing and editing the discharge summary by all clinicians contributing to the note (staff physician and, on teaching services, resident physicians), measured per discharge episode using Epic audit log "time in note" metadata. Includes teaching and non-teaching services.

   Time frame: Per discharge episode, from note creation to final signature (including staff attestation where applicable), assessed up to 30 days after discharge.

Secondary outcomes

1. Time from discharge to discharge summary signature (hours)

   Time in hours between the patient's recorded hospital discharge and the final signature of the discharge summary, from Epic metadata. Summaries signed before discharge are recorded as negative values.

   Time frame: Per discharge episode, assessed up to 30 days after discharge.

2. Expert-rated documentation quality

   Blinded manual review of a random sample of 200 discharge summaries (100 per arm), each rated independently by a hospitalist and a primary care physician using the validated and standardized Provider Documentation Summarization Quality Instrument (PDSQI-9). Eight attributes (accurate, cited, comprehensible, organized, succinct, synthesized, thorough, useful) are each rated on a 5-point Likert scale (1 to 5) and summed to a total score ranging from 8 to 40; higher scores indicate better quality. The ninth attribute (free of stigmatizing language) is rated yes/no and reported separately. Reviewers will also verify accuracy against the medical record by chart review in Epic, recording omissions and factual errors in: administrative details (admission and discharge dates), diagnoses, medications and treatments, investigations, and subspecialty consultations.

   Time frame: Per discharge episode, assessed up to 30 days after discharge.

3. Automated documentation quality (PDSQI-9, LLM-as-judge)

   PDSQI-9 total score for all discharge summaries, assigned by a large language model using the published PDSQI-9 LLM-as-judge approach (Croxford et al., JAMIA 2025). Total score range 9 to 45; higher scores indicate better quality.

   Time frame: Per discharge episode, assessed up to 30 days after discharge.

4. Physician task load (PTL-4)

   Four NASA-TLX subscales (mental demand, physical demand, temporal demand, effort), each rated 0 to 100 while reflecting on a representative clinical workday in the past 1 to 2 weeks. Subscales are summed into a composite score of 0 to 400; higher scores indicate greater workload. Measured once per clinician.

   Time frame: Once per clinician, within 2 weeks of completion of an inpatient service block during the active study period (for clinicians completing more than one block, the first completed block).

5. Frequency of AI tool use

   Proportion of discharge summaries in which an AI-generated hospital course draft was generated, reported by arm (use in the control arm measures contamination).

   Time frame: Per discharge episode during the active study period.

Other outcomes

1. Total attending time in the patient chart on the day of discharge (minutes)

   Total minutes the staff physician spends with the patient's chart open on the calendar day of discharge (00:00 to 23:59), measured from Epic audit log metadata.

   Time frame: From randomization through the end of the active study period.

2. Discharge time of day

   Clock time of the patient's recorded hospital discharge, from Epic metadata, analyzed as a continuous measure and as the proportion of patients discharged before 11:00.

   Time frame: From randomization through the end of the active study period.

3. Staff physician attestation lag (hours)

   Time in hours between the resident's signature and the staff physician's attestation of the discharge summary. Teaching services only.

   Time frame: Per discharge episode, assessed up to 30 days after discharge.

4. Physician wellbeing (Professional Fulfillment Index)

   Stanford Professional Fulfillment Index subscale scores (professional fulfillment, work exhaustion, interpersonal disengagement), each scaled 0 to 100. Measured per clinician.

   Time frame: Once per clinician, within 2 weeks of completion of an inpatient service block during the active study period (for clinicians completing more than one block, the first completed block).

5. 30-day all-cause readmission

   Proportion of discharge episodes followed by an all-cause readmission to the discharging hospital within 30 days.

   Time frame: Up to 30 days after patient discharge

6. 30-day emergency department visit

   Proportion of discharge episodes followed by an ED visit to the discharging hospital within 30 days.

   Time frame: Up to 30 days after patient discharge.

7. System Usability Scale (SUS) score for the AI hospital course tool (intervention arm only)

   Perceived usability of the Epic Hospital Course Generator, measured with the System Usability Scale (Brooke, 1996). The SUS has 10 items rated on a 5-point Likert scale (strongly disagree to strongly agree), with alternating positively and negatively worded items. Item scores are converted to a total score from 0 to 100; higher scores indicate better usability. Item wording will refer to the Hospital Course Generator in place of "this system." Completed once by staff physicians randomized to the intervention arm and reported descriptively; no between-arm comparison.

   Time frame: Once per clinician, within 2 weeks of completion of an inpatient service block during the active study period (for clinicians completing more than one block, the first completed block).

Individual participant data

Plan to share: No — Individual participant data will not be made publicly available. Participants include clinicians enrolled under an opt-out consent model and patients whose health information is used under a waiver of consent; neither consented to external data sharing. Clinician-level outcome data derived from EHR audit logs and free-text discharge documentation carry meaningful re-identification risk and are subject to PHIPA, HIPAA, and institutional data governance at each site. The study protocol, statistical analysis plan, and analytic code will be made publicly available. Requests for aggregate or summary-level data from qualified researchers will be considered by the study steering committee, subject to research ethics board approval and data sharing agreements with participating institutions.