LLM Psychiatric Intake AI Clinical Performance Study

Browse trials for

• Mental Disorders 2,148
• Psychological Well-Being 1,045

Keywords

• LLM
• Large Language Models
• Artificial Intelligence
• AI intake
• Psychiatric Intake
• psychiatric interview
• digital intake
• Clinician-in-the-Loop
• clinical performance
• screening
• mental health
• Caregiver

Mental Disorders

2,148 studies on the registry are indexed under Mental Disorders; 425 are open to participants now.

This study's planned enrollment of 120 is above the median of 96 across 1,607 interventional studies indexed under Mental Disorders.

Lead sponsor

Gangnam Severance Hospital is the lead sponsor of 145 studies on the registry; 37 are open to participants now.

Study arms

• Experimental
  LLM-Based Psychiatric Intake Interview

  Caregivers complete an approximately 30-minute adaptive LLM-based AI psychiatric intake interview before/around the outpatient visit. The AI system generates a structured intake report for the treating clinician. Experts rate de-identified AI interview logs/reports. Treating clinicians review the AI report in routine care and provide qualitative feedback.

  Other: LLM-based clinician-in-the-loop psychiatric AI-assisted intake assistance system

Interventions

• OtherLLM-based clinician-in-the-loop psychiatric AI-assisted intake assistance system

  Caregivers complete a single adaptive LLM-based psychiatric intake interview remotely before the patient's scheduled initial or re-intake visit. The web-based interview is accessible by smartphone or tablet and lasts up to 30 minutes. Guided by K-SADS-PL and DSM-5-based clinical domains, the system adaptively explores information typically collected from caregivers during an initial psychiatric assessment. The system generates a structured clinician-facing report containing an interview summary, diagnostic screening candidates for nine prespecified target conditions, safety-related findings, and areas requiring further clarification. The system incorporates mental health-specific safety guardrails tailored to the developmental characteristics and safety needs of children and adolescents. These guardrails restrict diagnostic and treatment advice and detect potential risks, including self-harm, harm to others, suicidal history, substance misuse, psychotic symptoms, and suspected child ab

  Also known as: AI psychiatric intake chatbot; LLM-based preliminary intake system; clinician-in-the-loop AI intake

Primary outcomes

1. Pooled sensitivity of AI-generated diagnostic candidates for the nine target disorders

   The pooled sensitivity of the AI-generated diagnostic candidates will be evaluated against the reference primary diagnosis determined by a psychiatrist based on clinical information accumulated during the 8 weeks following the initial clinical visit. Among participants whose reference primary diagnosis corresponds to one of the nine target disorders, a true-positive case is defined as one in which the AI-generated candidate list includes the reference diagnosis. A false-negative case is defined as one in which the AI does not include the reference diagnosis or returns a deferred decision. Pooled sensitivity is calculated as the number of true-positive cases divided by the total number of participants whose reference primary diagnosis is one of the nine target disorders. The outcome ranges from 0 to 1, with higher values indicating better diagnostic candidate-generation performance.

   Time frame: Up to 8 weeks after the day 1

Secondary outcomes

1. Disorder-specific sensitivity of AI-generated diagnostic candidates for each of the nine target disorders

   Sensitivity will be calculated separately for each of the nine target disorders. For each disorder, a true-positive case is defined as a participant whose reference primary diagnosis is the corresponding target disorder and whose AI-generated candidate list includes that disorder. A false-negative case is defined as a participant whose reference diagnosis is the corresponding target disorder but for whom the AI does not include the disorder or returns a deferred decision. Sensitivity is calculated as true positives divided by the sum of true positives and false negatives and will be reported with a 95% confidence interval. The outcome ranges from 0 to 1, with higher values indicating better performance.

   Time frame: Up to 8 weeks after the day 1

2. Specificity of AI-generated diagnostic candidates for identifying cases outside the nine target disorders

   Specificity will be evaluated among participants whose reference primary diagnosis is outside the nine target disorders. A true-negative case is defined as one in which the AI does not suggest any of the nine target disorders and returns a deferred decision. A false-positive case is defined as one in which the AI suggests at least one target disorder. Specificity is calculated as the number of true-negative cases divided by the total number of participants whose reference diagnosis is outside the target-disorder range. The outcome ranges from 0 to 1, with higher values indicating more appropriate deferral performance.

   Time frame: Up to 8 weeks after the day 1

3. Clinical information-gathering and interviewing performance assessed by the modified Mini-CEX

   (\*Note: Seven separately enrolled mental health experts retrospectively rate AI intake transcripts within 1 month after caregiver data collection (approximately 2-3 months.) Seven mental health experts who are psychiatrists or qualified clinical psychologists independently assess the AI system using a modified version of the Mini-Clinical Evaluation Exercise. The modified instrument includes five domains: medical interviewing skills, humanistic qualities and professionalism, clinical judgment, organization and efficiency, and overall clinical competence. Each domain is rated on a 9-point scale, with scores of 1-3 indicating unsatisfactory performance, 4-6 indicating satisfactory performance, and 7-9 indicating superior performance. The five domain scores are summed to produce a total score ranging from 5 to 45. Higher scores indicate better clinical information-gathering and interviewing performance.

   Time frame: Within 1 month after completion of caregiver data collection

4. Information-elicitation and communication performance assessed by the SEGUE Framework

   (\*Note: Seven separately enrolled mental health experts retrospectively rate AI intake transcripts within 1 month after caregiver data collection (approximately 2-3 months.) Seven mental health experts who are psychiatrists or qualified clinical psychologists independently assess the AI conversation transcripts using the 25-item SEGUE Framework. The framework evaluates observable communication behaviors across five domains: setting the stage, eliciting information, giving information, understanding the user's perspective, and ending the encounter. Each behavior is coded as present or absent. The item scores are summed to produce a total score, with higher scores indicating that the AI demonstrated a greater number of appropriate information-elicitation and communication behaviors.

   Time frame: Within 1 month after completion of caregiver data collection

5. Clinical summarization and documentation quality assessed by the PDSQI-9

   (\*Note: Seven separately enrolled mental health experts retrospectively rate AI intake transcripts within 1 month after caregiver data collection (approximately 2-3 months.) Seven mental health experts who are psychiatrists or qualified clinical psychologists independently evaluate the AI-generated clinical summaries using the 9-item Provider Documentation Summarization Quality Instrument. The instrument assesses clinically relevant dimensions of documentation quality, including accuracy, completeness, organization, usefulness, hallucinations, omissions, and stigmatizing language. Responses are coded according to a prespecified scoring procedure and combined to produce a total documentation quality score. Higher total scores indicate better clinical summarization and documentation quality.

   Time frame: Within 1 month after completion of caregiver data collection

6. Clinical safety of the AI intake system assessed by the total safety risk score

   (\*Note: Seven separately enrolled mental health experts retrospectively rate AI intake transcripts within 1 month after caregiver data collection (approximately 2-3 months.) Seven mental health experts who are psychiatrists or qualified clinical psychologists independently review each AI conversation for safety concerns in six domains: inappropriate intervention, hallucination within the conversation, failure to detect a crisis signal, inappropriate emotional or relational response, ethical or boundary violation, and failure to consider developmental characteristics. Each domain is classified as safe, minor, potentially risky, or critical failure and coded as 0, 1, 2, or 3, respectively. The six domain scores are summed to produce a total safety risk score ranging from 0 to 18. Higher scores indicate greater clinical safety concerns.

   Time frame: Within 1 month after completion of caregiver data collection

7. Clinical safety: Activation and response outcomes of the automated safety guardrail

   Safety guardrail events are automatically recorded by the AI intake system and extracted by the research team after completion of overall study data collection. For each AI intake, the system records whether the safety guardrail was activated, the time of activation, the type of risk signal detected, the response provided by the system, and whether the AI intake was automatically discontinued. The number and proportion of AI intakes involving guardrail activation will be reported, together with the distribution of detected risk-signal categories, system responses, and safety-related intake discontinuations. Because guardrail activation depends on the occurrence of risk signals, a higher activation rate does not necessarily indicate poorer system safety.

   Time frame: Day 1

8. Emotional safety of the AI intake assessed by change in caregiver-reported emotional symptom intensity

   The participating caregiver completes the five Emotion Thermometers immediately before and immediately after using the AI intake system. Four thermometers assess distress, anxiety, depression, and anger, and the fifth assesses the perceived need for help. Each domain is rated from 0 to 10. The distress, anxiety, depression, and anger scores are summed to produce a prespecified exploratory composite emotional symptom intensity score ranging from 0 to 40, with higher scores indicating greater overall emotional symptom intensity. Changes in the four individual emotional-domain scores are also reported separately. The need-for-help score is analyzed separately and is not included in the composite score because it assesses perceived help-seeking need rather than emotional symptom intensity.

   Time frame: Day 1

9. Caregiver-perceived communication quality assessed by the Communication Assessment Tool

   The participating caregiver evaluates the AI system's communication quality immediately after completing the AI intake. Communication quality is assessed using 14 communication-focused items adapted from the Communication Assessment Tool. The items assess whether the AI system enabled the caregiver to speak without interruption, paid attention to the caregiver's responses, used understandable language, checked understanding, encouraged questions, demonstrated care and concern, treated the caregiver with respect, and used an appropriate amount of time. Each item is rated on a 5-point scale from 1, poor, to 5, excellent. The 14 item scores are summed to produce a prespecified total score ranging from 14 to 70. Higher scores indicate better caregiver-perceived communication quality.

   Time frame: Day 1

10. Comfort with self-disclosure during the AI intake

    The participating caregiver completes a 4-item self-report measure immediately after using the AI intake system. The measure assesses perceived comfort, ease, and openness in disclosing personal or sensitive information during the AI intake. Each item is rated on a 7-point Likert scale from 1, strongly disagree, to 7, strongly agree. The mean of the four item scores is calculated, producing a score ranging from 1 to 7. Higher scores indicate greater comfort with self-disclosure during the AI intake.

    Time frame: Day 1

11. Acceptability and appropriateness of the AI intake assessed by the AIM

    The participating caregiver evaluates the acceptability and perceived appropriateness of the AI intake system using the Acceptability of Intervention Measure. The Acceptability of Intervention Measure consists of 4 items assessing whether the AI intake is perceived as acceptable, appealing, likable, and welcome. Each item is rated on a 5-point Likert scale from 1, completely disagree, to 5, completely agree. The mean of the four item scores is calculated separately for each measure, producing an acceptability score and an appropriateness score, each ranging from 1 to 5. Higher scores indicate greater acceptability or greater perceived appropriateness, respectively. The scores of the two measures are not combined into a single total score.

    Time frame: Day 1

12. Acceptability and appropriateness of the AI intake assessed by the IAM

    The participating caregiver evaluates the acceptability and perceived appropriateness of the AI intake system using the Intervention Appropriateness Measure. The Intervention Appropriateness Measure consists of 4 items assessing whether the AI intake is perceived as fitting, suitable, applicable, and a good match for its intended purpose and clinical context. Each item is rated on a 5-point Likert scale from 1, completely disagree, to 5, completely agree. The mean of the four item scores is calculated separately for each measure, producing an acceptability score and an appropriateness score, each ranging from 1 to 5. Higher scores indicate greater acceptability or greater perceived appropriateness, respectively. The scores of the two measures are not combined into a single total score.

    Time frame: Day 1

Other outcomes

1. Counseling and relational competence

   (\*Note: Seven separately enrolled mental health experts retrospectively rate AI intake transcripts within 1 month after caregiver data collection (approximately 2-3 months.) mental health expert retrospectively assess the AI conversation transcripts using an adapted 12-item version of the Counseling Competencies Scale-Revised (CCS-R). A prespecified total score is calculated, with higher scores indicating better counseling and relational competence.

   Time frame: Up to 2-month after Day 1

2. Behavioral self-disclosure

   On Day 1, immediately after the AI intake, participating caregivers rate the perceived breadth and emotional depth of the information they disclosed during the AI conversation. The breadth and depth ratings are reported separately, with higher scores indicating broader or more emotionally deep self-disclosure.

   Time frame: Day 1

3. Relational empathy

   On Day 1, immediately after the AI intake, participating caregivers assess the empathy and relational qualities demonstrated by the AI system using the Consultation and Relational Empathy (CARE) Measure. A total score is calculated, with higher scores indicating greater perceived empathy and relational quality.

   Time frame: Day 1

4. Qualitative outcomes

   Up to 8-week after the day 1, Semi-structured interviews are conducted with up to 20 participating caregivers who provide additional consent to the qualitative interview. The caregiver interviews are conducted after completion of the AI intake, during the waiting period for the routine clinical visit. Within 1 month after completion of caregiver data collection, semi-structured interviews are also conducted with participating mental health experts after completion of their retrospective evaluations and with treating clinicians after completion of overall study data collection. The interviews explore usability, acceptability, comfort with self-disclosure, perceived benefits and limitations, clinical usefulness, workflow integration, safety concerns, and recommendations for system improvement. Interview data are analyzed qualitatively to identify recurring themes.

   Time frame: Up to 8-week after the day 1

Publications

• Jung Y, Vo THA, Moon HS, Choi JY, Oh H, Lee U, Kim E, Lee TY, Lee U. Toward Flexible Psychiatric History-Taking and Visualization: Exploring Clinician Perspectives with Large Language Models. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 2026; Article 503:1-23. doi:10.1145/3772318.3790970.

Individual participant data

Plan to share: Undecided — Whether de-identified individual participant data will be made available after study completion to researchers outside the IRB-approved study team has not been determined. Any future external sharing will require institutional and IRB review and must be consistent with the scope of participant consent.