An observational study in Emergency Medicine, Artificial Intelligence (AI) and Artificial Intelligence (AI) in Diagnosis, sponsored by Marmara University Pendik Training and Research Hospital. Recruiting at 1 site in Turkey (Türkiye). Open to participants aged 18 Years and older. Per ClinicalTrials.gov, last updated 2026-06-23.
Sponsored by Marmara University Pendik Training and Research Hospital · Observational
This prospective observational diagnostic accuracy study evaluates whether large language models (LLMs) - GPT-4o (OpenAI, gpt-4o-2024-11-20) and Claude (Anthropic, claude-sonnet-4-6) - can accurately calculate HEART scores from unstructured Turkish clinical notes and predict 30-day major adverse cardiac events (MACE) in emergency department patients presenting with non-traumatic chest pain.
The study will enroll 600 consecutive adult patients. For each patient, the same anonymized data (free-text anamnesis, ECG report text, troponin value, and age) will be independently processed by both LLMs via separate API calls with deterministic settings (temperature=0, JSON format). A three-expert consensus HEART score - derived through blinded independent scoring by three emergency medicine physicians with majority-vote adjudication - serves as the reference standard for agreement analysis. Actual 30-day MACE (all-cause death, AMI Type 1/2/4b, unplanned revascularization) determined via national health database and telephone follow-up serves as the outcome for diagnostic accuracy analysis.
A secondary documentation-quality sub-study will quantify how spontaneously Turkish emergency anamnesis notes capture HEART score parameters.
AI SYSTEM SPECIFICATIONS AND PROMPT PROTOCOL Two distinct large language models (LLMs) will be evaluated as index tests: OpenAI GPT-4o (model string: gpt-4o-2024-11-20) and Anthropic Claude (model string: claude-sonnet-4-6). To ensure reproducibility and eliminate stochastic variation, both models will be accessed via standardized API calls using deterministic parameters (temperature = 0, max_tokens = 500, and strict JSON response format). The exact system prompt layout will be locked prior to initialization, and its integrity will be verified using a SHA-256 cryptographic hash. The models will evaluate each patient record independently in zero-shot isolation, with no cross-contamination or conversational history retention between runs.
REFERENCE STANDARD CONSENSUS PROTOCOL The reference standard consists of a structured consensus HEART score established by three independent emergency medicine physicians (each possessing >=3 years of clinical experience and specific training on HEART score criteria). The physicians will review the anonymized clinical charts while remaining strictly blinded to the LLM outputs and the final 30-day MACE outcomes. For each of the 5 HEART components (scored 0, 1, or 2), a majority vote (2/3 agreement) will determine the final component score. In the event of complete disagreement across all three reviewers on a specific component, a fourth independent adjudicator will resolve the tie.
INDETERMINATE RESULTS MANAGEMENT
In strict compliance with STARD-AI 2025 guidelines, cases with missing or uninterpretable parameters within the free-text clinical notes will be classified into predefined indeterminate tiers:
STATISTICAL ANALYSIS AND AGREEMENT WEIGHTING Statistical power and sample size calculation are based on the Hanley-McNeil methodology for the Area Under the ROC Curve (AUC). To achieve an expected AUC of 0.85 with a non-inferiority margin of 0.05, a power of 80%, and a two-sided alpha of 0.05, the primary complete-case analysis requires 600 evaluable patients. Accounting for an anticipated 15% indeterminate rate, a total enrollment target of 690 patients is set. Inter-rater agreement between each LLM and the expert consensus will be computed using quadratic weighted Cohen's Kappa for the ordinal total HEART score (0-10) and linear weighted Kappa for individual components (0-2). Diagnostic performance metrics (sensitivity, specificity, PPV, NPV) will be calculated at prespecified binary (>=4) and trimodal thresholds with 95% Wilson confidence intervals. Pairwise comparison of AUC values between GPT-4o and Claude will be executed using the DeLong test.
DATA ANONYMIZATION AND PRIVACY To ensure full compliance with local personal data protection legislation (KVKK), all free-text emergency department notes will undergo strict de-identification. Patient names, institutional ID numbers, precise dates, and specific demographic identifiers will be stripped entirely before formatting the data payload for API transmission.
PATIENT AND PUBLIC INVOLVEMENT BEYANI Patient and public involvement was not applicable to this study as it involves the analysis of routinely collected clinical data.
Marmara University Pendik Training and Research Hospital is the lead sponsor of 34 studies on the registry; 15 are open to participants now.
Counted across the registry records on this site, refreshed daily.
The study population consists of consecutive adult patients presenting with a chief complaint of non-traumatic chest pain to the emergency department of Marmara University Pendik Training and Research Hospital, a tertiary care academic medical center in Istanbul, Turkey. This target population comprises real-world emergency medicine admissions that require acute coronary syndrome risk stratification and evaluation with the HEART score. It excludes individuals presenting with traumatic pain etiologies or acute ST-elevation myocardial infarction (STEMI) requiring immediate, time-critical reperfusion pathways.
Go through the criteria one by one — yes, no or not sure — and take the list to your doctor or the study team. Kept in this browser only.
EXCLUSION CRITERIA:
WITHDRAWAL CRITERIA:
Underlined terms in the criteria open these too. Explained by clinicaltrials.gg, not part of the registry record. The study team decides who can take part.
Before you join, the study team explains what taking part involves — its purpose, what happens, the risks and possible benefits — and answers your questions, so you can decide freely. Consent is usually recorded on a form (the informed consent form, ICF) signed by you or, where the law allows, by someone legally authorised to decide for you.
You can keep asking questions during the study, you are told of new information that matters, and you can choose to leave at any time.
Informed consent
Before you join, the study team explains what taking part involves — its purpose, what happens, the risks and possible benefits — and answers your questions, so you can decide freely. Consent is usually recorded on a form (the informed consent form, ICF) signed by you or, where the law allows, by someone legally authorised to decide for you.
You can keep asking questions during the study, you are told of new information that matters, and you can choose to leave at any time.
Explained by clinicaltrials.gg, not part of the registry record. The study team decides who can take part.
OpenAI GPT-4o (model: gpt-4o-2024-11-20, temperature=0, max\_tokens=500, response\_format=JSON). Each patient's anonymized anamnesis text, ECG report text, troponin value, and age are submitted via a separate API call with no conversation history. Output: HEART score components (0-2 each), total score (0-10), risk group, and indeterminate status.
Also known as: GPT-4o İndeks Testi
Anthropic Claude (model: claude-sonnet-4-6, temperature=0, max\_tokens=500, response\_format=JSON). Identical system prompt and input format as GPT-4o. Processed independently with no cross-contamination between models. Output: same JSON schema as GPT-4o.
Also known as: Claude İndeks Testi
Three emergency medicine physicians (\>=3 years experience, HEART-score trained) independently score each anonymized record. Majority vote (2/3) determines component scores; a 4th adjudicator resolves ties. Experts are blinded to LLM scores, each other's scores, and MACE outcomes.
Also known as: Referans Standart
Area Under the ROC Curve (AUC) of GPT-4o and Claude HEART Score for 30-Day MACE Prediction
AUC calculated separately for GPT-4o and Claude using the Hanley-McNeil method. MACE is defined as a composite of all-cause death, acute myocardial infarction (Type 1/2/4b), and unplanned revascularization within 30 days. HEART score range is 0-10; a higher score indicates a higher risk of MACE. Analysis will be performed on complete cases only (0 indeterminate components).
Time frame: 30 days after index emergency department visit
Sensitivity and Specificity of GPT-4o and Claude HEART Score at Prespecified Thresholds
Diagnostic sensitivity and specificity calculated at two threshold types: (a) total score \>=4 (binary high-risk cutoff) and (b) trimodal cutoffs (0-3 low risk, 4-6 intermediate risk, 7-10 high risk). Metrics will be reported with 95% Wilson confidence intervals separately for each LLM.
Time frame: 30 days after index emergency department visit
Component-Level and Total-Score Agreement (Cohen's Kappa) Between LLMs and Expert Consensus
Inter-rater agreement will be computed using quadratic weighted Cohen's Kappa for the ordinal total HEART score (range 0-10) and linear weighted Kappa for the individual components (range 0-2). Calculated separately for GPT-4o vs. expert consensus and Claude vs. expert consensus. Values will be interpreted using the Landis \& Koch scale (\<0.20 poor, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 good, \>0.80 excellent).
Time frame: Baseline (At index emergency department visit)
Comparative AUC Difference Between GPT-4o and Claude (DeLong Test)
Statistical comparison of paired ROC curves between GPT-4o and Claude using the DeLong et al. (1988) method. The formal hypothesis is non-inferiority with an expected delta AUC \<= 0.05. The correlation coefficient between the paired LLM measurements is estimated as rho \>= 0.70.
Time frame: 30 days after index emergency department visit
Proportion of Indeterminate Results for GPT-4o and Claude
The proportion of cases classified into predefined missing data tiers: Complete (0 indeterminate components), Partial indeterminate (exactly 1 missing component preventing definitive score calculation), and Full indeterminate (\>=2 missing components). Reported separately for each LLM and statistically compared between the two models.
Time frame: Baseline (At index emergency department visit)
HEART Parameter Documentation Rate in Routine Turkish Anamnesis Notes
For each of the 5 individual HEART components, the proportion of emergency department free-text anamnesis notes that spontaneously contain sufficient objective clinical information for scoring. Rates will be categorized as: Present and scorable, Partiall
Time frame: Baseline (At index emergency department visit)
Subgroup AUC by Age Group and Sex (Algorithmic Bias Assessment)
AUC values for 30-day MACE prediction were calculated separately across demographic strata: age groups (\<45, 45-64, \>=65 years) and biological sex (male vs. female). This analysis serves as the formal algorithmic bias assessment required by the STARD-AI 2025 guidelines.
Time frame: 30 days after the index emergency department visit
Plan to share: Yes — Anonymized individual participant data (including de-identified baseline demographics, clinical presentation characteristics, index test outputs from GPT-4o and Claude, and the reference standard expert consensus HEART scores) will be made publicly available to support academic transparency and replication. Additionally, the complete deterministic system prompt texts (verified with SHA-256 cryptographic hashes) and the complete statistical analysis code will be included as supplementary material.
Supporting information: Study protocol, Statistical analysis plan, Analytic code
Time frame: The anonymized dataset, protocol documents, and analytic code will be made available immediately upon formal publication of the study results.
Access criteria: Data and code will be accessible via an open-access repository on the Open Science Framework (OSF) for researchers and clinicians interested in replication or meta-analysis.
Eligibility is decided by the study team. Share this record with your doctor or contact the team directly.
Contact study teamGet an email when the registry record changes — status, dates, results — or when someone posts here.
Sign in to followQuestions and observations about this study, from anyone following it. Not medical advice, and not a channel to the study team — their contact details are on the registry record.
Sign in to join the discussion. Reading takes no account; posting does. You choose a display name, and a pseudonym is the default.
Nothing here yet. If you are running this trial, taking part in it, or weighing whether to, this is the place to say so.
Marmara University Pendik Training and Research Hospital