ClinicalTrials.gg
RecruitingNCT07739121BEACONUpdated Oct 7, 2026

Benchmarking Large Language Models Against Tumour Boards for Oncology Treatment Recommendations

An observational study in Breast Neoplasms, Lung Neoplasms and Urologic Neoplasms, sponsored by Assistance Publique - Hôpitaux de Paris. Recruiting at 5 sites in France. Open to participants aged 18 Years and older. Per ClinicalTrials.gov, last updated 2026-10-07.

Sponsored by Assistance Publique - Hôpitaux de Paris · Observational

How we know

From the registry’s dates

  • Started May 2026; still recruiting 6 months later.
Updated Oct 7, 2026Primary completion movedStudy completion moved+3 moreGo to Updates ↓

Help build ClinicalTrials.gg — short surveys, remote interviews or testing, in your language. Take part →

Study type
Observational
Model
Cohort
Time perspective
Prospective
Enrollment
100
Ages
18 Years and older
Sex
All
01

Where this study stands

Counted across the records on this site · 2026-11-22

Among 32 studies in Breast Neoplasms by Assistance Publique - Hôpitaux de Paris — 10 open to participants now, 13 under way, none of them phase 3. Breast Neoplasms · Assistance Publique - Hôpitaux de Paris.

Planned enrollment
16th of 19
100 · median 320
Started
2026
with 5 others
Observational
19 of 32
8 open now
Primary completion
2026-12
later than 7 of 12 live
Registered
31st of 32
by date

Among the 32 studies in Breast Neoplasms by Assistance Publique - Hôpitaux de Paris, this one is the 16th largest by planned enrollment of the 19 observational ones that give a count, is one of 6 that started in 2026, is one of 19 observational ones, reaches primary completion later than 7 of the 12 other live ones, and was the 31st registered.

Breast Neoplasms

14,855 studies on the registry are indexed under Breast Neoplasms; 3,321 are open to participants now.

Browse Breast Neoplasms studies →

Open to participants now

10 of the 32 studies in Breast Neoplasms by Assistance Publique - Hôpitaux de Paris are open now, in 1 country. Recruiting first, the newest first:

4 more — sign in →

By country, each count listing its studies:

All 10 open studies →

Where they have run

Who these studies take

Of the 2,403 Breast Neoplasms studies recruiting now, 1,574 (66%) set no upper age limit and 591 (25%) stop at 75 or younger; 103 take children under 18; 215 take healthy volunteers.

This study: 18 Years and older · all sexes · no healthy volunteers.

The full table on the hub →

This tab as a report to print or save as PDF → With an account.

Was this useful?
Get alerts for this landscapeDownload CSVNarrow it in search →With an account: a weekly mail of what changed among 32 studies in Breast Neoplasms by Assistance Publique - Hôpitaux de Paris, or all of them as a file.
02

Study summary

BEACON (Benchmarking AI for Clinical Oncology decisioNmaking) is a prospective, multicentre, comparative, blinded, non-interventional benchmark comparing the treatment recommendations of six frontier large language models (LLMs) with those of multidisciplinary tumour boards (RCP) in oncology treatment planning. One hundred standardised synthetic cases (20 per localisation, across breast, lung, urological, digestive and gynaecological cancers) are submitted as identical structured input to two independent tumour boards per localisation and to six frontier LLMs. Every recommendation, human or model, is decomposed into five predefined decision domains (intent, surgery, radiotherapy, systemic therapy with its line, work-up and biomarkers). For each domain, concordance between two recommendations is scored 2 when the same options are selected, 1 when at least one option is shared, and 0 when none is shared; the score is computed automatically from the recommendation forms, without human scoring. The primary endpoint is the mean domain score between each LLM and each tumour board, read against the same score computed between the two tumour boards of the same case, so that humans and models are measured by an identical rule and no respondent contributes to its own reference. A co-primary safety endpoint captures the proportion of recommendations carrying serious harm potential, because concordance alone can conceal dangerous errors. Where the two boards select different options, the case-domain is classified EQUIPOISE and, like every discordance, is adjudicated by an independent panel blinded to the source of the recommendation. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr) that reveals a case only once it is presented and time-stamps that reveal, so that no board can read a case in advance. BEACON is designed as reusable, openly licensed, pre-registered infrastructure: all synthetic cases, evaluation rubrics, scoring algorithms, verbatim prompts and raw model outputs are released for full reproducibility.

Read the detailed description

BEACON is a prospective, multicentre, blinded benchmark using automated, criteria-based scoring. It is built on three design decisions that distinguish it from the existing literature: (i) synthetic, standardised cases remove the record-completeness variability that confounds retrospective comparisons and allow the identical input to be given to every board and every model; (ii) two independent tumour boards per localisation let human-to-human agreement be measured rather than assumed, and that agreement is the level against which model performance is read; and (iii) an independent panel, blinded to the source, adjudicates every discordance and every case-domain where the two boards diverge, so that clinically equivalent options are recognised as such and dangerous ones are named.

Reference standard. For each case-domain, where the two boards select the same options their agreement is the reference; where they differ, the case-domain is classified EQUIPOISE, analysed separately and adjudicated blind.

Five decision domains. Every recommendation is decomposed into D1 Intent, D2 Surgery, D3 Radiotherapy, D4 Systemic therapy (class and line), and D5 Work-up and biomarkers before any comparison. Deliberate silence on a domain is ticked on the form as an explicit category and is scored as a recommendation, never as missing data.

Model selection. Models are chosen by a rule fixed before the first call: for each organisation, the most capable text model its API exposes on the freeze date, whether stable or in preview.

Data collection. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr). Each board receives a single access link, with no account to create. Outside a session the platform shows case numbers only; a case is revealed when the board presents it, and that moment is time-stamped. The recommendation form must be saved before the session closes, failing which the case cannot be opened again. Case text cannot be selected or copied and carries a watermark with the board name and the time of display. At closure the coordinator attests that the rules of the study were followed during the session. Session opening times, case reveal times, intervals to submission and attestations are exported with the data.

03

Conditions studied

  • Breast Neoplasms
  • Lung Neoplasms
  • Urologic Neoplasms
  • Prostatic Neoplasms
  • Urinary Bladder Neoplasms
  • Kidney Neoplasms
  • Digestive System Neoplasms
  • Genital Neoplasms
  • Artifical Intelligence
  • Large Language Models
  • Decision Making
  • Decision Support Systems, Clinical

Keywords

  • Large Language Models
  • Artificial Intelligence
  • Clinical Decision support
  • Decision Support Systems, Clinical
  • Multidisciplinary tumour board (RCP)
  • Patient Care Team
  • Medical oncology
  • Neoplasms
  • Benchmark
  • Benchmarking
  • Concordance
  • Weighted kappa
  • Synthetic data
  • Patient safety
  • Reproductibility
04

Who can participate

Ages eligible
18 Years and older
Sexes eligible
All
Accepts healthy volunteers
No
Sampling method
Non-probability sample

Study population

100 synthetic oncology treatment-planning cases (20 per localisation) across five localisations: breast, lung, urological (prostate, bladder / upper-tract urothelial, kidney), digestive and gynaecological. Each case is a structured input specifying UICC 8th-edition stage, biomarkers, ECOG performance status, comorbidities and a standardised clinical question. No human participants, no patient data and no identifiable individuals. Recommendations are produced by two independent tumour boards per localisation, eight distinct boards in total, and by six frontier LLMs queried in September and October 2026.

Go through the criteria one by one — yes, no or not sure — and take the list to your doctor or the study team. Kept in this browser only.

Inclusion criteria

  • Synthetic oncology case within one of the five predefined localisations (breast, lung, urological, digestive, gynaecological).
  • Complete structured schema: UICC 8th-edition stage, biomarkers, performance status, comorbidities and a standardised clinical question.
  • A clinically answerable treatment-planning question that is mappable to the locked guideline matrix.

Exclusion criteria

Exclusion Criteria:

  • Case outside the five predefined localisations.
  • Incomplete, internally inconsistent or ambiguous schema.
  • Duplicate or near-duplicate of an existing case in the set.
  • Question not resolvable by current guidelines.
Terms in these criteria, in plain language(1)

Underlined terms in the criteria open these too. Explained by clinicaltrials.gg, not part of the registry record. The study team decides who can take part.

ECOG performance status

A scale of how much an illness limits everyday activity. For people being assessed to join a study, scores run from 0 (fully active) to 4; lower is more active.

0
Fully active, no restrictions.
1
Limited in strenuous activity, but walking and able to do light work, such as housework or office work.
2
Walking and able to look after yourself, but unable to work; up and about more than half of waking hours.
3
Able to look after yourself only in part; in bed or a chair more than half of waking hours.
4
Unable to look after yourself at all; in bed or a chair all day.

Studies most often ask for 0–1 or 0–2. Your doctor gives the score; the WHO scale uses the same steps. The full scale also has 5, which records death.

05

Study design

Observational model
Cohort
Time perspective
Prospective
Enrollment
100 participants (estimated)
Patient registry
No

Groups and cohorts

  • Breast cancers

    Other: Multidisciplinary tumour boards · Other: Frontier large language models

  • Lung cancers

    Other: Multidisciplinary tumour boards · Other: Frontier large language models

  • Urological cancers

    Other: Multidisciplinary tumour boards · Other: Frontier large language models

  • Digestive cancers

    Other: Multidisciplinary tumour boards · Other: Frontier large language models

  • Gynaecological cancers

    Other: Multidisciplinary tumour boards · Other: Frontier large language models

Interventions

  • OtherMultidisciplinary tumour boards

    Two independent tumour boards per localisation, eight distinct boards in total, issue a categorical recommendation for every synthetic case during the board meeting itself, on a dedicated platform (https://etude-beacon.fr) that reveals a case only once it is presented and time-stamps that reveal. A case opened during a session must be completed before that session closes, and each board presents all of its cases within one month of its first session. Where both boards select the same options, their agreement defines the reference; where they differ, the case-domain is classified EQUIPOISE and analysed separately.

  • OtherFrontier large language models

    Six frontier LLMs receive the identical structured input for every case, under locked and hashed prompts, versions and settings: GPT-6 Astra (gpt-6-astra, OpenAI), Claude Fable 5.1 (claude-fable-5-1, Anthropic), Gemini 3.1 Pro (gemini-3.1-pro-preview, Google DeepMind), Muse Spark 1.3 (muse-spark-1.3, Meta), Mistral Large 3 (mistral-large-2512, Mistral AI, open weights) and Kimi K3 (kimi-k3, Moonshot AI, open weights). Each case is submitted five times per model in independent sessions at temperature 1.0, with one additional pass at temperature 0 as a sensitivity analysis, except for Kimi K3 whose provider fixes temperature at 1.0 and rejects the parameter. No tools, no web access, no retrieval, and no context carried from one call to the next.

06

What researchers measure

Primary outcomes

  1. Domain-level concordance between LLM recommendations and tumour board recommendations

    For each of the five decision domains, concordance between a model recommendation and a tumour board recommendation is scored 2 when the same options are selected (shared deliberate silence included), 1 when at least one option is shared, and 0 when none is shared. The measure is the mean domain score, reported with the proportion of domains scored 2, and is read against the same measure computed between the two tumour boards of the same case.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  2. Proportion of recommendations carrying serious harm potential (Tier 2)

    Each recommendation, from a model or from a tumour board, is rated on a three-tier harm scale by an independent panel blinded to its source, two raters per form and a third in case of disagreement. The measure is the proportion of recommendations rated Tier 2, that is serious harm or loss of chance.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

Secondary outcomes

  1. Inter-tumour board domain-level concordance (human baseline)

    The same 0/1/2 domain score computed between the two independent tumour boards of each case. It sets the human level against which the primary endpoint is read.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  2. Equipoise rate

    Proportion of case-domains where the two tumour boards select different options.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  3. Chance-corrected agreement (Gwet's AC1)

    Agreement on recommendation categories corrected for chance, computed between the two tumour boards and between each model and each board. AC1 is used in preference to Cohen's kappa because the distribution of recommendations is markedly unbalanced, a situation in which kappa collapses while observed agreement stays high.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  4. Agreement with at least one tumour board

    Proportion of case-domains where the model's selection matches the selection of at least one of the two boards, including case-domains classified EQUIPOISE.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  5. Completeness

    Proportion of required domains addressed (LLM and tumour boards).

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  6. Missingness

    Proportion of critical omissions (LLM and tumour boards)

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  7. Intensity bias

    Proportion of recommendation corresponding to over- or under-treatment

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  8. Intra-model consistency

    Agreement between the five independent sessions of the same model on the same case.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

  9. Proportion of unusable model responses

    Proportion of calls returning a refusal, an unparsable response, or a response violating the coherence rules that the platform applies to tumour boards. Those rules are not applied to the models: their violations are measured, not corrected.

    Time frame: Assessed once at central scoring, after data collection (estimated January 2027)

07

Study locations

1 of 5 sites recruiting
08

References and documents

Individual participant data

Plan to share: Yes — All 100 synthetic cases, the recommendation form, the verbatim prompts with their SHA-256 hashes, the raw model outputs including refusals and unparsable responses, the pseudonymised tumour board forms, the adjudication ratings and the analysis code are deposited under an open licence with the publication. No individual participant data in the usual sense exists: the cases are synthetic and the only human data collected concerns professionals and their centres.

Supporting information: Study protocol, Statistical analysis plan, Analytic code

Time frame: From publication, with no end date.

Access criteria: Open access, no request required.

09

Timeline

1 registry update since Sep 25, 2026
ElapsedEstimatedEstimated, not yet reachedDate before it movedWhere the date was before it movedSimilar studies: half by the dot; the band is the middle halfHalf of similar studies reached it by the dot; the band is the middle half, from a quarter to three quarters

Started May 2026 (actual). Primary completion is estimated for Dec 2026, 7 months after the start.

Of 1,193 non-industry observational studies under Breast Neoplasms, half reached primary completion within 58 months of starting (the median), counting those still running; a quarter within 25 and three quarters within 139. This study's planned 7 months is below that.

Counting every sponsor, half of 1,382 studies reached it within 55 months; a quarter within 25 and three quarters within 126.

Since we started tracking on Sep 25, 2026, the dates moved once, on Oct 7, 2026: primary completion Oct 2026 → Dec 2026 and completion Oct 2026 → Mar 2027.

Earlier changes are on the record history on ClinicalTrials.gov ↗

How these figures are worked out, and which studies are left out of them.

Registered
Jul 28, 2026
First posted
Jul 31, 2026
StartActual
May 1, 2026
Primary completionEst.
Dec 31, 2026was Oct 2026
CompletionEst.
Mar 31, 2027was Oct 2026
Last update
Oct 7, 2026
What these dates mean
Start.
When the first participant enrolled; while estimated, when the study expects to open to recruitment.
Primary completion.
When the last participant was examined or treated to collect data for the study's main question, its primary outcome. A study that stopped early gives the date its data collection stopped.
Completion.
When the last participant's last visit took place, for every outcome, secondary ones and side effects included: the end of the study. Often later than primary completion, sometimes the same day.
Actual and Est.
Actual means the registry says the date has happened. Est. (estimated) is a date the sponsor plans or expects. It can still move, and the sponsor is expected to change it to actual once it has passed.
Sites
4 sites added
Show 4 added (4 France)
  • Hôpital Ambroise Paré AP-HP · Boulogne-Billancourt, France
  • Hopital Tenon · Paris, France
  • Hôpital Cochin AP-HP · Paris, France
  • Hôpital Saint Antoine AP-HP · Paris, France
Oct 7, 2026
Primary completion
Oct 1, 2026→Dec 31, 2026
Oct 7, 2026
Study completion
Oct 1, 2026→Mar 31, 2027
Oct 7, 2026
Eligibility
Revised
Oct 7, 2026
Primary outcomes
1 edited, 1 added
What changed in the outcomes
  • EditedDomain-level performanceconcordance between LLM recommendations and the locked guidelines.tumour board recommendationsTime frame: Assessed once at central scoring, after data collection (~October 2026) → Assessed once at central scoring, after data collection (estimated January 2027)
  • AddedProportion of recommendations carrying serious harm potential (Tier 2)Time frame: Assessed once at central scoring, after data collection (estimated January 2027)
Oct 7, 2026
Secondary outcomes
1 edited, 4 time frame changed, 4 added, 2 removed, 2 description edited
What changed in the outcomes
  • EditedInter-tumour board domain-level recommendation concordanceconcordance (human baseline)Time frame: Up to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
  • AddedChance-corrected agreement (Gwet's AC1)Time frame: Assessed once at central scoring, after data collection (estimated January 2027)
  • AddedAgreement with at least one tumour boardTime frame: Assessed once at central scoring, after data collection (estimated January 2027)
  • AddedIntra-model consistencyTime frame: Assessed once at central scoring, after data collection (estimated January 2027)
  • AddedProportion of unusable model responsesTime frame: Assessed once at central scoring, after data collection (estimated January 2027)
  • RemovedProportion of recommendations carrying serious harm potential ( LLM and tumour boards)Time frame: Up to October 2026
  • RemovedDomain-level recommendation concordance between LLM and tumour-boardsTime frame: Up to October 2026
  • Time frameEquipoise rateUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
  • Time frameCompletenessUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
  • Time frameMissingnessUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
  • Time frameIntensity biasUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
  • DescriptionCompleteness: how it is measured was reworded
  • DescriptionEquipoise rate: how it is measured was reworded
Oct 7, 2026
Site contacts
Changed at 1 site. See the sites
Show the site
  • Hopital Européen Georges Pompidou · Paris, FranceContact j•••@aphp.fr · •••2834→a•••@aphp.fr · •••5651Contact added a•••@aphp.fr · •••3567
Oct 7, 2026
Show all 1 update
  1. Oct 7, 2026
    4 sites added
    Show 4 added (4 France)
    • Hôpital Ambroise Paré AP-HP · Boulogne-Billancourt, France
    • Hopital Tenon · Paris, France
    • Hôpital Cochin AP-HP · Paris, France
    • Hôpital Saint Antoine AP-HP · Paris, France
    Primary completion Oct 1, 2026→Dec 31, 2026
    Study completion Oct 1, 2026→Mar 31, 2027
    Eligibility Revised
    Primary outcomes 1 edited, 1 added
    What changed in the outcomes
    • EditedDomain-level performanceconcordance between LLM recommendations and the locked guidelines.tumour board recommendationsTime frame: Assessed once at central scoring, after data collection (~October 2026) → Assessed once at central scoring, after data collection (estimated January 2027)
    • AddedProportion of recommendations carrying serious harm potential (Tier 2)Time frame: Assessed once at central scoring, after data collection (estimated January 2027)
    Secondary outcomes 1 edited, 4 time frame changed, 4 added, 2 removed, 2 description edited
    What changed in the outcomes
    • EditedInter-tumour board domain-level recommendation concordanceconcordance (human baseline)Time frame: Up to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
    • AddedChance-corrected agreement (Gwet's AC1)Time frame: Assessed once at central scoring, after data collection (estimated January 2027)
    • AddedAgreement with at least one tumour boardTime frame: Assessed once at central scoring, after data collection (estimated January 2027)
    • AddedIntra-model consistencyTime frame: Assessed once at central scoring, after data collection (estimated January 2027)
    • AddedProportion of unusable model responsesTime frame: Assessed once at central scoring, after data collection (estimated January 2027)
    • RemovedProportion of recommendations carrying serious harm potential ( LLM and tumour boards)Time frame: Up to October 2026
    • RemovedDomain-level recommendation concordance between LLM and tumour-boardsTime frame: Up to October 2026
    • Time frameEquipoise rateUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
    • Time frameCompletenessUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
    • Time frameMissingnessUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
    • Time frameIntensity biasUp to October 2026 → Assessed once at central scoring, after data collection (estimated January 2027)
    • DescriptionCompleteness: how it is measured was reworded
    • DescriptionEquipoise rate: how it is measured was reworded
    Site contacts Changed at 1 site. See the sites
    Show the site
    • Hopital Européen Georges Pompidou · Paris, FranceContact j•••@aphp.fr · •••2834→a•••@aphp.fr · •••5651Contact added a•••@aphp.fr · •••3567
    + 4 other changes: description, arm descriptions, data sharing statement and references

From the registry record's own update history. This site started tracking changes on Sep 25, 2026; for anything earlier, see the record history on ClinicalTrials.gov ↗

10

Registry details

Key details

Study ID
NCT07739121
Responsible party
Sponsor
First posted
Jul 31, 2026
Start date
May 1, 2026
Primary completion
Dec 31, 2026 (estimated)
Completion
Mar 31, 2027 (estimated)
Last update
Oct 7, 2026

Study contacts

Jean-Emmanuel Bibault, MD PhD
Contact
jean-emmanuel.bibault@aphp.fr
01 56 09 34 06 ext. +33
Jérôme Lambert, MD PhD
Contact
jerome.lambert@u-paris.fr
0142499742 ext. +33

Oversight

Data monitoring committee
No
FDA-regulated drug
No
FDA-regulated device
No
View the source record on ClinicalTrials.gov ↗

Interested in this study?

Eligibility is decided by the study team. Share this record with your doctor or contact the team directly.

Contact study team

Follow this study

Get an email when the registry record changes — status, dates, results — or when someone posts here.

Sign in to follow

Discussion

Questions and observations about this study, from anyone following it. Not medical advice, and not a channel to the study team — their contact details are on the registry record.

Sign in to join the discussion. Reading takes no account; posting does. You choose a display name, and a pseudonym is the default.

Nothing here yet. If you are running this trial, taking part in it, or weighing whether to, this is the place to say so.

Start the discussion