SCT-Bench Demo

Evaluating Uncertainty SCT-Bench

Measuring how AI adjusts diagnostic thinking when new evidence arrives -- the way real clinicians reason under uncertainty

Based on Script Concordance Testing -- a decades-old medical assessment tool now applied to AI

750 Questions / 10 International Datasets / 9 Previously Unreleased / NEJM AI

Clinical Scenario

Adelaide SCT -- University of Adelaide, Australia

SCTs test three dimensions of clinical reasoning:

Dx Diagnostic Hypotheses Ix Investigative Actions Rx Management Decisions

If you were thinking of...

And then you find...

Does this new information make the hypothesis more or less likely?

-2 Much less
likely

-1 Slightly less
likely

0 Neither more
nor less

+1 Slightly more
likely

+2 Much more
likely

Expert Panel Scoring

Grounded in Script Theory

Clinicians organize knowledge into "illness scripts" -- mental networks linking diseases, symptoms, and actions. SCTs measure how well new information is integrated into these scripts, not whether a fixed answer is recalled.

There is no single right answer. Experts disagree -- and that's the point. The most common response scores 1.0; partial credit for others.

Modal expert answer: +2 (Much more likely) receives full credit. Selecting 0 earns 0.8 points because 80% of that weight of experts chose it.

How Models Score Against Expert Panels

Reasoning Model (o1-preview)

-2 Much less likely -- overly decisive

Expert Concordance Score

0.0

GPT-4o

+1 Slightly more likely -- matching expert mode

Expert Concordance Score

1.0

Reasoning-optimized models snap to extremes while expert clinicians remain nuanced. o1-preview chose -2 (0% of experts agreed), scoring zero.

First of its kind The first medical benchmark where o1-preview underperforms GPT-4o -- inverting the pattern seen on every other medical evaluation.

The Overconfidence Problem

Expert Clinicians

Reasoning-Tuned Models

The Chain-of-Thought Paradox

MedQA (MCQ)

86.4% → 93.0%

SCT-Bench

77.7% → 74.7%

Chain-of-thought prompting improves multiple-choice scores but decreases clinical reasoning under uncertainty. Explicit reasoning leads to overconfidence.

✖ Not multiple choice ✖ No memorizable answers ✓ 9 of 10 datasets never previously published

Unlike saturated benchmarks, SCT-Bench resists data contamination. Scoring against expert panels means memorizing questions provides no advantage.

The Expertise Gradient

Models can pass the test. They can't yet think like a doctor.

SCT-Bench Leaderboard

Measuring clinical reasoning under uncertainty across frontier models

#	Model	Score	Performance

Published in NEJM AI -- McCoy et al. "Assessment of Large Language Models in Clinical Reasoning"