A small experiment with real support requests
95% confident
doesn’t mean 95% correct.
We gave LLMs 770 customer-support requests from BANKING77. For each one, the model chose an intent label and reported the probability that its answer was exactly right. Then we checked against the benchmark’s gold label.
The models were more certain than they were right.
Overall accuracy and average stated confidence should be close if confidence is well calibrated. Here they are not.
Each point is a confidence band; its size reflects how many requests fell in that band. Hover, tap, or keyboard-focus a point for the exact numbers. Points below the diagonal mean the model was overconfident.
Show a few high-confidence mistakes
How you ask for confidence changes the curve.
We held everything else fixed: same GPT-5.6 Luna, same 770 requests, same labels. Only the confidence instruction changed. The quoted text below is the exact fragment appended to the prompt—copy it directly if you want to try the same technique.
Best practical result: prompting the model to keep uncertainty across competing labels reduced the overconfidence gap without changing the task. A literal “X% should mean X out of 100” instruction was also tested on all 770 cases, but barely improved calibration, so it is omitted from this chart.
Also tested: long-run frequency wording (little benefit)
Screen cheaply, then promote.
We tried five additional mechanisms on a balanced 154-case screen (2 per intent). The most promising new one—allocating probability across the top competing labels—was then run on all 770 cases.
| Screened instruction | Accuracy | Mean confidence | Brier ↓ | Error when ≥95% |
|---|
Brier score measures probabilistic accuracy: lower is better. Screening is only for choosing what to test fully; the headline conclusions use all 770 cases.
More context can make a model more confident—not more right.
We replayed the same 770 requests with 0, 500, or 2,000 words of deliberately irrelevant neutral metadata. The extra text was explicitly marked incidental. If confidence is robust, irrelevant context should not systematically move it.
Paired stress test: every condition uses exactly the same cases. For Nano, the worsening Brier score is statistically clear under a paired bootstrap; for Luna, the confidence intervals include no change.
So what should we auto-pass?
A confidence threshold is useful only if it tells us how much review we can skip—and what error rate remains among the cases we skip.
Everything below this score goes to a human reviewer.
Don’t choose 95 because it sounds safe.
The right threshold is empirical. The same loop works for classification, extraction, RAG answers, matching, OCR, or any workflow where humans already approve or correct AI output.
Run the AI on examples where the final human-approved answer is already known.
For every confidence band and threshold, measure actual correctness—not how certain the model sounds.
Choose the threshold that meets your error SLA while removing the most human review. Confirm it on held-out cases.
Calibrate the prompt before calibrating the threshold.
The 770-case run shows that confidence is not a fixed property of the model. The way you elicit the score changes which cases rise above a review threshold.
Test two or three confidence instructions on historical gold cases. Keep the one that best separates safe cases from risky ones.
Measure the actual error rate above each threshold and pick the amount of human review your SLA can safely remove.
The full BANKING77 test split has 3,080 cases. Larger samples matter when you need credible estimates around 1–2% residual error.
Simple enough to audit.
BANKING77 contains fine-grained online-banking support intents. This run uses 770 requests: a balanced sample of 10 from each of all 77 intents. Both models saw the same request and the same list of allowed labels. Correctness is exact label match against the dataset—no LLM judge. Confidence is the model’s own 0–100 estimate of the probability that its chosen label exactly matches the gold label.