A small experiment with real support requests
95% confident
doesn’t mean 95% correct.
We gave LLMs 770 customer-support requests from BANKING77. For each one, the model chose an intent label and reported the probability that its answer was exactly right. Then we checked against the benchmark’s gold label.
Don’t trust the number. Test what the number means.#
Treat an AI confidence score as a signal to test, not a guarantee. Ask the model to express uncertainty, replay it on past cases where you already know the right answer, then choose the score and cutoff that remove the most human review while staying inside your acceptable error rate.
Don’t force the model to sound certain. Ask it to consider competing answers and report the winning probability. In this experiment, that simple prompt change made “95% confident” much safer.
Take examples whose correct outcome is already known. For each score, measure how often the AI was actually wrong. Never choose “95%” just because it sounds reassuring.
If you can access logprobs, test them as a ranking signal too. Choose a cutoff on historical cases, freeze it, and confirm that exact rule on untouched cases before removing review.
The practical recipe: better confidence prompt → historical gold cases → compare risk signals → choose a cutoff → confirm it on untouched cases → auto-pass only after it survives.
What changes a confidence score?#
Each experiment isolates a different source of confidence. Jump directly to the result you care about.
The models were more certain than they were right.#
Overall accuracy and average stated confidence should be close if confidence is well calibrated. Here they are not.
Each point is a confidence band; its size reflects how many requests fell in that band. Hover, tap, or keyboard-focus a point for the exact numbers. Points below the diagonal mean the model was overconfident.
Show a few high-confidence mistakes
How you ask for confidence changes the curve.#
We held everything else fixed: same GPT-5.6 Luna, same 770 requests, same labels. Only the confidence instruction changed. The quoted text below is the exact fragment appended to the prompt—copy it directly if you want to try the same technique.
Best practical result: prompting the model to keep uncertainty across competing labels reduced the overconfidence gap without changing the task. A literal “X% should mean X out of 100” instruction was also tested on all 770 cases, but barely improved calibration, so it is omitted from this chart.
Also tested: long-run frequency wording (little benefit)
Screen cheaply, then promote.
We tried five additional mechanisms on a balanced 154-case screen (2 per intent). The most promising new one—allocating probability across the top competing labels—was then run on all 770 cases.
| Screened instruction | Accuracy | Mean confidence | Brier ↓ | Error when ≥95% |
|---|
Brier score measures probabilistic accuracy: lower is better. Screening is only for choosing what to test fully; the headline conclusions use all 770 cases.
More context can make a model more confident—not more right.#
We replayed the same 770 requests with 0, 500, or 2,000 words of deliberately irrelevant neutral metadata. The extra text was explicitly marked incidental. If confidence is robust, irrelevant context should not systematically move it.
Paired stress test: every condition uses exactly the same cases. For Nano, the worsening Brier score is statistically clear under a paired bootstrap; for Luna, the confidence intervals include no change.
Use logprobs to rank risk—not as probabilities.#
On the same 770 requests, GPT-5.6 Luna produced both a token logprob and a prompted confidence score for the same answer. We asked two different questions: does the number itself mean what it says, and does it at least tell us which answers are safer?
Which should you use? If you need a number that roughly means “chance this answer is right,” the prompted confidence score is easier to interpret. If you need to decide which answers can skip human review, logprobs were more useful here—but only after choosing and validating a cutoff on historical cases.
Raw token probabilities were badly overconfident. Treat them as scores, not probabilities.
That ordering was better than stated confidence at separating correct from incorrect answers, even though the logprob values themselves were misleading.
Use reviewed historical cases to find a cutoff for your error budget. Freeze it and test it on new cases before removing human review.
Practical rule: confidence score for an interpretable probability; logprob for ranking risk. Never assume either is safe enough for automation until the exact cutoff has been tested on your own held-out cases.
Why this scale? Both axes plot −log10(1 − p), the “number of nines” of certainty: 90% = 1, 99% = 2, 99.9% = 3, and so on. Applying the same monotonic transform to reported certainty and actual accuracy makes the perfect-calibration reference a true diagonal again. Tick labels remain ordinary percentages. This chart pools the 770 development cases and 2,310 holdout cases after the prospective test; the frozen holdout result below remains separate and was not retuned.
This is the ranking chart. Every point is an actual score threshold you could deploy; ties are never split. That is more faithful than equal-count deciles for these data. Probability calibration is omitted because a monotonic rescaling cannot improve the ordering.
| Signal | Raw Brier ↓ | Ranks correctness · AUROC ↑ | Development coverage at ≤5% error ↑ |
|---|
Did the frozen rule survive new cases?#
Before running the remaining 2,310 BANKING77 test requests, we froze the raw-score cutoffs that had kept observed training error at or below 5%. We then applied those exact cutoffs unchanged—no retuning on the holdout.
How the one-token logprob experiment works
Each request gets a deterministic random mapping from the 77 intents to 77 one-character codes that are single tokens in both modern OpenAI tokenizers. Randomizing the mapping per case prevents one intent from always benefiting from an intrinsically easier code token. The response begins with that code, followed by the model's stated confidence.
Raw first-token probability is exp(logprob). The routing result above does not depend on probability calibration; it uses raw-score ordering and frozen raw-score cutoffs. We separately tried Platt scaling to translate both scores into approximate probabilities. Cross-fitted calibration looked promising on the 770 development cases, but on the 2,310 holdout calibrated logprob and calibrated stated confidence had very similar Brier scores (0.111 vs 0.113), so we do not treat calibration as the main result.
So what should we auto-pass?#
A confidence threshold is useful only if it tells us how much review we can skip—and what error rate remains among the cases we skip.
Everything below this score goes to a human reviewer.
Don’t choose 95 because it sounds safe.#
The right threshold is empirical. The same loop works for classification, extraction, RAG answers, matching, OCR, or any workflow where humans already approve or correct AI output.
Run the AI on examples where the final human-approved answer is already known.
For every confidence band and threshold, measure actual correctness—not how certain the model sounds.
Choose the threshold that meets your error SLA while removing the most human review. Confirm it on held-out cases.
Calibrate the prompt before calibrating the threshold.#
The 770-case run shows that confidence is not a fixed property of the model. The way you elicit the score changes which cases rise above a review threshold.
Test two or three confidence instructions on historical gold cases. Keep the one that best separates safe cases from risky ones.
Measure the actual error rate above each threshold and pick the amount of human review your SLA can safely remove.
The full BANKING77 test split has 3,080 cases. Larger samples matter when you need credible estimates around 1–2% residual error.
Simple enough to audit.#
BANKING77 contains fine-grained online-banking support intents. This run uses 770 requests: a balanced sample of 10 from each of all 77 intents. Both models saw the same request and the same list of allowed labels. Correctness is exact label match against the dataset—no LLM judge. Confidence is the model’s own 0–100 estimate of the probability that its chosen label exactly matches the gold label.
The logprob experiment uses a separate paired Chat Completions protocol: Luna emits a one-token class code followed by Top-two stated confidence, letting both signals refer to the same prediction. Probability calibration is evaluated out-of-fold; risk–coverage evaluates ranking.