BANKING77 · 77-case paired pilot · updated 8 October 2026
Decision models now own
the cost-quality frontier.
17 AI systems classified exactly the same banking-support requests. Clef is still the most accurate, Mercury Decide reaches 92.2% for free, and Liquid d1 matches Astra at a tiny fraction of its cost. But Perplexity Decider v1.1 barely beats Jev here. The architecture is promising; the model still matters enormously.
Highest observed accuracy96.1%Clef · 74 of 77 correct
Free92.2%Mercury Decide · 71 of 77 correct
Matches Astra88.3%Liquid d1 · ~324× cheaper here
There is no single winner; there is a new frontier. Mercury Decide is free and got 71/77 right. Clef-flash buys 2 more correct answers for $0.17 per 1,000 requests; full Clef buys one more again. Meanwhile Liquid d1 tied Astra at 68/77 while costing roughly 324× less in this run.
What happened?
We selected one request from each of BANKING77's 77 support categories. Every model saw the same request and the same 77 allowed labels. There was no LLM judge: a prediction counted only when it exactly matched the dataset's gold routing label.
The new models spread across the whole ranking. Mercury Decide landed third at 71/77; Liquid d1 tied Astra at 68/77; Kev 4B tied Gemini at 67/77; Perplexity Decider v1.1 managed 59/77, only one more than Jev. Clef and Clef-flash remain the top two. This is still only a 77-case pilot, so treat exact percentages as directional rather than production SLAs.
Model price is rarely the business bottleneck
The chart puts quality and inference cost on the same page. Free models occupy a separate left edge; paid models use a log scale because their costs span hundreds of times. Click any model for its details.
Up and left is better: more accurate and cheaper. “Free” is plotted separately at the left; paid models are log-scaled. The vertical axis is intentionally clipped to the observed range and does not start at zero.Swipe the chart horizontally →
The table below translates the same experiment into operational units. “Errors per 100” is simply what this small pilot observed; it is not a production SLA.
| Model | Accuracy | Errors / 100 | Cost / 1,000 | Median latency |
|---|---|---|---|---|
| Clef | 96.1% | 3.9 | $0.44 | 0.75s |
| Clef-flash | 94.8% | 5.2 | $0.17 | 0.49s |
| Mercury Decide | 92.2% | 7.8 | Free | 0.46s |
| Liquid d1 | 88.3% | 11.7 | $0.02 | 0.81s |
| GPT-6 Astra | 88.3% | 11.7 | $6.28 | 1.72s |
| Kev 4B | 87.0% | 13.0 | $0.04 | 1.05s |
| Gemini 3.8 Flash | 87.0% | 13.0 | $1.19 | 2.82s |
| GPT-5.6 Luna | 83.1% | 16.9 | $0.12 | 0.97s |
| GPT-5.6 Sol | 83.1% | 16.9 | $1.12 | 1.45s |
| Claude Fable 5.1 | 83.1% | 16.9 | $9.89 | 3.33s |
| DeepSeek V4.1 Flash | 79.2% | 20.8 | $0.08 | 0.49s |
| OpenAI Decisions | 79.2% | 20.8 | $0.10 | 0.31s |
| Claude Opus 5 | 79.2% | 20.8 | $4.86 | 2.31s |
| GPT-6 Luna | 77.9% | 22.1 | $0.06 | 1.07s |
| Claude Sonnet 5 | 77.9% | 22.1 | $1.95 | 2.02s |
| Perplexity Decider v1.1 27B | 76.6% | 23.4 | $0.02 | 0.72s |
| Jev | 75.3% | 24.7 | $0.07 | 0.39s |
Cost uses each system's measured token usage: OpenRouter's reported usage.cost for hosted models, current direct OpenAI token pricing for OpenAI Decisions, and Cloudflare's listed rates for Clef/Clef-flash. Mercury Decide is currently free on OpenRouter. Cloudflare includes a daily free Workers AI allocation, so its actual billed cost can also be lower. Development retries are excluded.
Business implication: for bounded classification, a frontier general model is no longer the obvious default. Start with a cheap decision model that sits on your own measured Pareto frontier—Mercury, d1, Clef-flash here—and pay upward only when the avoided errors are worth it.
Confidence is not a guarantee
A useful confidence score should behave like a probability. Clef-flash was 92.8% confident on average and 94.8% correct; Clef was 90.6% confident at 96.1% accuracy. Both were slightly underconfident rather than aggressively overconfident.
More importantly, their confidence ranked risk extremely well: Clef-flash's AUROC was 0.966 and Clef's was 0.946. Their binary Brier scores (0.033 and 0.035) were the best in this pilot. By contrast, Jev and OpenAI Decisions remained materially overconfident.
Business implication: never turn “confidence ≥ 95%” directly into “no human review.” Learn that threshold from historical cases with known answers, freeze it, and validate it on untouched cases.
What should a company do with decision models?
Benchmark the implementation, not the architecture label. The same broad idea spans 75.3% for Jev, 76.6% for Perplexity Decider v1.1, 88.3% for Liquid d1, 92.2% for Mercury Decide, and 96.1% for Clef on these exact cases.
Start with the cheapest model that meets the error budget. Mercury is free and only two cases behind Clef-flash. Liquid d1 exactly tied Astra on accuracy while costing roughly 324× less here. Kev 4B tied Gemini while costing roughly 34× less. Pay for Clef-flash or Clef only when those extra avoided errors are worth the premium.
OpenAI Decisions tells a different story: using GPT-6 Luna through the typed Decisions API produced 61/77 correct versus 60/77 through chat—essentially no quality change—but cut observed median latency from 1.07s to 0.31s, about 3.4× faster. Here, its value is structure and speed, not an accuracy leap.
Also note the dataset itself is imperfect for this label-name-only setup: 3 of 77 requests were missed by all 17 systems. Several depend on fine distinctions that are clearer with a routing guide than with label names alone. Production evaluation should include the real policy or taxonomy definitions users expect the model to follow.
Technical detail
See all 77 classification results
Each row is the actual customer request; each model column shows its predicted label. ✓ green cells are correct. ✕ red cells are wrong; hover them to see the expected label. Scroll horizontally to compare models.
| Customer request | Clef | Clef-flash | Mercury Decide | Liquid d1 | GPT-6 Astra | Kev 4B | Gemini 3.8 Flash | GPT-5.6 Luna | GPT-5.6 Sol | Claude Fable 5.1 | DeepSeek V4.1 Flash | OpenAI Decisions | Claude Opus 5 | GPT-6 Luna | Claude Sonnet 5 | Perplexity Decider v1.1 27B | Jev |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I requested a refund from a store but it hasn't arrived. | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up | Refund not showing up |
| I would like to get help from someone with activating my card. | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card | activate my card |
| How old do you have to be to be able to open an account? | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit | age limit |
| Why isn't my top up working using my saved American Express in ApplePay? | apple pay or google pay | apple pay or google pay | apple pay or google pay | apple pay or google pay | apple pay or google pay | top up failed | supported cards and currencies | apple pay or google pay | apple pay or google pay | apple pay or google pay | apple pay or google pay | top up failed | top up failed | topping up by card | top up failed | top up failed | top up failed |
| Which ATMs accept this card? | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support | atm support |
| What's the limit on automatic top-up? | automatic top up | automatic top up | automatic top up | automatic top up | automatic top up | automatic top up | automatic top up | automatic top up | automatic top up | automatic top up | top up limits | automatic top up | top up limits | automatic top up | automatic top up | top up limits | automatic top up |
| I just completed a bank transfer and the balance didn't update | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer | balance not updated after bank transfer |
| My cash deposit from a week ago hasn't shown up. I need the money - how quickly can you sort this out? | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit | balance not updated after bank transfer | balance not updated after cheque or cash deposit | balance not updated after cheque or cash deposit |
| Can I transfer money to a beneficiery? | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | transfer into account | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | transfer into account | transfer into account | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | beneficiary not allowed | transfer into account |
| I want to reverse my transaction from earlier | cancel transfer | cancel transfer | cancel transfer | cancel transfer | cancel transfer | cancel transfer | cancel transfer | cancel transfer | cancel transfer | cancel transfer | request refund | cancel transfer | request refund | request refund | cancel transfer | cancel transfer | reverted card payment |
| I need a new card, my current card is nearly expired. | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire | card about to expire |
| What places can I use my card? | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance | card acceptance |
| Is there a way to know when my card will arrive? | card arrival | card arrival | card arrival | card delivery estimate | card arrival | card arrival | card delivery estimate | card arrival | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate |
| Can I have my card delivered on a specific day? | card delivery estimate | card delivery estimate | card delivery estimate | card arrival | card delivery estimate | card arrival | card delivery estimate | card arrival | card arrival | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate | card delivery estimate |
| Can I put my old card back into the system? I just found it/ | card linking | card linking | card linking | card linking | card linking | card linking | card linking | card linking | card linking | card linking | card linking | activate my card | card linking | card linking | activate my card | activate my card | activate my card |
| Can someone assist me by explaining why my card is not working? | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working | card not working |
| I would appreciate that someone let me know when there is an extra charge for payments. I was checking the app earlier and saying a charge on one of my payments that was additional that no one made me aware of before. | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | card payment fee charged | transfer fee charged | card payment fee charged | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement |
| I'm not familiar with a card payment. | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised |
| I believe my card payment exchange rate is incorrect. | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate | card payment wrong exchange rate |
| I was at an ATM and it swallowed my card. | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed | card swallowed |
| Hi, I am calling about a recent transaction that has happened on my account. I recently went to the ATM, and tried to make a withdrawal on my account. I just happened to check my transaction slip and noticed a fee for my withdrawal. How can I resolve this, I didn't know fees were charged for this type of action. | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge | cash withdrawal charge |
| I didn't get cash from an ATM, but according to the app a transaction was made when i didn't make it. | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised | cash withdrawal not recognised |
| How do I update my PIN to a new number? | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin | change pin |
| If I feel someone has my card information, can I get a new card? | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card | compromised card |
| When I tried to pay for my bus ride, the contactless payment wouldn't work. How can I prevent this in the future? | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working | contactless not working |
| Where can I find your locations? | country support | country support | country support | atm support | country support | order physical card | country support | atm support | country support | country support | country support | country support | country support | country support | country support | atm support | atm support |
| My latest payment was declined, I was told everything was back to working order. What happened? | declined card payment | reverted card payment | reverted card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment |
| The ATM is declining my card. I tried at two different ATMs. Can you tell me if there's anything wrong with my account? | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal | declined cash withdrawal |
| Good morning. I tried to make a purchase with my credit card last night and again this morning. Both times it was declined. Can you investigate? | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment |
| I am seeing a weird payment showing up that I know I did not make, how can I get it cancelled? | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised | card payment not recognised |
| How can i get multiple disposble cards. | disposable card limits | disposable card limits | disposable card limits | get disposable virtual card | disposable card limits | disposable card limits | disposable card limits | get disposable virtual card | disposable card limits | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | getting spare card | get disposable virtual card | get disposable virtual card |
| Can I change my information? | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details | edit personal details |
| I will need to exchange currencies frequently, will I be able to get a discount? | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge | exchange charge |
| From where are you getting your exchange rates? | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | card payment wrong exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate | exchange rate |
| I think I want AUD and not GBP, can you help? | exchange via app | exchange via app | exchange via app | exchange via app | exchange via app | exchange via app | exchange via app | fiat currency support | fiat currency support | fiat currency support | exchange rate | exchange via app | fiat currency support | exchange via app | fiat currency support | exchange via app | fiat currency support |
| Why does my statement have these extra charges? | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement | extra charge on statement |
| I am trying to make a transfer and am unsuccessful, can you please tell me why? | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer | failed transfer |
| Do you accept exchanges to EUR? | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | fiat currency support | exchange rate | fiat currency support | exchange via app | exchange via app | fiat currency support | fiat currency support |
| Where's the best place to get a disposable virtual card? | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card | get disposable virtual card |
| I cannot locate the card PIN. | get physical card | get physical card | get physical card | get physical card | change pin | get physical card | change pin | change pin | change pin | change pin | passcode forgotten | change pin | change pin | change pin | passcode forgotten | change pin | change pin |
| Am I able to get a second card? | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card | getting spare card |
| what is the virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card | getting virtual card |
| Somehow I am missing my card. What should I do? | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card | lost or stolen card |
| I believe I left my smartphone at the hotel I was staying at. | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone | lost or stolen phone |
| Please tell me how to get a physical card. | order physical card | order physical card | get physical card | order physical card | get physical card | order physical card | order physical card | get physical card | get physical card | get physical card | get physical card | get physical card | get physical card | get physical card | order physical card | get physical card | get physical card |
| I need to reset my passcode. How do I do it? | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten | passcode forgotten |
| I bought some things this morning but the payment shows that it is pending | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment | pending card payment |
| Cash withdrawal still pending? | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal | pending cash withdrawal |
| Is there a reason my top up hasn't gone through | pending top up | pending top up | pending top up | pending top up | top up failed | pending top up | top up failed | pending top up | top up failed | top up failed | top up failed | pending top up | top up failed | top up failed | top up failed | pending top up | top up failed |
| Why hasn't my transfer gone through yet? | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | balance not updated after bank transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer | pending transfer |
| Will you reinstate my PIN? | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | change pin | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked | pin blocked |
| Do you know if I can use this for my salary? | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money | receiving money |
| I want to reverse a purchase. Can I cancel it? | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | request refund | reverted card payment | reverted card payment |
| I think my card payment has been stopped | reverted card payment | reverted card payment | reverted card payment | reverted card payment | declined card payment | reverted card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment | declined card payment |
| can you tell me what cards and currencies you take? | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies |
| Please delete my account, this company is not working for me! | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account | terminate account |
| What'll it charge me for a SEPA transfer? | top up by bank transfer charge | top up by bank transfer charge | top up by bank transfer charge | top up by bank transfer charge | top up by bank transfer charge | top up by bank transfer charge | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged |
| What are the fees for top ups? | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge | top up by card charge |
| Will you accept a cheque to top up my account? | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque | top up by cash or cheque |
| I'm not sure why my top up request was denied | top up failed | top up failed | top up failed | top up reverted | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed | top up failed |
| Tell me if there are any top-up limits? | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits | top up limits |
| has my top-up has been cancelled | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted | top up reverted |
| How can I top up my account with a card? | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card | topping up by card |
| I think something may have happened that caused a charge to show up twice. | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice | transaction charged twice |
| Why am I seeing a transfer fee? | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged | transfer fee charged |
| I would like to transfer some money from my other bank account into this one. | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account | transfer into account |
| Is there a reason why my transaction is taking so long? | transfer not received by recipient | transfer not received by recipient | pending transfer | pending transfer | pending card payment | pending transfer | transfer timing | pending transfer | pending transfer | pending card payment | transfer timing | transfer timing | pending transfer | pending transfer | pending transfer | transfer timing | transfer timing |
| Do transfers from Europe take longer? | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing | transfer timing |
| How do I show this stupid system that this is really my identity? | unable to verify identity | unable to verify identity | unable to verify identity | unable to verify identity | verify my identity | unable to verify identity | unable to verify identity | verify my identity | verify my identity | verify my identity | unable to verify identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | unable to verify identity |
| Let me know the steps for the identity checks | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | verify my identity | why verify identity | verify my identity | verify my identity | verify my identity |
| I have to verify the source of my funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds | verify source of funds |
| Why does the top-up need verification? | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up | verify top up |
| virtual card is not working for me | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working | virtual card not working |
| Am I allowed to use any card to make a payment? | card acceptance | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | supported cards and currencies | card acceptance | card acceptance | supported cards and currencies | supported cards and currencies | card acceptance | supported cards and currencies | card acceptance | card acceptance | card acceptance | supported cards and currencies |
| Why do you require so many identity details? | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity | why verify identity |
| The amount of cash I received was incorrect | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received | wrong amount of cash received |
| In receiving cash, the wrong exchange rate was used for my transaction. | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | wrong exchange rate for cash withdrawal | card payment wrong exchange rate | wrong exchange rate for cash withdrawal |
Calibration and ranking metrics
Brier and ECE are lower-is-better probability-quality measures. AUROC asks only whether confidence ranks correct cases above incorrect ones; higher is better.
| Model | Accuracy | Mean confidence | Gap | Brier ↓ | ECE ↓ | AUROC ↑ |
|---|---|---|---|---|---|---|
| Clef | 96.1% | 90.6% | -5.5 pp | 0.035 | 0.078 | 0.946 |
| Clef-flash | 94.8% | 92.8% | -2.0 pp | 0.033 | 0.069 | 0.966 |
| Mercury Decide | 92.2% | 97.5% | +5.2 pp | 0.060 | 0.066 | 0.838 |
| Liquid d1 | 88.3% | 87.4% | -0.9 pp | 0.089 | 0.065 | 0.824 |
| GPT-6 Astra | 88.3% | 94.0% | +5.7 pp | 0.088 | 0.057 | 0.840 |
| Kev 4B | 87.0% | 74.8% | -12.2 pp | 0.123 | 0.128 | 0.809 |
| Gemini 3.8 Flash | 87.0% | 94.0% | +7.0 pp | 0.108 | 0.089 | 0.766 |
| GPT-5.6 Luna | 83.1% | 96.4% | +13.3 pp | 0.140 | 0.136 | 0.754 |
| GPT-5.6 Sol | 83.1% | 93.8% | +10.7 pp | 0.139 | 0.127 | 0.733 |
| Claude Fable 5.1 | 83.1% | 83.6% | +0.4 pp | 0.120 | 0.076 | 0.786 |
| DeepSeek V4.1 Flash | 79.2% | 84.5% | +5.2 pp | 0.158 | 0.095 | 0.666 |
| OpenAI Decisions | 79.2% | 85.5% | +6.3 pp | 0.176 | 0.154 | 0.719 |
| Claude Opus 5 | 79.2% | 79.9% | +0.7 pp | 0.124 | 0.045 | 0.843 |
| GPT-6 Luna | 77.9% | 92.0% | +14.1 pp | 0.190 | 0.183 | 0.739 |
| Claude Sonnet 5 | 77.9% | 81.6% | +3.6 pp | 0.152 | 0.131 | 0.748 |
| Perplexity Decider v1.1 27B | 76.6% | 86.3% | +9.7 pp | 0.139 | 0.100 | 0.850 |
| Jev | 75.3% | 87.4% | +12.1 pp | 0.171 | 0.138 | 0.811 |
Methodology and reproducibility
One deterministic frozen case per BANKING77 class, 77 total. Ordinary chat models receive the 77 labels in the prompt and return one label plus a 0–100 probability of being exactly correct. Decision-model systems receive the same 77 label values plus the same humanized label descriptions through their native typed-choice interface: Jev, Perplexity Decider v1.1, Liquid d1, Kev 4B, and Mercury Decide via OpenRouter Decisions; GPT-6 Luna via OpenAI's direct Decisions API; and Clef/Clef-flash via Cloudflare Workers AI. For all decision models, the chosen label's probability is the comparable confidence score.
Latency is observed end-to-end from this machine, not model-only latency; provider and network paths differ. Reasoning was disabled for GPT-5.6 Luna, GPT-6 Luna, GPT-5.6 Sol and DeepSeek; low reasoning was requested for Astra, Opus, Fable, Gemini and Sonnet.
The runner is resumable: an existing (model, case) result is skipped. Re-running uv run run.py --dry-run should report zero pending calls.
Current interface/pricing references: OpenAI Decisions API · OpenAI pricing · Clef · Clef-flash · OpenRouter Decisions ranking.
How the 8 October additions were selected
We started from OpenRouter's public Decisions ranking and added popular models that could express this exact 77-way Choice task without changing the benchmark: Perplexity Decider v1.1, Liquid d1, Kev 4B, and Mercury Decide. Their successful calls cost $0.00569 in total, including preflights.
Span-01 Lite was excluded after a 77-way Choice preflight returned HTTP 400; its native interface is specialized for Noul/behavior scoring. Solar Decide was excluded after HTTP 422 because its Choice endpoint supports at most 26 options. Tev1 was not run because its Choice formulation is limited to about 20 options. Splitting BANKING77 into a hierarchy would make those models runnable, but it would no longer be the same experiment.
Files
Frozen 77 cases · Raw results · Computed summary · Model configuration · Prompt · Runner · Analysis