Jev is very cheap.
Cheap is not the hard part.

What happened?

We selected one request from each of BANKING77's 77 support categories. Every model saw the same request and the same 77 allowed labels. There was no LLM judge: a prediction counted only when it exactly matched the dataset's gold routing label.

Do not over-read the top two. Astra got 68 cases right; Gemini got 67. They disagreed in only five decisive cases, and this 77-case pilot cannot establish that one is generally better. It can establish that both were substantially stronger than Jev in this sample.

Model price is rarely the business bottleneck

The chart puts quality and inference cost on the same page. Cost uses a log scale because the models span more than two orders of magnitude. Click any model for its details.

The table below translates the same experiment into operational units. “Errors per 100” is simply what this small pilot observed; it is not a production SLA.

Costs are the OpenRouter usage.cost reported for the 77 successful calls, extrapolated to 1,000 requests. They include each model's actual token usage in this run, including reasoning tokens where reported. Initial development retries are excluded.

Confidence is not a guarantee

A useful confidence score should behave like a probability: decisions called “90% confident” should be right about 90% of the time. Jev was 87.4% confident on average but only 75.3% correct. Luna showed a similar gap. Opus and Fable looked better calibrated in aggregate, but 77 cases is too small to certify production thresholds.

Jev's confidence was still useful for ranking risk: its AUROC was 0.811. That means the score contains signal about which cases are dangerous—even though the number itself is too optimistic.

What should a company do with Jev?

Use it as a specialist, not as a frontier-model replacement. Jev is attractive when you need huge volumes of bounded, typed decisions, latency matters, and mistakes are cheap or easily caught downstream. Its median response here was 0.39s.

For higher-stakes classification, this pilot points instead toward a strong general model—especially Gemini 3.8 Flash or Astra—because the absolute inference cost is still tiny. A sensible architecture may be a cheap first pass plus escalation, but the escalation rule should be learned from a larger held-out benchmark, not invented from this pilot.

Also note the dataset itself is imperfect for this prompt-only setup: 6 of 77 requests were missed by all nine models. Several depend on fine distinctions that are clearer with a routing guide than with label names alone. Production evaluation should include the real policy or taxonomy definitions users expect the model to follow.

Technical detail