SLM vs LLM on FinQA

Luna still leads at 83%, while MiMo-V2.6-Flash reaches 80%.

Accuracy by model size and cost

Accuracy vs. API cost per 100 questions. X-axis is logarithmic.

Frontier

Colors use published total parameter counts: Large ≥100B, Medium 20–99B, Small <20B. Click a size to highlight that group; click it again to restore all points. Mixture-of-experts rows include active counts in the table. GPT-6 Luna has no published parameter count. GLM 5.3 Flash remains in the benchmark data but is omitted from this chart and the comparison tables because its endpoint required reasoning. Nemotron ran on OpenRouter's free NVIDIA endpoint; its chart cost is a paid-equivalent estimate.

Best overall

83%

GPT-6 Luna
0.37¢ / 100

Best non-Luna

80%

MiMo-V2.6-Flash
0.35¢ / 100

Best compact model

55%

Ministral 3 8B

Finance-specialized

77%

Ling 3.0 Flash Fin
0.21¢ / 100

Exact benchmark results


frontier OpenAI · unknown	Undisclosed	83%	0.37¢	1.04s	1.49s	0.64	16	off
frontier Xiaomi / MiMo · fp8	309B total (15B active)	80%	0.35¢	1.92s	5.16s	0.57	9	off
frontier inclusionAI / Ling · fp4	124B total (5.1B active)	77%	0.21¢	0.81s	1.26s	0.57	23	off
Qwen · fp4	27B	76%	0.53¢	1.39s	2.28s	0.54	22	off
NVIDIA / Nemotron · unknown	120B total (12B active)	74%	0.32¢*	0.84s	1.59s	0.60	23	off
DeepSeek · fp8	552B total (8B input / 16B output active)	72%	0.38¢	1.26s	2.52s	0.70	17	off
Google Gemma · fp4	31B	70%	0.35¢	0.84s	1.46s	0.51	29	off
Mistral · unknown	8B	55%	0.43¢	1.27s	2.07s	0.52	43	off
Qwen · fp4	9B	41%	0.33¢	1.19s	1.64s	0.53	58	off
IBM Granite · bf16	8B	25%	0.28¢	0.56s	0.77s	0.64	58	off

* Nemotron cost is a paid-equivalent estimate; measured benchmark spend on the free endpoint was 0.00¢. All displayed API costs are cents per 100 questions.

Confidence is still the weak link

At 50% local coverage, each model answers its 50 highest-confidence cases and sends the rest to Luna. This is descriptive on the same 100 questions, not a production threshold.

Local only	50% confidence route	Oracle error route	Conf. AUROC	Observed cutoff
80%	82%	88%	0.57	95%
77%	80%	89%	0.57	95%
76%	77%	90%	0.54	95%
74%	77%	89%	0.60	100%
72%	81%	89%	0.70	95%
70%	75%	90%	0.51	100%

Do not operationalize these cutoffs. Calibrate on separate development data and evaluate on a holdout. The oracle is only an upper bound on routing headroom.

Where models differ

Each row is a model; columns compare accuracy on addition, subtraction, multiplication, and division. Cell colors use a red–yellow–green scale.

Add	Subtract	Multiply	Divide
92%	82%	73%	84%
92%	71%	55%	81%
92%	71%	73%	70%
85%	76%	64%	70%
85%	71%	64%	65%
77%	65%	55%	70%
62%	71%	73%	65%
69%	65%	55%	32%
46%	53%	27%	22%
38%	29%	0%	14%

High-confidence errors from the strongest open models

MiMo-V2.6-Flash

AMAT/2013/page_37.pdf-1 · multiply · 3 step(s)what is the total return if 100000 are invested in applied materials in 2008 and sold in 2011?

Actual: 18210 · Model: 18.21% · Confidence: 95%

JPM/2010/page_273.pdf-2 · add · 3 step(s)in 2010 and 2009 , what was the total fair value in billions of assets segregated for the benefit of securities and futures brokerage customers?

Actual: 68.9 · Model: 34.7 · Confidence: 95%

AMAT/2018/page_33.pdf-2 · divide · 2 step(s)what is the roi for applied materials if the investment made on october 2013 was sold 5 years later?

Actual: 98.27% · Model: 243.16% · Confidence: 95%

Ling 3.0 Flash Fin

JPM/2010/page_273.pdf-2 · add · 3 step(s)in 2010 and 2009 , what was the total fair value in billions of assets segregated for the benefit of securities and futures brokerage customers?

Actual: 68.9 · Model: 2010: $34.7 billion; 2009: $34.2 billion · Confidence: 100%

AAL/2013/page_172.pdf-1 · divide · 1 step(s)what portion of the total bankruptcy settlement obligations are related to labor deemed claims?

Actual: 15.7% · Model: 849/5424 · Confidence: 100%

AAPL/2006/page_100.pdf-1 · multiply · 1 step(s)what was the 2006 tax expense?

Actual: $ 240.41 · Model: $829 · Confidence: 100%

Qwen3.8-27B

GS/2017/page_179.pdf-1 · divide · 2 step(s)what was the percentage change in dividends declared per common share between 2015 and 2016?

Actual: 2% · Model: 19.61% · Confidence: 100%

HWM/2015/page_173.pdf-1 · subtract · 2 step(s)how bigger were the interest and penalties concerning the interest income in the year 2015?

Actual: 300% · Model: 6 · Confidence: 100%

IP/2009/page_84.pdf-4 · add · 2 step(s)what was the sum of the temporary differences between 2007 and 2009 in billions

Actual: 9.8 · Model: 7.2 · Confidence: 100%

Nemotron 3 Super 120B-A12B

AOS/2010/page_18.pdf-1 · subtract · 5 step(s)what was the difference in the cumulative total return for a o smith corp and the s&p small cap 600 index for the five year period ended 12/31/10?

Actual: 55% · Model: 78.8% · Confidence: 100%

GS/2013/page_220.pdf-2 · subtract · 5 step(s)what was the difference in percentage cumulative total return for goldman sachs group inc . and the s&p 500 index for the five year period ending 12/31/13?

Actual: 12.16% · Model: 148.36 · Confidence: 100%

ETR/2017/page_143.pdf-1 · add · 4 step(s)what was the sum of the annual long-term debt maturities outstanding as of december 31 , 2017 , for the next five years

Actual: 4781374 · Model: 760000 · Confidence: 100%

Model diversity helps. Luna misses 17 questions; at least one other model gets 11 of those right. Only 6 questions defeat all 11 models.

Add	Subtract	Multiply	Divide
92%	82%	73%	84%
92%	71%	55%	81%
92%	71%	73%	70%
85%	76%	64%	70%
85%	71%	64%	65%
77%	65%	55%	70%
62%	71%	73%	65%
69%	65%	55%	32%
46%	53%	27%	22%
38%	29%	0%	14%

Add	Subtract	Multiply	Divide
92%	82%	73%	84%
92%	71%	55%	81%
92%	71%	73%	70%
85%	76%	64%	70%
85%	71%	64%	65%
77%	65%	55%	70%
62%	71%	73%	65%
69%	65%	55%	32%
46%	53%	27%	22%
38%	29%	0%	14%

Add	Subtract	Multiply	Divide
92%	82%	73%	84%
92%	71%	55%	81%
92%	71%	73%	70%
85%	76%	64%	70%
85%	71%	64%	65%
77%	65%	55%	70%
62%	71%	73%	65%
69%	65%	55%	32%
46%	53%	27%	22%
38%	29%	0%	14%