OckBench: Measuring the Efficiency of LLM Reasoning

Introduction

"Entities must not be multiplied beyond necessity." — The Principle of Ockham's Razor

Large Language Models (LLMs) like GPT-5, Gemini 3, and Claude serve as the frontier of automated intelligence, fueled by reasoning techniques like Chain of Thought (CoT). As the field embraces test-time compute scaling, models are increasingly trained to generate extensive token chains to tackle tougher problems. However, this massive inflation of decoding tokens introduces a critical bottleneck: solving just six problems in the International Olympiad in Informatics can now take over ten hours, and complex mathematical challenges frequently explode into millions of tokens.

While the community celebrates gains in reasoning capability, prevailing benchmarks like HELM and Chatbot Arena focus almost exclusively on output quality, ignoring this token efficiency crisis. In reality, many models consume vastly more tokens than necessary to reach the correct answer. Models of identical size (7B) achieving similar accuracy can differ by over 3.4× in token consumption and 5.0× in end-to-end latency. As accuracy on standard tasks approaches saturation, tokens must be treated as a cost — not a free resource.

We introduce OckBench, the first model- and hardware-agnostic benchmark that jointly measures accuracy and token efficiency. Our key contributions include:

Per-Token Intelligence: We introduce a new evaluation dimension — a superior model must not only achieve high accuracy but do so with minimal token consumption
OckBench & OckScore: A benchmark with a novel Differentiation Filter that isolates tasks exposing the efficiency gap, paired with a unified metric that rewards high accuracy achieved with fewer tokens
The Overthinking Tax: We formally quantify how smaller models often incur paradoxically higher deployment costs due to excessive verbose reasoning chains
Optimization Pathways: We demonstrate that efficiency is tractable — both training-free model interpolation and difficulty-aware RL significantly improve OckScore

Through extensive evaluations on 47 frontier and open-source models, we find that top open-source models have nearly closed the accuracy gap but consume up to 5.1× more tokens than commercial counterparts for comparable accuracy. Meanwhile, frontier commercial models are rapidly co-optimizing both dimensions, validating Per-Token Intelligence as the next key axis of LLM evaluation.

Accuracy vs. Token Consumption for all 47 evaluated models — Accuracy vs. Average Tokens across 47 evaluated models. Models in the upper-left corner are ideal. The Pareto frontier is dominated by commercial models; open-source models cluster to the right.

7B models: AceReason-7B is 5x faster than DeepSeek-R1-7B at same parameter scale — **Same Scale, 5× Faster.** At identical 7B parameters, AceReason-7B is 5.0× faster than DeepSeek-R1-7B by using 3.4× fewer tokens.

Open vs Closed Source: DeepSeek-V3.2 uses 5.1x more tokens than GPT-5.2 at similar accuracy — **Open vs. Closed Gap.** DeepSeek-V3.2 matches GPT-5.2 in accuracy but consumes 5.1× more tokens — open-source lags in per-token intelligence.

The Overthinking Tax: larger Qwen3 models use fewer tokens than smaller ones — **The Overthinking Tax.** In the Qwen3 family, larger models are paradoxically *more* token-efficient — smaller models over-generate to compensate for lower capacity.

Leaderboard

Performance of various LLMs on OckBench-Math (200 questions). Models are ranked by OckScore = Accuracy − 10 × log(Tokens / 10,000) — higher is better. All models evaluated with single-shot prompts and greedy decoding (temperature = 0).

OckBench-Math

#	Model	License	Avg Tokens	Accuracy (%)	OckScore ↑
1	gemini-3.1-pro-preview Gemini	Commercial	24,765	75.5	70.09
2	gemini-3-pro-preview Gemini	Commercial	20,154	72.0	67.21
3	gemini-3-flash-preview Gemini	Commercial	36,212	71.0	64.35
4	Qwen3.5-397B-A17B-FP8 Qwen	Open-Source	33,994	68.5	62.07
5	gpt-5.2 (high) OpenAI	Commercial	19,541	66.0	61.30
6	gpt-5.2 (medium) OpenAI	Commercial	15,683	63.0	58.90
7	gpt-5-mini (high) OpenAI	Commercial	29,297	52.0	46.06
8	Kimi-K2-Thinking Kimi	Open-Source	41,746	52.5	45.36
9	gpt-5.2 (low) OpenAI	Commercial	5,003	41.5	39.74
10	DeepSeek-V3.2-Thinking DeepSeek	Open-Source	25,492	43.5	38.00
11	o4-mini (high) OpenAI	Commercial	25,677	43.5	37.98
12	gpt-5-mini (medium) OpenAI	Commercial	8,831	38.5	35.75
13	gemini-2.5-pro Gemini	Commercial	30,622	41.0	34.91
14	o4-mini (medium) OpenAI	Commercial	6,514	36.0	33.82
15	Qwen3-235B-A22B-Thinking-2507 Qwen	Open-Source	28,558	38.5	32.64
16	o3-mini (high) OpenAI	Commercial	21,623	37.0	32.00
17	Qwen3-30B-A3B-Thinking-2507 Qwen	Open-Source	30,513	37.0	30.92
18	Qwen3-32B-Thinking Qwen	Open-Source	19,098	31.0	26.36
19	o3-mini (medium) OpenAI	Commercial	7,251	27.5	25.13
20	Qwen3-235B-A22B-Instruct-2507 Qwen	Open-Source	7,707	27.0	24.52
21	DeepSeek-R1 DeepSeek	Open-Source	24,685	28.5	23.10
22	AReaL-boba-2-32B AReaL	Open-Source	23,327	27.0	21.77
23	AceReason-Nemotron-14B AceReason	Open-Source	19,424	25.0	20.31
24	AReaL-boba-2-14B AReaL	Open-Source	21,309	25.0	20.04
25	gpt-5-mini (low) OpenAI	Commercial	2,148	20.5	19.65
26	DeepSeek-V3.2-Instruct DeepSeek	Open-Source	9,379	21.0	18.13
27	o3-mini (low) OpenAI	Commercial	1,898	18.5	17.75
28	Kimi-K2-Instruct-0905 Kimi	Open-Source	3,823	18.5	17.09
29	AceReason-Nemotron-7B AceReason	Open-Source	12,315	20.5	17.01
30	o4-mini (low) OpenAI	Commercial	2,022	17.5	16.70
31	Qwen3-14B-Thinking Qwen	Open-Source	19,381	21.0	16.32
32	Qwen3-4B-Thinking-2507 Qwen	Open-Source	27,238	22.0	16.29
33	Qwen3-30B-A3B-Instruct-2507 Qwen	Open-Source	12,475	18.5	14.98
34	AReaL-boba-2-8B AReaL	Open-Source	24,127	20.0	14.67
35	DeepSeek-R1-Distill-Qwen-32B DeepSeek	Open-Source	12,895	15.5	11.90
36	DeepSeek-R1-Distill-Qwen-14B DeepSeek	Open-Source	13,211	15.0	11.34
37	Qwen3-8B-Thinking Qwen	Open-Source	24,207	16.5	11.16
38	Qwen3-4B-Instruct-2507 Qwen	Open-Source	11,859	13.5	10.10
39	Qwen3-14B-Instruct Qwen	Open-Source	8,205	10.0	7.40
40	DeepSeek-R1-Distill-Qwen-7B DeepSeek	Open-Source	41,415	14.5	7.39
41	gpt-4.1 OpenAI	Commercial	4,668	8.5	6.84
42	gemini-2.5-flash Gemini	Commercial	59,447	13.0	4.58
43	Qwen3-8B-Instruct Qwen	Open-Source	10,655	7.5	4.35
44	Qwen3-32B-Instruct Qwen	Open-Source	9,233	6.0	3.16
45	gpt-4o OpenAI	Commercial	947	3.5	3.11
46	gemini-2.5-flash-lite Gemini	Commercial	61,040	1.0	-7.52

#	Model	Category	#Tokens	Accuracy (%)	Reasoning Efficiency
1	Qwen3-235B-A22B (non-thinking) Alibaba	Open-Source	519.31	61.00	851.3
2	gpt-4.1 OpenAI	Commercial	639.80	74.50	858.1
3	Qwen3-32B (non-thinking) Alibaba	Open-Source	493.23	53.50	921.9
4	gpt-4o OpenAI	Commercial	573.54	61.50	932.6
5	Qwen3-30B-A3B (non-thinking) Alibaba	Open-Source	631.42	54.00	1,169.3
6	gpt-5 OpenAI	Commercial	1,584.41	96.50	1,641.8
7	Qwen3-14B (non-thinking) Alibaba	Open-Source	990.27	51.00	1,941.7
8	Qwen/Qwen3-4B Alibaba	Open-Source	893.80	39.50	2,263.8
9	Qwen3-4B (non-thinking) Alibaba	Open-Source	893.80	39.50	2,263.8
10	o3 OpenAI	Commercial	2,335.24	93.50	2,497.6
11	Qwen3-8B (non-thinking) Alibaba	Open-Source	1,073.26	41.50	2,586.2
13	Qwen3-30B-A3B-Thinking-2507 Alibaba	Open-Source	4,846.07	85.50	5,671.6
14	Qwen3-235B-A22B-Thinking-2507 Alibaba	Open-Source	5,927.84	92.00	6,443.3
15	Qwen3-32B (thinking) Alibaba	Open-Source	6,135.86	85.50	7,179.0
16	Qwen3-235B-A22B (thinking) Alibaba	Open-Source	6,545.30	85.00	7,700.4
17	gemini-3-pro-preview Google	Commercial	7,523.07	95.00	7,929.5
18	AReaL-boba-2-32B inclusionAI	Open-Source	6,901.34	85.00	8,119.2
19	Qwen3-30B-A3B (thinking) Alibaba	Open-Source	6,412.52	76.50	8,388.9
20	Qwen3-14B (thinking) Alibaba	Open-Source	6,850.90	77.00	8,897.3
21	AReal-boba-2-14B inclusionAI	Open-Source	7,519.10	79.50	9,461.4
22	AceReason-Nemotron-7B NVIDIA	Open-Source	7,082.74	67.50	10,419.6
23	AceReason-Nemotron-14B NVIDIA	Open-Source	8,515.95	76.50	11,128.7
24	gemini-2.5-flash Google	Commercial	9,884.73	85.00	11,629.1
25	gemini-2.5-pro Google	Commercial	9,808.19	79.00	12,417.9
26	AReal-boba-2-8B inclusionAI	Open-Source	10,521.25	70.50	14,959.2
27	Qwen3-8B (thinking) Alibaba	Open-Source	10,637.60	65.50	16,251.0
28	Qwen3-4B (thinking) Alibaba	Open-Source	13,639.89	58.00	23,517.1
30	gemini-2.5-flash-lite Google	Commercial	54,042.28	40.50	133,435.3

Benchmark Overview

OckBench provides comprehensive evaluation across multiple dimensions

Questions per Domain

200

Task Domains

3

Source Datasets

8+

Models Evaluated

47

Benchmark Composition

OckBench aggregates tasks across three complementary reasoning domains. Rather than random sampling, we apply the Differentiation Filter: selecting problems where accuracy across models falls within 10%–90% (avoiding floor/ceiling effects) and token variance is maximized — isolating instances that reveal intrinsic efficiency differences.

Mathematics & Reasoning: GSM8K, AIME 2024/2025, OlympiadBench, MATH500, AMO-Bench, and the mathematics subset of Humanity's Last Exam — spanning grade-school arithmetic to competition-level number theory.
Software Engineering: A lightweight variant of MBPP and LiveCodeBench, assessing practical code generation and planning skills verified via unit test execution.
Scientific Reasoning: ScienceQA, MMLU (STEM subsets), and GPQA-Diamond, testing knowledge-constrained reasoning and concision under technical load.

Example Tasks

Sample problems from OckBench-Math and OckBench-Coding

These examples illustrate the types of problems where token efficiency varies significantly across models.

Math Problem (GSM8K)

Question: A store sells notebooks for $3 each. If you buy more than 10, you get a 20% discount on the total price. How much would it cost to buy 15 notebooks?

Domain: Mathematics | Source: GSM8K

$3 × 15 × 0.8 = $36 — a 3-second mental calculation. Yet some reasoning models spend 2,000+ tokens setting up formal equations, double-checking edge cases, and re-reading the problem before arriving at the obvious answer.

Math Problem (AIME)

Question: Find the number of ordered pairs (a,b) of integers such that a² + b² = 2024 and both a and b are positive.

Domain: Mathematics | Source: AIME 2024

Efficient models notice 2024 = 4 × 506 and apply modular arithmetic to eliminate large search spaces in a few steps. Verbose models enumerate all 44² candidate pairs one by one — correct eventually, but at 10× the token cost.

Coding Problem (MBPP)

Task: Write a function to find the longest common subsequence of two strings. For example, lcs("ABCDGH", "AEDFHR") should return 3 (the LCS is "ADH").

Domain: Coding | Source: MBPP variant

A clean DP solution is ~10 lines. But many models first write a recursive solution, identify the redundancy, rewrite with memoization, then pivot to bottom-up — producing correct code buried under 800 tokens of self-tutoring.

Science Problem (GPQA-Diamond)

Question: A molecule undergoes a photochemical reaction in which it absorbs a photon and transitions to an excited state. If the excited state has a lifetime of 10 ns, what is the natural linewidth (in Hz) of the corresponding spectral line?

Domain: Scientific Reasoning | Source: GPQA-Diamond

Efficient models recall Δν = 1/(2πτ) and plug in τ = 10 ns for an instant answer. Overthinking models re-derive the time-energy uncertainty relation from first principles — impressively thorough, but it's a textbook formula lookup.

Key Findings

5× latency gap at the same scale: Among 7B models, AceReason-7B uses 12,315 tokens vs DeepSeek-R1-Distill-7B's 41,415 — a 3.4× token gap translating to a 5.0× end-to-end latency difference, despite identical parameter counts.
Open-source accuracy ≈ commercial, but efficiency lags: Kimi-K2-Thinking matches GPT-5-mini-high in accuracy (52.5% vs 52.0%) but consumes 43% more tokens (41,746 vs 29,297). Open-source models have closed the capability gap but trail significantly in per-token intelligence.
The Overthinking Tax: In the Qwen3 family, larger models are paradoxically more token-efficient — Qwen3-32B-Thinking uses 19,098 tokens at 31.0% accuracy, while Qwen3-4B-Thinking burns 27,238 tokens at just 22.0%. Smaller models compensate for lower capacity with verbose, inefficient reasoning chains.
Cheaper per-token ≠ cheaper per-query: DeepSeek-R1-Distill-7B generates 3.13× more tokens than 14B (41,415 vs 13,211) yet achieves lower accuracy (14.5% vs 15.0%). Despite costing 50% less per token, the 7B model is 57% more expensive per query in practice.
Gemini 3 Pro leads the frontier: Top OckScore of 70.09 with 75.5% accuracy at 24,765 avg tokens. Gemini 3 Pro resolves complex reasoning paths more succinctly than Gemini 3 Flash (36,212 tokens, OckScore 64.35), demonstrating that efficiency and accuracy are co-optimized at the frontier.
Frontier models are rapidly converging: Comparing successive model generations reveals simultaneous improvement in both accuracy and token efficiency — validating Per-Token Intelligence as the primary next optimization objective for the community.

Citation

If you find OckBench useful for your research, please cite our work

@article{du2025ockbench,
  title={OckBench: Measuring the Efficiency of LLM Reasoning},
  author={Du, Zheng and Kang, Hao and Han, Song and Krishna, Tushar and Zhu, Ligeng},
  journal={arXiv preprint arXiv:2511.05722},
  year={2025}
}

OckBench

Tokens are Not to Be Multiplied without Necessity