OckBench
"Entities must not be multiplied beyond necessity." β The Principle of Ockham's Razor
Large Language Models (LLMs) like GPT-5, Gemini 3, and Claude serve as the frontier of automated intelligence, fueled by reasoning techniques like Chain of Thought (CoT). As the field embraces test-time compute scaling, models are increasingly trained to generate extensive token chains to tackle tougher problems. However, this massive inflation of decoding tokens introduces a critical bottleneck: solving just six problems in the International Olympiad in Informatics can now take over ten hours, and complex mathematical challenges frequently explode into millions of tokens.
While the community celebrates gains in reasoning capability, prevailing benchmarks like HELM and Chatbot Arena focus almost exclusively on output quality, ignoring this token efficiency crisis. In reality, many models consume vastly more tokens than necessary to reach the correct answer. Two models reaching the same accuracy can differ by more than 25Γ in tokens generated β one answering in ~1,600 tokens where another spends ~42,000. As accuracy on standard tasks approaches saturation, tokens must be treated as a cost β not a free resource.
We introduce OckBench, the first model- and hardware-agnostic benchmark that jointly measures accuracy and token efficiency. Our key contributions include:
Through extensive evaluations on 37 frontier and open-source models, we find that top open-source models have nearly closed the accuracy gap but consume up to 26Γ more tokens than commercial counterparts for comparable accuracy. Meanwhile, frontier commercial models are rapidly co-optimizing both dimensions, validating Per-Token Intelligence as the next key axis of LLM evaluation.
GPT-5.5 (medium) tops OckBench at an OckScore of 82.2 β 86% accuracy in just 4,692 tokens. The top eight settings are all GPT-5.x or Claude Opus; no open-weight model reaches the top ten.
Kimi-K2.6 matches GPT-5.5 (low) at 75.0% accuracy but spends 26Γ more tokens (42,243 vs 1,603). Open weights have closed the accuracy gap β not the per-token-intelligence gap.
In the Qwen3.5 family, the 9B model burns 116,222 tokens for 21.5% accuracy, while the 397B model scores 67.5% using 4.4Γ fewer tokens. Bigger models are both smarter and leaner.
Performance of various LLMs on OckBench (Selected) β 200 problems (100 math + 60 coding + 40 science). Models are ranked by OckScore = Accuracy β 10 Γ ln(AvgTokens / 10,000 + 1) β higher is better. All models evaluated single-shot with greedy decoding (temperature = 0); math accuracy is graded by an LLM judge.
| # | Model | Category | Avg Tokens | Accuracy (%) | OckScore β |
|---|
OckBench provides comprehensive evaluation across multiple dimensions
OckBench aggregates tasks across three complementary reasoning domains. Rather than random sampling, we apply the Differentiation Filter: selecting problems where accuracy across models falls within 10%β90% (avoiding floor/ceiling effects) and token variance is maximized β isolating instances that reveal intrinsic efficiency differences.
Sample problems from OckBench's math, coding, and science domains
These examples illustrate the types of problems where token efficiency varies significantly across models.
Question: A store sells notebooks for $3 each. If you buy more than 10, you get a 20% discount on the total price. How much would it cost to buy 15 notebooks?
Domain: Mathematics | Source: GSM8K
$3 Γ 15 Γ 0.8 = $36 β a 3-second mental calculation. Yet some reasoning models spend 2,000+ tokens setting up formal equations, double-checking edge cases, and re-reading the problem before arriving at the obvious answer.
Question: Find the number of ordered pairs (a,b) of integers such that aΒ² + bΒ² = 2024 and both a and b are positive.
Domain: Mathematics | Source: AIME 2024
Efficient models notice 2024 = 4 Γ 506 and apply modular arithmetic to eliminate large search spaces in a few steps. Verbose models enumerate all 44Β² candidate pairs one by one β correct eventually, but at 10Γ the token cost.
Task: Write a function to find the longest common subsequence of two strings. For example, lcs("ABCDGH", "AEDFHR") should return 3 (the LCS is "ADH").
Domain: Coding | Source: MBPP variant
A clean DP solution is ~10 lines. But many models first write a recursive solution, identify the redundancy, rewrite with memoization, then pivot to bottom-up β producing correct code buried under 800 tokens of self-tutoring.
Question: A molecule undergoes a photochemical reaction in which it absorbs a photon and transitions to an excited state. If the excited state has a lifetime of 10 ns, what is the natural linewidth (in Hz) of the corresponding spectral line?
Domain: Scientific Reasoning | Source: GPQA-Diamond
Efficient models recall ΞΞ½ = 1/(2ΟΟ) and plug in Ο = 10 ns for an instant answer. Overthinking models re-derive the time-energy uncertainty relation from first principles β impressively thorough, but it's a textbook formula lookup.
If you find OckBench useful for your research, please cite our work
@article{du2025ockbench,
title={OckBench: Measuring the Efficiency of LLM Reasoning},
author={Du, Zheng and Kang, Hao and Han, Song and Krishna, Tushar and Zhu, Ligeng},
journal={arXiv preprint arXiv:2511.05722},
year={2025}
}