Logo OckBench

Tokens are Not to Be Multiplied without Necessity

Zheng Du1*, Hao Kang1*, Song Han2, Tushar Krishna1, Ligeng Zhu3
*Equal contribution
1Georgia Institute of Technology 2MIT 3NVIDIA
arXiv GitHub

πŸ€—

 Dataset

πŸ†

 Leaderboard

Introduction

"Entities must not be multiplied beyond necessity." β€” The Principle of Ockham's Razor

Large Language Models (LLMs) like GPT-5, Gemini 3, and Claude serve as the frontier of automated intelligence, fueled by reasoning techniques like Chain of Thought (CoT). As the field embraces test-time compute scaling, models are increasingly trained to generate extensive token chains to tackle tougher problems. However, this massive inflation of decoding tokens introduces a critical bottleneck: solving just six problems in the International Olympiad in Informatics can now take over ten hours, and complex mathematical challenges frequently explode into millions of tokens.

While the community celebrates gains in reasoning capability, prevailing benchmarks like HELM and Chatbot Arena focus almost exclusively on output quality, ignoring this token efficiency crisis. In reality, many models consume vastly more tokens than necessary to reach the correct answer. Two models reaching the same accuracy can differ by more than 25Γ— in tokens generated β€” one answering in ~1,600 tokens where another spends ~42,000. As accuracy on standard tasks approaches saturation, tokens must be treated as a cost β€” not a free resource.

We introduce OckBench, the first model- and hardware-agnostic benchmark that jointly measures accuracy and token efficiency. Our key contributions include:

  • Per-Token Intelligence: We introduce a new evaluation dimension β€” a superior model must not only achieve high accuracy but do so with minimal token consumption
  • OckBench & OckScore: A benchmark with a novel Differentiation Filter that isolates tasks exposing the efficiency gap, paired with a unified metric that rewards high accuracy achieved with fewer tokens
  • The Overthinking Tax: We formally quantify how smaller models often incur paradoxically higher deployment costs due to excessive verbose reasoning chains
  • Optimization Pathways: We demonstrate that efficiency is tractable β€” both training-free model interpolation and difficulty-aware RL significantly improve OckScore

Through extensive evaluations on 49 frontier and open-source model settings, we find that top open-source models have nearly closed the accuracy gap but consume up to 26Γ— more tokens than commercial counterparts for comparable accuracy. Meanwhile, frontier commercial models are rapidly co-optimizing both dimensions, validating Per-Token Intelligence as the next key axis of LLM evaluation.

Accuracy vs. Token Consumption for all evaluated models
Accuracy vs. Average Tokens across 49 evaluated model settings. Models in the upper-left corner are ideal. The Pareto frontier is dominated by commercial models; open-source models cluster to the right.

Closed Models Own the Frontier

GPT-5.5 (medium) still leads at 82.2 OckScore, while Claude Opus 4.8 (xhigh) is second at 81.6 with 91.5% accuracy. The top ten settings are all GPT-5.x or Claude Opus.

Open vs. Closed Gap

DeepSeek-V4-Pro (high) reaches 84.0%, within two points of GPT-5.5 (medium), but uses 9.3Γ— more tokens (43,652 vs 4,692). Open weights are close on accuracy, not yet on efficiency.

The Overthinking Tax

DeepSeek-V4-Pro scores the same 84.0% at high and max effort, but max uses 1.9Γ— more tokens (81,408 vs 43,652), dropping OckScore from 67.20 to 61.87.

Leaderboard

Performance of various LLMs on OckBench (Selected) β€” 200 problems (100 math + 60 coding + 40 science). Models are ranked by OckScore = Accuracy βˆ’ 10 Γ— ln(AvgTokens / 10,000 + 1) β€” higher is better. Sampling follows each model's validated evaluation configuration, and math accuracy is graded by an LLM judge. MiniMax-M2.7 is the equal-weight pooled average of four InfiniAI requested effort-label runs (800 model-problem observations).

# Model Category Avg Tokens Accuracy (%) OckScore ↑

Benchmark Overview

OckBench provides comprehensive evaluation across multiple dimensions

Questions per Domain
200
Task Domains
3
Source Datasets
8+
Settings Evaluated
49

Benchmark Composition

OckBench aggregates tasks across three complementary reasoning domains. Rather than random sampling, we apply the Differentiation Filter: selecting problems where accuracy across models falls within 10%–90% (avoiding floor/ceiling effects) and token variance is maximized β€” isolating instances that reveal intrinsic efficiency differences.

  • Mathematics & Reasoning: GSM8K, AIME 2024/2025, OlympiadBench, MATH500, AMO-Bench, and the mathematics subset of Humanity's Last Exam β€” spanning grade-school arithmetic to competition-level number theory.
  • Software Engineering: A lightweight variant of MBPP and LiveCodeBench, assessing practical code generation and planning skills verified via unit test execution.
  • Scientific Reasoning: ScienceQA, MMLU (STEM subsets), and GPQA-Diamond, testing knowledge-constrained reasoning and concision under technical load.

Example Tasks

Sample problems from OckBench's math, coding, and science domains

These examples illustrate the types of problems where token efficiency varies significantly across models.

Math Problem (GSM8K)

Question: A store sells notebooks for $3 each. If you buy more than 10, you get a 20% discount on the total price. How much would it cost to buy 15 notebooks?

Domain: Mathematics  |  Source: GSM8K

$3 Γ— 15 Γ— 0.8 = $36 β€” a 3-second mental calculation. Yet some reasoning models spend 2,000+ tokens setting up formal equations, double-checking edge cases, and re-reading the problem before arriving at the obvious answer.

Math Problem (AIME)

Question: Find the number of ordered pairs (a,b) of integers such that aΒ² + bΒ² = 2024 and both a and b are positive.

Domain: Mathematics  |  Source: AIME 2024

Efficient models notice 2024 = 4 Γ— 506 and apply modular arithmetic to eliminate large search spaces in a few steps. Verbose models enumerate all 44Β² candidate pairs one by one β€” correct eventually, but at 10Γ— the token cost.

Coding Problem (MBPP)

Task: Write a function to find the longest common subsequence of two strings. For example, lcs("ABCDGH", "AEDFHR") should return 3 (the LCS is "ADH").

Domain: Coding  |  Source: MBPP variant

A clean DP solution is ~10 lines. But many models first write a recursive solution, identify the redundancy, rewrite with memoization, then pivot to bottom-up β€” producing correct code buried under 800 tokens of self-tutoring.

Science Problem (GPQA-Diamond)

Question: A molecule undergoes a photochemical reaction in which it absorbs a photon and transitions to an excited state. If the excited state has a lifetime of 10 ns, what is the natural linewidth (in Hz) of the corresponding spectral line?

Domain: Scientific Reasoning  |  Source: GPQA-Diamond

Efficient models recall Δν = 1/(2πτ) and plug in Ο„ = 10 ns for an instant answer. Overthinking models re-derive the time-energy uncertainty relation from first principles β€” impressively thorough, but it's a textbook formula lookup.

Key Findings

  • Closed models own the efficiency frontier: GPT-5.5 (medium) remains first at 82.15 OckScore, and Claude Opus 4.8 (xhigh) enters second at 81.61 with the highest accuracy in the table (91.5%). The top ten settings are all GPT-5.x or Claude Opus; the best open-weight setting ranks 16th.
  • Open-source accuracy is close, but efficiency still lags: DeepSeek-V4-Pro (high) reaches 84.0%, within two points of GPT-5.5 (medium), but spends 9.3Γ— more tokens (43,652 vs 4,692). The accuracy gap has nearly closed; the per-token-intelligence gap has not.
  • The Overthinking Tax: In the Qwen3.5 family, larger models are both smarter and leaner β€” Qwen3.5-9B burns 116,222 tokens for 21.5% accuracy (last place at OckScore βˆ’3.85), while Qwen3.5-397B-A17B reaches 67.5% using 4.4Γ— fewer tokens. Smaller models over-generate to compensate for lower capacity.
  • More effort is not monotonically better: Claude Opus 4.8 peaks at xhigh (91.5%, OckScore 81.61). Moving to max lowers accuracy to 90.0% while more than doubling tokens (16,876 β†’ 36,544), dropping OckScore to 74.62.
  • Open-weight leaders are tightly matched: DeepSeek-V4-Pro (high) reaches 84.0% at 43,652 tokens and 67.20 OckScore, while GLM-5.2 (max) reaches 83.5% at 41,234 tokens and 67.16 OckScore.
  • MiniMax-M2.7 is reported as a repeated-run average: its row pools minimal/low/medium/high requested labels to 66.875% accuracy, 35,538 average output tokens, and 51.72 OckScore. The labels were accepted by the gateway but are not presented as a verified reasoning-depth ladder.

Citation

If you find OckBench useful for your research, please cite our work 

@article{du2025ockbench,
  title={OckBench: Measuring the Efficiency of LLM Reasoning},
  author={Du, Zheng and Kang, Hao and Han, Song and Krishna, Tushar and Zhu, Ligeng},
  journal={arXiv preprint arXiv:2511.05722},
  year={2025}
}