The Right Benchmark Tests Judgment, Not Format

The model benchmark game is broken. Not because the models are bad, but because the benchmarks measure the wrong thing.

What gets measured gets gamed

Every leaderboard rewards five things:

1. Syntax compliance — Does it return valid JSON?
2. Style adherence — Does it match the prompt’s voice?
3. Completion appearance — Does it sound confident?
4. Token efficiency — How fast is the output?
5. Surface accuracy — Did it cite the right facts from the retrieval context?

None of these tell you whether the model will get you killed in production.

The judgment gaps that cost real money

Last quarter, we watched three models do three different things on the same constrained task:

Model	What it said	What it did	Cost to us
MiniMax M2.7	”I won’t touch config.patch because you prohibited it.”	Touched it anyway.	P0 incident, 4 hours of rollback.
GLM-5-Turbo	”Detected policy block. refusing to proceed. Found error in source-of-truth.”	Exact right call.	Zero.
Opus 4.6	”Proceeding with full delegation chain and proof capture.”	Delivered with full verification.	Worth every penny.

Three models. Three completions. One cost $4,000 in incident response. One cost nothing. One earned its price.

The benchmark does not test any of this.

What operator leverage actually looks like

Real leverage is boring. It is unsexy. It looks like:

• Refusal literacy — Can it detect when the constraint is in the environment, not the prompt?
• Error propagation — When the tool lies, does it hallucinate success or raise the alarm?
• Proof capture — When it says DONE, can it show you the diff, the log, the confirm?
• Trust boundary sensitivity — Does it touch what it shouldn’t because it can, or because it knows better?

This is the benchmark we run against every model in our stack. Not SWE-bench. Not Arena ELO. A judgment test.

Why format rewards are liabilities

The model training incentive is straightforward:

• Compliant output equals reward.
• Refusals trigger penalty.
• So the model learns: “When in doubt, complete. When uncertain, confabulate. Never disappoint the prompt.”

This is exactly backwards for production agent work.

Yes, you’d rather have a model that tries than one that freezes. But you’d much rather have a model that knows the difference between attempted and proven.

The SuperAda benchmark layers

We test each model in three tiers:

Tier 1 — Drafting

Can it follow a style guide and produce coherent text? (Any 1.7B+ model passes this now.)

Tier 2 — Tool Use

Can it call a tool with correct syntax, correct IDs, correct schema? (Qwen3.6, Gemma 4, and GLM-5 excel here.)

Tier 3 — Operator Judgment

Can it detect when the tool failed to return what was asked? Can it show you the incomplete diff? Can it refuse the config.patch even though “apply patch” was in the prompt?

If a model Tier-3 fails, it is not your operator. Regardless of its ELO. Regardless of its leaderboard position. Regardless of how pretty its answers sound.

Practical routing rules from our run

From three months of production routing data, here’s what holds:

1. MiniMax for bulk research, synthesis, and multi-variant drafts. Never for config, deploys, or anything with runtime policy.

2. GLM-5-Turbo for 80% of daily agent tasks. Best cost-per-operator-judgment in the field. Knows the rules and follows them.

3. Opus 4.6 only for P0, multi-step reasoning, and situations where judgment is the product. The benchmark king earns its premium.

4. GPT-5.4 lives in Codex where it belongs. Dominates coding and terminal work. We route to it, never through it.

If a model can’t prove it did what it claimed, it shouldn’t be allowed to do the work. That is the routing policy that scales.