The Model Ranked Third Never Performs Third
- evals
- llm
- statistics
- deployment
Second and third place on a ranking I ran last week are separated by one hundredth of a point.
| # | Model | Score / 5 |
|---|---|---|
| 2 | thinkingcap-qwen3.6-27b | 3.38 |
| 3 | poolside/laguna-s-2.1 | 3.37 |
The obvious reading is that these two are indistinguishable, and in one narrow sense that is right: the gap is far smaller than either model’s confidence interval. The usual advice at this point is to call it a tie and pick on price.
That advice is wrong here, and it is wrong for a reason no leaderboard column can show you. On the twelve prompts behind those two numbers, one of these models never scored below 2.14 and spent two thirds of its run in the middle of the scale. The other scored above 4.0 six times and below 2.0 four times, with a low of 0.32, nearly the bottom of the range.
They have the same mean because a mean will average anything you give it. They do not have the same behaviour, and the difference between them is the single most important thing you would want to know before putting either into production.
By the end you should be able to answer:
- why two models with the same score can carry completely different deployment risk,
- how to spot a bimodal model from data your eval already stored,
- why a confidence interval on the score does not warn you about this,
- what high per-prompt variance is usually telling you about your prompt set,
- and which of these two you would actually want behind an unattended pipeline.
The rule first: a leaderboard score is a mean, and a mean is only a summary when the thing it summarises has one mode. Before you trust a rank, look at the spread of the scores that produced it. The rest of this is one worked case where the spread inverts the recommendation.
The same number, twice, from two different shapes
Here is every prompt in the run, with both models’ scores, sorted by how well Laguna did.
| Prompt | laguna-s-2.1 | rank | thinkingcap | rank |
|---|---|---|---|---|
| hard-math | 4.94 | 1 | 4.09 | 3 |
| sweep-smoke | 4.74 | 1 | 2.25 | 6 |
| judge-payload | 4.74 | 2 | 3.75 | 4 |
| svg-pelican | 4.73 | 2 | 2.46 | 5 |
| html-page | 4.67 | 1 | 3.46 | 4 |
| physics-canvas | 4.28 | 2 | 4.24 | 3 |
| long-essay | 3.90 | 1 | 2.14 | 7 |
| 3d-earth | 3.25 | 4 | 2.61 | 6 |
| rag-qa | 1.77 | 6 | 3.33 | 4 |
| coding | 1.59 | 8 | 4.17 | 1 |
| List two primes | 1.54 | 7 | 3.21 | 2 |
| algorithms | 0.32 | 8 | 4.89 | 1 |
| Mean | 3.37 | 3.38 |
(These scores are win rates in disguise, not marks. A 4.94 means “beat a typical rival in this field about 99% of the time on this prompt”; 0.32 means about 6%.)
Summarise each column and the difference is stark:
laguna-s-2.1 | thinkingcap | |
|---|---|---|
| Mean | 3.37 | 3.38 |
| Standard deviation | 1.56 | 0.85 |
| Lowest score | 0.32 | 2.14 |
| Highest score | 4.94 | 4.89 |
| Prompts scoring above 4.0 | 6 | 4 |
| Prompts scoring below 2.0 | 4 | 0 |
| Prompts in the middle (2.0–4.0) | 2 | 8 |
Laguna has almost nothing in the middle. Ten of its twelve prompts are at one extreme or the other. Its distribution has two humps and a hole where its own average sits.
Laguna’s per-prompt rank across the twelve prompts is 1, 1, 1, 1, 2, 2, 2, 4, 6, 7, 8, 8. It placed third on none of them.
A mean describes a model that behaves like its average. This model has no average behaviour to describe. It has two behaviours, and the leaderboard reports the midpoint of a gap it never occupies.
What the spread is worth knowing
Behind an unattended pipeline (batch summarisation, classification, an agent step nobody reviews) Laguna fails outright on a third of the work and does so without warning. You cannot tell in advance which third, because the thing that predicts it is the task, and the task varies. ThinkingCap never produced a bottom-tier answer in twelve attempts. It also produced fewer top-tier ones, and if you only ever look at the average you will never notice you traded that away.
With a human in the loop the calculus flips. Laguna’s excellent answers are genuinely better than ThinkingCap’s, six scores above 4.0 against four, and a reviewer who can discard the bad ones is buying the top of that distribution and paying nothing for the bottom. Under review, variance is cheap and peaks are valuable. Unattended, variance is the whole cost.
Same two numbers on the leaderboard. Opposite recommendations. The leaderboard cannot express the difference because it threw away the shape when it took the mean, and it did so before you ever saw the table.
The per-prompt scores are not lost, incidentally. They are sitting in the same stored record the leaderboard was rendered from. Every eval harness that ranks models has this data. Almost none of them show it.
The error bar does not warn you about this
The natural objection is that the confidence interval should catch this. A high-variance model ought to get a wide interval.
Laguna’s interval is 3.41–4.17, a width of 0.77. ThinkingCap’s is 3.14–3.81, a width of 0.68. Laguna’s is wider by 0.09, on a five-point scale, when its standard deviation across prompts is nearly twice as large.
The interval barely reacts, and the reason is in how it is built. The bootstrap resamples individual judgments, drawing them with replacement from the pooled set of all comparisons across all prompts. So it answers: if this same judge re-judged this same twelve prompts, how far would the number move?
The question you have is: if I had chosen twelve different prompts, how far would it move?
For a model whose score depends heavily on which task it got, those two questions have wildly different answers, and only the second one is about deployment. Resampling individual observations rather than whole clusters is the standard way to understate variance when observations correlate inside a cluster, and judgments within a single prompt are strongly correlated, because they are all reactions to the same task. A model that misreads the task loses most of that prompt’s comparisons together, as one event, not as eight independent ones.
Resample prompts with replacement instead of judgments, and Laguna’s interval would open up dramatically while ThinkingCap’s would not. That version of the interval would tell you what you wanted to know. The version on the page tells you about judge noise.
The variance is usually telling you about your prompt set
Here is the part that changed how I read the whole table.
Look at where Laguna failed. Its two worst results in the entire run are algorithms (0.32, last place) and coding (1.59, last place).
Poolside describes Laguna S 2.1 as its “strongest model for agentic coding and long-horizon work”: 118 B total parameters with 8 B active, a 1 M-token context, and a Terminal-Bench 2.1 score of 70.2 with maximum thinking enabled. It is a coding model. It came last on the coding prompts.
The tempting conclusion is that the vendor is overselling and the eval caught it. I do not think that is what happened, and the article would be worse if I claimed it.
Terminal-Bench measures long-horizon agentic work: many turns, tool calls, a terminal, a task that unfolds. This ranking measures single-turn prompt-and-response, scored by a judge reading one reply. Those are different jobs. A model trained and tuned for the first is being examined on the second, in a run where the judge was shown both the reasoning trace and the final answer. That mismatch is a perfectly ordinary explanation for a wild score distribution, and it is more likely than the vendor being wrong about their own benchmark.
Which is exactly the point. High per-prompt variance is usually a message about the fit between the model and the prompt set, not a property of the model. Laguna’s spread is the eval telling you that this prompt set is measuring something orthogonal to what this model was built for — sometimes catching it in a task it handles well, sometimes in one it was never aimed at.
The mean silently averaged those two regimes into a single number and put it third.
I will not claim Laguna is bad at coding. This is one study, one judge, one rubric, two coding-flavoured prompts out of twelve, in a format the model was not designed for. The finding is about the leaderboard, not the model.
The field was never a field
Once you look at variance as a fit signal, the composition of the run stops looking like a comparison and starts looking like a category error. Here is what the nine “competitors” actually were:
| Session | What it is |
|---|---|
qwen/qwen3-coder-480b | 480 B total / 35 B active agentic coding MoE |
thinkingcap-qwen3.6-27b | Fine-tune of Qwen3.6-27B reaching comparable capability with ~50% fewer thinking tokens |
poolside/laguna-s-2.1 | 118 B / 8 B MoE for long-horizon agentic coding |
qwen/qwen3.6-27b | The 27 B base model the two above are derived from or compared against |
qwen/qwen3.6-35b-a3b | 35 B / 3 B MoE — and the judge scoring this entire run |
prism-ml/bonsai-27b | 1-bit / ternary compression of Qwen3.6-27B, 3.9 GB, retaining a stated 90% of baseline on the vendor’s own suite |
google/gemma-4-26b-a4b | 26 B / 4 B general-purpose MoE |
qwen-agentworld-35b-a3b-oq8-mlx | 8-bit quantisation of a world model trained to simulate agentic environments — to predict the next environment state given an action |
diffusiongemma-26b-a4b-it | Experimental discrete-diffusion model that Google documents as scoring below Gemma 4 and does not recommend where output quality matters |
Three of those are not general assistants at all. qwen-agentworld is an environment simulator; asking it to draw an SVG pelican and ranking the result eighth is not a finding. diffusiongemma is shipped as an experiment with a written warning attached. bonsai-27b is a one-bit compression, so its position relative to qwen3.6-27b is a quantisation measurement wearing a leaderboard costume.
A mean is a summary of comparable things. Two thirds of this table compares things that were built to do different jobs, and it will happily print a number for all of them to two decimal places.
That is not an argument against running the sweep. Putting a quantisation, a fine-tune, and their shared base model into one field is a genuinely useful experiment. It is how you find out what the compression cost. It is an argument against reading the resulting column as a ranking, because the column has no way to say “these three rows are a quantisation study and those two are not commensurable at all.”
What to do on Monday
Pull the per-prompt scores before you pick a model. They are already in your eval’s stored output. One standard deviation per model, next to the mean, is enough. You are looking for the model whose spread is double everybody else’s.
Decide which shape your system wants, explicitly. Unattended pipeline: take the model with the higher floor, even at a lower mean. Human review, or anything with retries: take the peaks and let the reviewer discard the failures. These are opposite choices and the mean cannot make either of them for you.
Bootstrap over prompts, not over judgments, if you want an interval that answers the deployment question. If your harness resamples individual comparisons, its intervals are measuring judge noise and will look reassuringly tight on exactly the models you should be most careful with.
Read high variance as a question about your prompt set first. Before concluding a model is unreliable, check whether the prompts match the regime it was built for. A long-horizon agentic model examined on single-turn replies will look erratic no matter how good it is.
Check that your field is a field. If the same table contains a world model, a one-bit compression, and something the vendor labels experimental, you have run several different studies at once and stacked their results in one column.
All of which comes back to the one number at the top. A mean is only a summary when the thing it summarises has a single mode, and nothing in a sorted column tells you whether that condition holds. The spread does, and you already have it.
References
- Deen & de Rooij (2020), “ClusterBootstrap” — Behavior Research Methods 52:572–590; why resampling observations rather than clusters understates variance
- Poolside — Laguna models — Laguna S 2.1 parameter counts, context length, positioning, and the Terminal-Bench 2.1 score of 70.2 with maximum thinking enabled (60.4 without)
- Qwen3-Coder-480B-A35B-Instruct — parameter counts and intended use
- bottlecapai/ThinkingCap-Qwen3.6-27B — the thinking-token reduction claim
- prism-ml/Bonsai-27B-gguf and PrismML docs — the 1-bit and ternary builds and retained-quality figures
- Qwen/Qwen-AgentWorld-35B-A3B — the language world model and what it was trained to predict
- DiffusionGemma model overview and Google’s announcement post — experimental status and the recommendation to prefer Gemma 4 for output quality
- Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B, and google/gemma-4-26B-A4B-it — parameter counts for the three remaining rows of the field table
- Terminal-Bench — what the benchmark Laguna is measured on actually asks a model to do
The ranking record behind the per-prompt tables is from a private evaluation harness on my own machine and is not linkable. Every score, rank, and standard deviation quoted here is computed directly from that record.