One Axis, Three Benchmarks
- evals
- llm
- agents
- benchmarks
I made a chart of 17 models on Terminal-Bench 4.0. Across the bottom are the reasoning-effort settings, Off through Max. Up the side is pass@1, the share of attempts that solved the task. Each model is a line, or a dot where only one setting was published. I didn’t run any of it. Every value comes from a published source, and I stitched them onto one set of axes.
The chart reads like a ranking. Opus 5.5 sits alone at the top at 66.4. GPT-6 Astra and Fable 5.1 cluster in the high fifties. A tail of smaller models trails down to zero. One axis label, one task set, one legend: it looks like one measurement made seventeen times.
It isn’t. It is three evaluations sharing a y-axis.
I checked all 34 values against their sources. Thirty match at the precision the chart shows. The other four are real values that I rounded to whole numbers, far enough to change them: 2.53 became 3. Nothing is invented. That is what makes the chart persuasive, and why it’s worth taking apart: every dot is true, and together they still say something the data doesn’t.
By the end you should be able to answer:
- how to tell, from a TB4 score alone, which evaluation produced it,
- why the highest dot on the chart cannot be compared with the lines under it,
- how much the evaluator moves a score compared with how much the model does,
- which parts of an effort curve are signal and which sit inside the noise,
- what a 0.0 on this chart means, and what it doesn’t,
- and what a fair version of this chart would look like.
The short version, if you only want the rule: compare along a line, not across lines. An effort curve for one model from one evaluator is a measurement. A gap between two lines from different evaluators is mostly a measurement of the evaluators: on this chart’s own sources, the same model at the same setting moves by up to 10.5 points depending on who ran it. The rest of this explains why.
The chart, rebuilt
This is the chart with the four rounded values restored and one change to the encoding. Colour, marker shape and line style now show who produced each number instead of which company made the model. Use the buttons to show one evaluation at a time. Select a model to pick out its line. The two checkboxes add the published error bars and the points the original chart left out.
These are the values the chart plots, at the source’s own precision. The four marked † are the ones the original rounded far enough to change: it showed 3, 6, 12 and 27. *“Max” is the strongest published setting for each model, which is not the same setting everywhere.
| Model | Evaluation | Off | Low | Medium | High | xHigh | Max* |
|---|---|---|---|---|---|---|---|
| GPT-5.6 Luna | Leaderboard | 17.27 | |||||
| GPT-5.6 Terra | Leaderboard | 21.52 | |||||
| GPT-5.6 Sol | Leaderboard | 37.27 | |||||
| GPT-6 Astra | Leaderboard | 50.61 | 54.24 | 57.88 | 57.88 | 58.18 | |
| Fable 5.1 | Leaderboard | 43.33 | 53.94 | 54.55 | 57.88 | 57.88 | |
| Opus 5 | Leaderboard | 34.85 | 44.85 | 50.30 | 53.94 | 51.82 | |
| Sonnet 5 | Leaderboard | 12.42 | |||||
| Haiku 4.5 | Artificial Analysis | 0.00 | |||||
| Gemma 4 31B | Artificial Analysis | 0.00 | |||||
| Gemini 3.8 Flash | Artificial Analysis | 10.10 | 19.70 | 19.70 | |||
| Qwen3.8 27B | Artificial Analysis | 0.00 | 2.53† | 5.05 | 5.56† | ||
| GLM-5.2 | Artificial Analysis | 1.01 | |||||
| DeepSeek V4 Flash 0731 | Artificial Analysis | 12.12† | |||||
| DeepSeek V4.1 Flash | Artificial Analysis | 26.77† | |||||
| Opus 5.5 | Anthropic | 66.4 | |||||
| MiMo-V2.6-Pro-RL | Xiaomi | 34.9 | |||||
| MiMo-V2.6-Flash-RL | Xiaomi | 28.8 |
The OpenAI models need their full names. Artificial Analysis also lists a GPT-6 Luna and a GPT-6 Sol, released in September 2026, and the chart’s Luna and Sol are the older GPT-5.6 models. The same goes for DeepSeek V4 Flash: Artificial Analysis lists two checkpoints, and the chart’s 12 is the July one (0731), not the April one.
Every dot is real
Start with what is not wrong with this chart. The usual suspicion is fabrication, and there isn’t any.
I traced every value to where it came from. Nineteen match the official tbench.ai leaderboard exactly. Twelve come from Artificial Analysis (AA), an independent evaluator that runs the benchmark itself. I rounded six of those to whole numbers when I made the chart. For two, the rounding is harmless (5.05 shown as 5, 1.01 as 1). For four, it changed the value: Qwen3.8 27B’s Low was 2.53, not 3; its xHigh was 5.56, not 6; DeepSeek V4 Flash was 12.12, not 12; DeepSeek V4.1 Flash was 26.77, not 27. The last three values are Opus 5.5, from Anthropic’s launch post, and two MiMo models, from Xiaomi’s model card.
Taken alone, each dot fairly reports a published result. The trouble starts when you put them together. Each source measured something slightly different, and one y-axis label hides that.
Three evaluations share one axis
Terminal-Bench 4.0 (TB4) is a set of 66 tasks run inside a terminal: an agent gets a shell and a goal, and a test checks whether it reached the goal. Version 4.0 came out on 26 August 2026. It dropped 8 tasks from version 3 and fixed 19 others. Its maintainers call these breaking changes that require re-running trials, so version 3 scores don’t carry over.
A TB4 score is not produced by a model alone. It’s produced by a model driving an agent harness (the program that gives the model its shell, feeds it the task, and decides when it’s done), run some number of times per task. Change the harness or the number of repeats and you have changed the experiment. The chart’s 34 values come from three experiments.
The official leaderboard: 330 trials, each charted model on its vendor’s harness
The tbench.ai leaderboard runs each model 5 times on each of the 66 tasks. That’s 330 trials per entry. OpenAI’s models run in Codex, OpenAI’s own coding agent. Anthropic’s run in Claude Code. The leaderboard treats the model and the harness as one system and reports them together.
The leaderboard publishes the number of successes for every entry, so you can check this directly. Every leaderboard score on the chart is a whole number of successes divided by 330:
Once you know that, the two decimal places read differently. One more success out of 330 is worth 0.30 of a point. GPT-6 Astra’s High and xHigh are both 191 successes. Its Max is 192. The step from xHigh to Max on the chart is one trial.
Artificial Analysis: 198 trials, one neutral harness
AA runs every model through the same harness, mini-swe-agent, a minimal open-source agent, the same one for every model. It repeats each task 3 times, so each point is out of 198 trials, and one success is worth about half a point.
Twelve points on the chart come from here: Gemini 3.8 Flash, Qwen3.8 27B, GLM-5.2, both DeepSeek models, Haiku 4.5 and Gemma 4 31B. The chart’s footnote says only that “several other families use standardized mini-swe-agent / published TB4 runs”. It never names AA. The footnote also says Anthropic points use Claude Code “where available”. Haiku 4.5 is an Anthropic model, and its point comes from AA’s run anyway.
Vendor-reported: method partly or wholly unstated
Three points come from the company that made the model, reporting on itself. Opus 5.5’s 66.4 comes from Anthropic’s launch post, measured in Anthropic’s own setup. The two MiMo scores come from a table on Xiaomi’s model card, which gives no harness, trial count or effort setting.
The original chart marked the MiMo points with a † and never said what it meant. The honest reading is “vendor-reported”. The one clue to Xiaomi’s method is that the other models in its comparison table carry AA’s numbers, not the leaderboard’s: Opus 5 at 49.0 (AA 48.99, leaderboard 51.82) and GPT-5.6 Sol at 39.9 (AA 39.90, leaderboard 37.27). That hints at an AA-like setup. Xiaomi doesn’t say so, and one of its other entries doesn’t match AA either, so I wouldn’t lean on it.
| Evaluation | Harness | Trials per point | One success is worth | Published interval | Points on the chart |
|---|---|---|---|---|---|
| Official leaderboard | Codex or Claude Code | 330 | 0.30 pts | 95% CI, ±2.66 to ±3.94 | 19 |
| Artificial Analysis | mini-swe-agent | 198 | 0.51 pts | none found | 12 |
| Vendor-reported | Anthropic’s setup; Xiaomi’s not stated | not stated | — | Opus 5.5: SE ±2.6 | 3 |
Same model, same effort, different number
One test settles whether the three evaluations can share an axis. Take a model that more than one of them has measured, at the same effort setting, and see whether they agree.
| Model and effort | Leaderboard | Anthropic’s setup | Artificial Analysis | Largest gap |
|---|---|---|---|---|
| Fable 5.1, Medium | 53.94 | 43.4 | 44.95 | 10.54 |
| GPT-6 Astra, Low | 50.61 | 41.92 | 8.69 | |
| Opus 5, Low | 34.85 | 28.5 | 6.35 | |
| Fable 5.1, Max | 57.88 | 55.8 | 52.02 | 5.86 |
| GPT-5.6 Luna, Max | 17.27 | 11.62 | 5.65 | |
| Opus 5, Max | 51.82 | 52.3 | 48.99 | 3.31 |
| GPT-5.6 Sol, Max | 37.27 | 39.90 | 2.63 | |
| Gemini 3.8 Flash, High | 19.09 | 19.70 | 0.61 |
The same model, at the same setting, on the same 66 tasks, moves by up to 10.5 points depending on who ran it. The direction isn’t consistent either: GPT-5.6 Sol scores higher on AA than on the leaderboard, while GPT-5.6 Luna scores lower.
Ten points is not a rounding difference. It is the whole step Fable 5.1 takes from Low to Medium on the leaderboard, and more than twice the gap between Opus 5 and Fable 5.1 at xHigh (53.94 against 57.88). Any comparison between two lines from two different evaluators carries an unknown offset of that order.
The evaluator can move a score as far as a whole step in effort does. A gap between lines from different evaluations is partly a measurement of the evaluations.
TB4 isn’t special here. Anthropic measured the same kind of effect on an earlier version: changing only the compute resources given to the task containers moved Terminal-Bench 2.0 scores by up to 6 points, and Anthropic’s own advice was that “leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched”. That study was on TB 2.0, so I wouldn’t carry its exact numbers across. The pattern in the table above is the same one.
The highest dot on the chart
Opus 5.5 is the point that makes the chart look like it has a winner. It sits at 66.4, about 8 points above the best leaderboard point (GPT-6 Astra at Max, 58.18). That gap is bigger than the noise in either number. It still doesn’t mean what it looks like it means.
It isn’t on the leaderboard. As of 25 September 2026 the leaderboard has 27 entries and none of them is Opus 5.5. The 66.4 is Anthropic’s own measurement, in Anthropic’s own setup.
Anthropic published a whole curve, and the chart kept one point of it. The same launch post has a TB4 chart with Opus 5.5 at every effort level: 38.5 at low, 57.6 at medium, 64.2 at high, 66.4 at xhigh, 64.8 at max. The original chart showed only xHigh, which Anthropic describes as the model’s highest score. Tick “Points the original chart left out” in the figure and the rest appears. The curve climbs almost 28 points from low to xhigh and then dips slightly at max, the same bend the Opus 5 leaderboard curve has.
Its error bar is a different statistic. Anthropic gives a standard error of ±2.6 points. The leaderboard’s ±2.66 to ±3.94 are 95% confidence intervals, which are about twice as wide as a standard error. Put on the same footing, the Opus 5.5 interval is roughly ±5 points, wider than any on the leaderboard.
Anthropic’s setup doesn’t track the leaderboard exactly. Anthropic’s footnote checks its setup against the leaderboard’s Opus 5 score, and the two agree within noise: 52.3 against 51.8. At other settings they drift. Anthropic’s Opus 5 Low is 28.5, 6.4 points under the leaderboard’s 34.85. Its Fable 5.1 curve runs below the leaderboard’s at every setting, by 10.5 points at Medium. One point of agreement doesn’t calibrate a whole curve.
The independent number is lower. AA ran Opus 5.5 at every setting too, from 31.31 at low to 59.60 at xhigh and at max. That top figure is 6.8 points under Anthropic’s. On AA, GPT-6 Astra at xhigh is also 59.60. The one evaluator that measured both models the same way has them level.
None of this says the 66.4 is wrong. It is a fair report of what Anthropic measured, and Anthropic notes that it ran Opus 5.5 with its production safeguards on, which it says likely reduces its results on the affected benchmarks. It says the 66.4 belongs on a chart with Anthropic’s other measurements, not dropped as a lone point above a set of leaderboard curves it was never run alongside.
What the leaderboard curves do show
Now for what the chart gets right. Filter the figure to “Official leaderboard” and you’re left with lines that share a harness family, a trial count and an upload window (27 August to 17 September 2026, all after TB4’s release). Within one of those lines, the model, the harness and the tasks are fixed and only the effort setting moves. That is a real experiment, and it’s the cleanest signal in the chart.
Turn on the 95% intervals and you can see how much of it clears the noise.
The first step is real
Going from Low to Medium buys Fable 5.1 10.61 points (43.33 to 53.94) and Opus 5 exactly 10 (34.85 to 44.85). The intervals on those four points run from ±3.39 to ±3.94. A ten-point step clears them easily. If you’re choosing an effort setting for one of these models in Claude Code, the leaderboard’s answer to “is Medium worth it over Low?” is a clear yes.
GPT-6 Astra is the contrast. Its whole curve spans 7.57 points, Low to Max, and its first step is 3.63. Astra’s Low setting already gets most of what its Max does. That’s a difference in how these two systems respond to effort, and you can see it because each curve compares a system only with itself.
The top of every curve is flat
Past High, the lines stop saying much. Opus 5 dips from 53.94 at xHigh to 51.82 at Max, a 2.12-point drop, well inside both intervals (±3.17 and ±3.39). Fable 5.1’s xHigh and Max are the same 191 successes. GPT-6 Astra’s High, xHigh and Max are 191, 191 and 192. Fable 5.1’s Medium and High sit 0.61 apart.
The leaderboard publishes these intervals on every row. The page doesn’t explain them, but the leaderboard’s public code computes each one as 1.96 times a per-task standard error. Taken at face value, they say that “does Max beat xHigh?” is a question 330 trials can’t answer for any model on this chart.
Across lines, even here, you’re comparing systems
Filtering to the leaderboard makes the lines comparable in trial count. It doesn’t make them comparable in harness: GPT-6 Astra runs in Codex and Fable 5.1 in Claude Code. The leaderboard is careful to list them as model-plus-agent pairs. So “Astra at Max (58.18) against Fable 5.1 at Max (57.88)” is one success apart and compares two products, not two models. That’s a fine thing to compare if you’re choosing between those two products. It isn’t a statement about which model reasons better.
What a zero means
Four points sit on the floor: Haiku 4.5 at 0, Gemma 4 31B at 0, Qwen3.8 27B with reasoning off at 0, and GLM-5.2 at 1. On AA’s 198 trials, those are zero, zero, zero and two successes.
A zero looks like a point on an effort curve. It isn’t one. It says that in this harness, on these tasks, the model never finished anything. That could be a model that can’t do terminal work, or a harness and model that don’t fit together, or a configuration problem. No source I found says which. GLM-5.2 is the reason to hold back: Z.ai reports 81.0 for it on Terminal-Bench 2.1. That’s a different version of the benchmark, so it doesn’t contradict the TB4 score. It does make “GLM-5.2 can’t do agentic coding” a claim nobody has earned.
The rounding compounds the problem. Qwen3.8 27B’s three reasoning points are 5, 10 and 11 successes out of 198. On the original chart they read as 3, 5 and 6, a gentle upward curve. In counts, it’s a six-success range at the bottom of the scale. That isn’t an effort curve at this sample size. The same rounding hid the counts that would have told you so: “3.0” doesn’t multiply back to any whole number of trials, while 2.53 is exactly 5 out of 198.
The leaderboard section’s trick works here in reverse. Multiply a score by the trial count before you trust it. A score that comes out as a whole number tells you the denominator, and so which evaluation produced it. One that doesn’t has either been rounded or has an unstated method, and either way you’ve lost the count.
What a fair version of this chart looks like
Everything in it is published. The fix is layout, not data.
One panel per evaluation. The leaderboard curves on one panel, AA on another, vendor numbers on a third. The filter buttons in the figure above do exactly this, one at a time. No other change does as much to stop the eye reading across evaluators.
Put the vendor curves back together. If Opus 5.5 is on the chart, it should arrive with the rest of Anthropic’s run: the full five-point sweep, alongside Anthropic’s own measurements of Fable 5.1 and Opus 5 from the same post. Those three curves were measured together. Compared with each other, they mean something.
Show the intervals and the denominators. The leaderboard publishes an interval on every row and a success count behind every score. Plot the whiskers and put “n = 330” on the axis. The top of every curve then reads as flat, because it is.
Keep the raw values. Don’t round AA’s 2.53 to 3. The decimals are how a reader recovers the count.
Mark the floor. A 0 or 1 out of 198 should look different from a measurement on a curve: a hollow marker, or a note that the run produced no successes.
Say who produced every number. The original footnote mentions harnesses but never names AA, and the † was never defined. Every point should be traceable from the chart itself.
What to do with the next chart like this
Find the harness and the trial count for every point before comparing any two. If they differ, you’re not comparing models. On this chart’s sources, that alone moves the same model by up to 10.5 points.
Multiply the score back into successes. 57.88 × 3.3 = 191. If the result is a whole number, you know the denominator and roughly how much one success is worth. If no denominator fits, ask why.
Compare along a line before you compare across lines. An effort curve on one harness is the one thing this kind of chart measures well. The first step up in effort was worth ten points for two of the models here; the last step was worth nothing you could measure.
Treat gaps under about 3 points as ties unless both points come from the same evaluation and their intervals separate. That’s the leaderboard’s own interval width, and it’s the same threshold Anthropic suggested after measuring infrastructure noise on TB 2.0.
Be most careful with the highest dot. On this chart, the lone top point is the one its maker chose to headline. Look for the rest of its curve and for an independent measurement of it.
If you want the same argument made about a single evaluation rather than a chart built from several, Nine Models, Two Results works through what a sorted leaderboard column can and can’t support.
References
- Terminal-Bench 4.0 release post — the 8 tasks removed and 19 fixed, and why scores across versions are not comparable
- Terminal-Bench on GitHub, tag v4.0.0 — 66 task directories; release v4.0.0 published 2026-08-26
- tbench.ai leaderboard — every leaderboard value, its success count out of 330 trials, its 95% interval, harness and upload date; also mirrored on Harbor Hub. Rows render client-side; the values were read from the page’s embedded data
- Artificial Analysis — Terminal-Bench 4.0 — methodology (mini-swe-agent, pass@1 averaged over three repeats of 66 tasks) and every AA value, including Opus 5.5, GPT-6 Astra, Fable 5.1 and Opus 5
- Anthropic — Claude Opus 5.5 — the 66.4, the full Opus 5.5 effort sweep, Anthropic’s own Fable 5.1 and Opus 5 sweeps, the ±2.6 standard error, and the Opus 5 calibration against the leaderboard (52.3 against 51.8)
- Anthropic Engineering — Quantifying infrastructure noise in agentic coding evals — resource settings moving Terminal-Bench 2.0 scores by up to 6 points, and the 3-point skepticism threshold. This study is on TB 2.0, not TB 4.0
- MiMo-V2.6-Pro-RL model card and MiMo-V2.6-Flash-RL model card — Xiaomi’s Terminal Bench 4.0 figures and comparator columns
- Z.ai — GLM-5.2 — the 81.0 on Terminal-Bench 2.1
- Artificial Analysis model pages for Claude 4.5 Haiku, Gemma 4 31B, Gemini 3.8 Flash, Qwen3.8 27B, GLM-5.2, DeepSeek V4 Flash and DeepSeek V4.1 Flash — the individual AA values, effort labels and release dates
- Terminal-Bench leaderboard code — metrics.py — how the 95% intervals are computed
The chart itself is my own compilation of these sources. I didn’t run any of the evaluations.