TalkA benchmark is not a brief
A benchmark table is the most persuasive object in this market and the least transferable. It reports how one configuration of one model performed on one task set under one harness on one date, which is three or four questions away from whether the thing will do your work. The useful part is that the vendors say so themselves, in footnotes, immediately below the figures. We read Anthropic's Claude Opus 5.5 announcement and OpenAI's GPT-6 Astra page on October 20, 2026. Between them they publish enough caveats to dismantle their own charts, and that is to their credit.
The headline score is a setting you are probably not using
Start with the sentence that governs Anthropic's entire comparison table.
"Unless otherwise noted, all Claude Opus 5.5 results use adaptive thinking at max effort." The Terminal-Bench row is noted separately: Opus 5.5 at xhigh effort and GPT-6 Astra at high effort, which "represent each model's highest score".
The table therefore reports best-case configurations. The same page's cost charts report something else: "Opus 5.5 at default effort beats Opus 5 at max effort for about a fifth of the cost."
Those are two different products from the buyer's seat. One is the model turned up as far as it goes; the other is the model as shipped, which is what your invoice and your latency will reflect.
The page even lets you see the gap. The table lists FrontierCode v1.1 (Main) at 54.4 per cent, while the chart caption below says "At default effort (medium), Opus 5.5 scores 54.6%". Both figures were on the page we read, which is a small discrepancy and a useful reminder that these numbers carry a configuration with them.
The error bar is published and the ranking ignores it
Anthropic prints its uncertainty, which not every vendor does, and the uncertainty is large relative to the gaps being celebrated.
Footnote one states: "The standard error is ±2.6 pts for Claude Opus 5.5 and ±1.6-2 pts for the other Claude models." For the science benchmark, footnote three gives "±3.5-5 pts per model".
The same footnote shows what that means in practice: "The public leaderboard (5 trials/task, Claude Code harness) reports Claude Opus 5 at 51.8%; our setup reproduces it at 52.3%, within noise."
Half a point of difference between a leaderboard and the vendor's own rerun of the same model. Any ranking that turns a two-point gap into a verdict is reading past its own error bars.
Safety systems move the score, and both pages say so
The most interesting admissions concern what was switched on during the run.
Anthropic states that "Claude Opus 5.5 was evaluated with its production safeguards enabled", and then explains what happened when those safeguards fired: "cybersecurity tasks were completed by Claude Opus 4.8, and biology and frontier LLM development tasks were completed by Claude Opus 5".
It adds the obvious consequence: "This likely reduces Claude Opus 5.5's performance on these benchmarks."
OpenAI's footnotes run the other way on a competitor. Footnote seventeen states that "For ScreenSpot-Pro and ExploitGym, the Fable scores we report come from Mythos, which is Fable with fewer safeguards."
And footnote twelve explains two blank rows: Claude models are "not included in LifeSciBench Gold v1, GeneBench Pro v13, and MedChemBench because they refuse the majority of questions in these evaluations".
A refusal is a product decision, not a capability ceiling. Depending on your work it is either the score that matters most or the one that matters least, and no single column can tell you which.
Each vendor grades the other, and the numbers disagree
This is the finding that should change how the tables are read.
On Terminal-Bench 4.0, Anthropic's table lists Claude Opus 5 at 52.3 per cent. OpenAI's coding table lists the same model at 52.6 per cent. Close enough to be reassuring.
On FrontierCode 1.1 (Main), Anthropic's table lists Claude Opus 5 at 48.0 per cent. OpenAI's table lists Claude Opus 5 at 53.4 per cent.
That is a 5.4-point spread on the same named benchmark for the same named model, published by two vendors in the same season, and it is larger than most of the gaps either page is asking you to care about.
Smaller disagreements sit beside it. Anthropic puts Claude Fable 5.1 at 50.3 per cent on that benchmark; OpenAI puts it at 50.9. Meanwhile OpenAI's coding table does not list Claude Opus 5.5 at all, so the newest competitor is absent from the comparison.
Neither page is dishonest. They ran different harnesses at different effort settings, which is precisely why a cross-vendor number cannot be lifted out of a table and quoted on its own.
The harness is part of the score
The footnotes make the apparatus visible, and the apparatus is substantial.
OpenAI's footnote eight says its FrontierCode run used "a developer message similar to a section of its developer message in Codex", and quotes it: "Avoid creating excessive test files... The goal is clean, mergeable code." The note adds that "The prompt was not optimized for the eval".
Footnote one says the ARC-AGI-3 run used a harness "which changes two settings to better match real-world performance".
Footnote sixteen is the frankest of all: "When we test across third-party models, we use a simpler research setup. Codex has a more complex production configuration, which can result in different raw-model error rates." It ends: "Users do not experience the no-confirmation scenario in Codex, as it's an internal research configuration."
A score is therefore a property of model, prompt, scaffold, effort level, time limit and turn limit together. Change any one and you have a different number for the same model.
Footnote fourteen demonstrates that with a figure: a 5.5 per cent result for GPT-5.6 Sol is described as "an artifact of the 300-turn limit in the benchmark, which is not a limit that real customers using max would have", with 11.5 per cent reported when fewer limits bound the run.
Saturation means the test is finished, not the job
OpenAI's page reports that Astra "saturates FrontierMath Tier 4 with a 98% score" and "saturates ARC-AGI-3 with a 99.9% score and ExploitBench with a 100% score".
Saturation is the honest word, and the page follows it honestly. Footnote fourteen notes of one benchmark that "Some included vulnerabilities may not permit arbitrary code execution under the evaluation's constraints, so a 100% success rate may not be achievable."
The page also raises contamination on itself: "Given concerns that exposure to historical software vulnerabilities may have affected benchmark results, we also evaluated Astra on two novel benchmarks." One was built from vulnerabilities "from the previous three months".
When a vendor builds a fresh test because the old one may be in the training data, the old score has become a historical artefact. Full marks mean the exam expired.
How to read a benchmark table against your own work
The tables are still worth reading. They are worth reading as evidence rather than as rank order:
- Find the effort setting. Anthropic's table is max effort; its cost charts are default effort. Price and plan against the one you will run.
- Check the error bar before the gap. At ±2.6 points, a two-point lead is not a lead.
- Ask what was switched on. Production safeguards lowered one vendor's own score; a lower-safeguard variant raised a competitor's on another page.
- Prefer the benchmark that resembles your task. FrontierCode asks whether a change would be merged; a mathematics tier does not predict that.
- Distrust any number quoted without its harness. The same model differs by 5.4 points between two vendors' tables on one benchmark.
- Re-read after the next release. Saturated benchmarks get replaced, and a score with no live comparison is a date stamp.
What none of these tables can supply is the only measurement that settles a purchase, which is your own task run twice on two tools with the result inspected by someone who knows what good looks like.
Why the charts keep working anyway
Benchmarks are not marketing noise. They are the only common language the field has, they catch regressions, and the footnotes above represent a real discipline that most software categories never developed.
The trouble is the frame rather than the data. A chart with five bars invites a ranking, and a ranking invites a decision, while the footnote underneath is explaining that the bars were measured under conditions you will not reproduce.
A brief says what the work is, what counts as done, and what it is worth. A benchmark says that a configuration scored a percentage. Those documents answer different questions, and only one of them is yours.
What follows in practice
Use the table to build a shortlist, then test the shortlist on one real task from your own week. The gap between a benchmark and a brief is the subject of the tool is not the workflow.
Scores also expire with the model behind them, which is argued in when a model you built on is retired.
Our own shortlist, with the documentation we read and the dates, is in the guide to the best AI coding assistants for small teams, with detail in the subject pages for Claude, ChatGPT, Cursor, GitHub Copilot and Kiro.
Two of them are set head to head in Cursor versus GitHub Copilot.
Read the footnotes first. They are shorter than the chart and they tell you whether the chart is about you.
Sources, read October 20, 2026: anthropic.com Claude Opus 5.5 announcement including benchmark table and footnotes; openai.com GPT-6 Astra page including coding and science tables and footnotes.