The Leaderboard Is Contaminated: Why Coding Benchmarks Stopped Telling You Anything

There is a specific kind of meeting happening in engineering organisations right now. Someone brings a slide with model scores. Someone else asks which model the team should standardise on. The scores are all above 70%, clustered within four points of each other, and every one of them is a number the vendor chose to publish. Nobody in the room can say what a four-point gap means for the codebase they actually maintain, because that gap is smaller than the measurement error, and the measurement itself has been quietly broken for about eighteen months.

What Contamination Actually Means Here

Contamination is a plain concept dressed in a technical word. A benchmark measures generalisation: can the model solve a problem it has not seen? If the problem and its solution appeared in the training corpus, the benchmark measures recall instead. The model is not reasoning its way to the patch. It is retrieving one. Both produce a green checkmark, and from the outside they are indistinguishable.

SWE-bench was constructed from real GitHub issues in popular open source Python repositories, paired with the commits that resolved them. That construction is what made it good. It is also what made contamination inevitable. The issues are public. The fixes are public. The discussion threads explaining the fixes are public. Every one of those artifacts sits in exactly the kind of high-quality code corpus that model labs scrape. The benchmark did not leak. It was public from the day it was built, and time did the rest.

Fifteen to twenty points is not a rounding error. It is larger than the entire spread between the top ten models on most public leaderboards. Put differently: the gaps teams argue about in model selection meetings are comfortably inside the noise floor created by differential exposure to the test set. A model that scores three points higher may simply have trained on more of the answer key.

When a frontier lab stops publishing a number it used to lead on, that is the clearest available signal about the number's value. It is not modesty. It is an acknowledgement that the metric has stopped separating anything.

HumanEval Is Dead and Has Been for Years

The same story played out earlier and more completely with HumanEval, and it is worth revisiting because the pattern repeats. HumanEval's 164 hand-written Python problems have been public since 2021. By 2024 frontier models were clearing 90%. By 2026 the benchmark discriminates nothing at all; the remaining failures are mostly ambiguities in the problem statements rather than capability gaps.

A benchmark has a life cycle, and it is short. It is built, it is hard, it becomes the standard, it is optimised against, it saturates, and it dies. HumanEval took about three years. SWE-bench took about two. Each generation of benchmark burns faster than the last, because the amount of compute and attention pointed at beating it keeps growing. Any team building a model selection process around a public leaderboard is building on something with a known expiry date and no label.

The 23.3% That Reset Expectations

Scale AI's response was to rebuild the benchmark with contamination resistance as a design constraint rather than an afterthought. SWE-bench Pro carries 1,865 problems across 41 repositories, deliberately spanning public code, held-out code, and commercial codebases that were never available to scrape. The problems are multi-file, which matters enormously, because multi-file changes are where real maintenance work lives and where single-function benchmarks were never testing.

Twenty-three percent is a genuinely useful number, and the useful part is not that it is low. It is that it discriminates. When the top of the leaderboard is at 23.3% and the field spreads meaningfully beneath it, differences between models mean something again. A benchmark where everyone scores above 80% has stopped being an instrument. A benchmark where the leader manages less than a quarter of the problems is measuring something the models have not yet absorbed.

The number also recalibrates a conversation that has drifted. Somewhere in the last two years the industry started talking about coding agents as though repository-scale autonomous maintenance were roughly solved and only reliability engineering remained. On problems designed to resist memorisation, the best available systems complete under a quarter of them. That is a remarkable capability and it is nowhere near the story the leaderboards implied. It also lines up much better with what teams report from practice, which is the tell that it is the more honest measurement.

LiveCodeBench and the Structural Fix

LiveCodeBench takes a different route to the same goal. Rather than holding out private repositories, it continuously sources fresh competitive programming problems from contests that take place after a given model's training cutoff. Contamination is not policed. It is made impossible by ordering: the problem did not exist when the model was trained, so it cannot have been memorised.

That design solves contamination properly and introduces a different limitation, which is worth stating plainly. Competitive programming problems are self-contained, have precise specifications, and are graded by exact output matching. Almost nothing in professional software engineering has those properties. Real work involves ambiguous requirements, existing architecture you did not choose, tests that encode decisions nobody remembers making, and a definition of correct that includes not breaking six things you were not thinking about. LiveCodeBench measures a real skill cleanly. It is not the skill you are buying a coding agent for.

Which leads to the uncomfortable conclusion underneath all of this. There is no public benchmark that both resists contamination and resembles your work, and there probably cannot be, because resembling your work means being your work, and being your work means not being public. The best available public benchmarks are triangulation instruments. They tell you the rough shape of the field. They do not tell you what will happen in your repository.

Build the Benchmark That Applies to You

This is where the practical work is, and it is more tractable than most teams assume. An internal coding benchmark is a set of tasks drawn from your own history, with a scoring function you trust. Two weeks of focused effort produces something more predictive than any leaderboard, and it keeps paying out every time a new model ships.

Harvest tasks from merged pull requests

Take thirty to fifty merged PRs from the last year, weighted toward the kind of work you want agents doing. For each, capture the repository state at the parent commit, the issue text or PR description as the prompt, and the merged diff plus the test suite as the reference. You now have tasks that are real, that are specific to your architecture, and that came with a verified correct answer. Exclude anything whose fix is publicly documented if your repository is open source, for the same contamination reason that broke SWE-bench.

Score the trajectory, not only the diff

Grading only the final output is the mistake that makes internal benchmarks as misleading as public ones. As we covered in eval-driven development, output-only grading inflates pass rates by 20 to 40% relative to full trajectory evaluation. An agent that arrives at a correct patch after twelve failed attempts, three unrelated file modifications, and a deleted test is not a passing result. It is a failing result with a lucky ending, and in production it is the one that costs you an afternoon.

Report consistency, not best-of-N

Run each task multiple times and report the all-runs pass rate rather than the best one. Coding agents are non-deterministic, and the difference between a task that passes eight times out of ten and one that passes once out of ten is the entire question of whether you can put it in a pipeline. Vendor benchmarks overwhelmingly report best-case or averaged figures, which is one more reason their numbers do not survive contact with your deploy process.

Re-run on every model release, then decide with data

Once the harness exists, evaluating a new model is a few hours of compute rather than a debate. This matters more each quarter, now that models ship at patch cadence rather than annually. Teams without an internal harness end up either upgrading on marketing or refusing to upgrade at all, which is the exact dynamic behind model upgrade fatigue. A harness turns a religious argument into a table.

Why Scores Rose While Your Experience Did Not

There is a common and disorienting experience among engineering teams: the published numbers improve steadily every quarter, and the day-to-day feel of working with these tools improves much more slowly. That gap is usually blamed on hype, and contamination explains most of it without needing anyone to be dishonest.

A benchmark score rises for three separable reasons: the model genuinely got better, the harness around it got better, or exposure to the test set increased. Only the first travels to your codebase automatically. The second travels if you adopt the same scaffold. The third does not travel at all, because your repository was never in anyone's training data, and it is the component that grows fastest as a benchmark ages.

So the honest expectation for a model generation that adds six points on a mature public benchmark is somewhere between zero and six points on your work, skewed toward the low end, and the only way to find out where is to measure. That is a deflating thing to tell a leadership team excited about a launch, and it is considerably less deflating than discovering it two quarters into a plan built on the headline figure.

How to Read a Public Leaderboard Without Being Misled

Public benchmarks still have a job. The job is coarse filtering, not selection. A few habits keep them useful.

Check the benchmark's construction date against the model's training cutoff before reading any score. If the benchmark predates the cutoff and its data is public, treat the number as an upper bound rather than an estimate. Prefer benchmarks that publish their contamination methodology; the ones that do not address it at all have usually not thought about it. Treat any score above 80% on a public set as saturated by default, and go looking for a harder set rather than celebrating.

Be especially careful with composite scores and vendor-reported figures on benchmarks the vendor selected. That is not an accusation of bad faith; it is a structural point. Any lab choosing which benchmarks to report will, entirely honestly, report the ones where its model looks strong. The aggregate effect across an industry is a set of leaderboards where every model is best at something, and none of it maps to your backlog.

Finally, weight independent evaluations above self-reported ones, and weight your own numbers above both. That ordering sounds obvious written down and is routinely inverted in practice, because a vendor's slide is available today and your harness takes two weeks.

The Harness Is Part of the Score

One more distortion is worth naming, because it explains why two teams running the same model report wildly different results. When you evaluate a coding agent, you are never evaluating a model alone. You are evaluating a model plus a scaffold: the tool set it can call, the context it gets handed, the retry policy, the test runner, the diff application logic, and the stopping condition. Change any of those and the score moves, sometimes by more than the difference between two model generations.

This is why a benchmark result reported by a lab and a benchmark result reported by a tooling company are not comparable even when they name the same benchmark and the same model. They ran different harnesses. It is also why the number your team gets in week one is usually well below the published figure, and why it rises substantially over the following month without anyone changing models. You were improving the scaffold. That is real progress, and it is invisible to anyone reading only the model name.

The practical consequence is that your internal benchmark should hold the harness constant when comparing models, and then separately hold the model constant while iterating on the harness. Teams that conflate the two spend months attributing scaffold improvements to model upgrades, and then get badly surprised when they switch providers and the gains do not travel. We wrote about the scaffold layer in more depth in the agent harness guide, and the measurement point is the same: the unglamorous infrastructure decides the outcome more often than the model does.

What No Benchmark Measures At All

Even a perfect, uncontaminated, repository-realistic benchmark would leave out most of what determines whether agent-assisted development works in an organisation. Benchmarks score whether a task was completed. They do not score whether the resulting code is something a maintainer will still understand in a year.

They do not measure review load, which is the cost that actually lands on your senior engineers. They do not measure whether the agent's change introduced a dependency, widened an interface, or quietly duplicated logic that already existed two directories over. They do not measure how often the agent gives up in a way that wastes an hour versus a way that wastes a minute. And they do not measure the maintenance tail, which is the part that arrives two quarters later and which we covered in the tech debt bill for AI coding speed.

If you are building an internal harness anyway, add two cheap proxies for these. Track the review time per agent-authored pull request alongside the pass rate, and track the number of net new direct dependencies introduced per completed task. Neither is precise. Both correlate with the costs that benchmarks hide, and both are already sitting in your version control and review data waiting to be counted.

The Honest Position

Benchmarks broke because they succeeded. A public benchmark that matters attracts optimisation, and optimisation against a fixed public target eventually becomes indistinguishable from memorisation. This is not a scandal and there is no villain in it. It is a structural property of measuring anything with a published answer key, and it will happen to SWE-bench Pro too, on a schedule set by how long its held-out repositories stay held out.

The teams making good model decisions in 2026 have stopped treating this as a problem to be solved by finding a better leaderboard. They have internalised that public benchmarks indicate the frontier and internal benchmarks decide procurement, and they have built the second thing. It is two weeks of work that pays back on the first model release, and it converts the most expensive recurring argument in an engineering organisation into a number that everyone can check. That is a better trade than any four-point gap on a contaminated leaderboard was ever going to be.