Key takeaways
A knowledge test, a coding task and an interactive support task measure different capabilities. Choose evidence that resembles the work you want done.
Agent scores depend on the model, its surrounding software, tools, execution limits and grader. A leaderboard cannot isolate the model when those conditions differ.
A benchmark can reject valid work or accept incomplete work. Recent coding audits and web-agent studies show why the evaluator needs inspection too.
A task-completion time horizon measures difficulty in human work time at a stated success rate, not how long an agent can operate reliably without supervision.
Use public results to shortlist systems. Test the complete configuration on your own inputs, permissions and resource limits before treating the score as evidence for deployment.
Choosing a model from scores means choosing which evidence to trust
Suppose you are choosing a model for an internal research assistant. The candidates report strong results on knowledge questions, coding and tool use. Your assistant will need to retrieve current documents, resolve inconsistent instructions and produce an answer someone can check. A correct answer must use the applicable source version, cover the question and expose unresolved conflicts. Which scores help select a system for that particular job?
The difficulty is that the scores describe different experiments. Some supply all the necessary information in a prompt. Others require the system to find it. Some grade an answer, while others inspect changed files or a simulated database. A model that knows the answer may still select the wrong document or send the wrong argument to a tool.
Public benchmarks can save substantial selection effort. They expose capabilities and failures across tasks that one organization could not afford to construct. But using them requires reading beyond the score. The cases below show what to inspect, which differences can change the result, and where a local evaluation must begin.
Answer accuracy and successful tool use are different measurements
MMLU evaluates multiple-choice questions across 57 subjects. It tests broad knowledge and problem solving in that answer format. It does not require the model to authenticate a customer, navigate an application or check that a requested change happened.
Agent benchmarks add software that interprets model outputs and carries out actions. This surrounding software is often called the harness. It controls such things as which tools are available, what observations reach the model, how errors are returned and when execution stops. The tested configuration includes that software as well as the model.
Consider tau2-bench’s original June 2025 paper. In its telecom environment, the support agent and a simulated customer have different tools. The agent can inspect service records; the customer can change phone settings. Success requires coordinating their actions, with telecom tasks graded by assertions about the resulting state. A model answering a troubleshooting question has not completed this experiment.
The distinction helps select evidence. Knowledge questions can identify inadequate subject understanding. Interactive tasks can test whether a configured agent uses tools and communicates under their rules. Neither establishes that the agent will handle your identity checks, missing documents or escalation process. A simulated customer also leaves interaction with real customers to be tested.
Figure 1. Read each score as the result of its stated experiment. Answer accuracy, configured task completion and human-duration horizons do not establish the same capability.
A task horizon measures human work time, not autonomous operating time
METR’s time-horizon evaluation relates agent success to estimated human completion time for the tasks. A 50% horizon marks the human-duration point where the fitted success probability reaches one half. It does not measure the agent’s runtime or promise that every task of that duration is equally tractable.
METR’s January 2026 update expanded the suite from 170 to 228 tasks. Such measurements help track progress beyond short questions. Their relevance to a deployment still depends on the work represented: primarily well-specified software, machine-learning and cybersecurity tasks, with limited prior context for the human baseline.
Do not read a working day on that scale as a working day inside your company. Identify the task distribution and success threshold before using a horizon to plan an automation trial. As checked on October 22, 2026, the dashboard was last updated September 8 and said it was no longer actively updated. It warned that estimates above 16 hours were unreliable with its task suite.
Execution limits can change which coding agent appears better
In February 2026, Anthropic varied infrastructure limits on Terminal-Bench 2.0, keeping the model, harness and tasks constant. Success differed by six percentage points between strict resource enforcement and uncapped execution.
From strict enforcement to three times the specified resources, infrastructure errors declined while task-success differences remained within noise. Many interrupted attempts would have failed even without the interruption. At larger budgets, success increased faster than infrastructure errors declined; the extra resources enabled approaches using larger dependencies or more memory. Improving execution stability and enabling more resource-intensive solutions are different effects of additional resources.
This result does not establish a six-point uncertainty allowance for every coding benchmark. It establishes that execution conditions affected this comparison. Before interpreting a small lead, inspect memory, time limits and their enforcement, alongside prompts and tools. If an evaluator cannot report those conditions, the result provides weaker evidence for choosing between closely scored systems.
Even with matched conditions, a small lead needs an uncertainty check. Inspect the task count, variation across repeated runs and reported uncertainty in the difference between systems evaluated on the same tasks. Ask whether the estimate covers repeated execution on a fixed task set, sampling different tasks, or changes from optimizing the system. In AI Agents That Matter, Appendix B reports repeated-run error bars that exclude optimization and resampling variability. Those intervals cannot describe every source of variation in a new evaluation.
For an application with a strict response deadline or restricted network access, test within those limits. A permissive benchmark configuration can still be a valid capability test, but it answers a different operational question.
Real repository issues do not automatically supply fair tests
Repository benchmarks have an advantage over answer-only tests. Generated code must execute. Their tests can still disagree with the task a developer was asked to complete.
OpenAI’s February 2026 SWE-bench Verified audit examined 138 tasks that its model did not consistently solve. It reported material issues in 59.4% of that selected subset. One example required an unmentioned function name in the tests, rejecting otherwise valid implementations. The percentage describes the audited difficult subset, not a random estimate for the entire benchmark.
OpenAI then recommended SWE-Bench Pro. Its July audit of Pro’s public split identified problems there too, including tests that reject acceptable solutions and tests that allow incomplete fixes. It retracted the earlier recommendation. Harder tasks had not eliminated the measurement problem.
Repairs are underway. The September preprint SWE-Bench Pro Verified, from Shanghai AI Laboratory and collaborating researchers, proposes revised tasks and restrictions on access to solutions. It is a separate release, not proof that every previously reported Pro score has been corrected. Its authors also acknowledge remaining task-quality and isolation limits.
Separately, Scale’s September 22 public V2 update reduced the set from 731 to 642 tasks and describes restricted network access, regrading on pristine images and checks that reference patches pass while empty patches fail. Those are maintainer-reported repairs, not our independent validation. The update still identifies patch-controlled code executed by the verifier as an unresolved risk. Its V2 protocol and the separate Verified proposal should not be conflated, nor should old Pro scores be relabeled V2.
These cases support checking a few instructions, reference solutions and tests before relying on a result. A wrong rejection conceals capability. A weak test can reward a patch that leaves the requested behavior unfinished. Benchmark names and labels such as “Verified” cannot replace that inspection.
Training exposure and access to the solution need different controls
A published problem may appear in model training. OpenAI’s Verified audit found examples of tested models reproducing reference patches or task-specific details. That evidence supports concern about exposure on those examples. It does not establish that every correct answer was memorized or that exposure was deliberate.
An agent can also find a solution while taking the test. Later Git commits, a reference patch inside its container or a reachable public repository can reveal the answer. This is execution-time leakage, even if the model never saw the task during training.
The two problems require different controls. New or withheld tasks can reduce training overlap. Repository reconstruction, hidden evaluator files and restrictions on solution access address leakage during execution. The Pro Verified proposal addresses the latter through its environment controls; it does not certify the contents of model training data.
Restrictions need to match the claim. If the test is meant to measure independent issue resolution, downloading the known fix defeats that purpose. If the product is intended to find and apply documented fixes, finding one can be legitimate work. Evaluate that product for correct applicability, permissions and execution rather than calling every successful lookup cheating.
A grader can change the ranking without changing the agent
Automated grading makes large evaluations possible. Neither rules nor model judges are automatically authoritative.
In AgentRewardBench, researchers compared web-agent evaluations with expert labels. They describe a rule rejecting an answer that correctly names Acadia National Park because the response includes an explanation instead of the exact configured string. Table 3 also shows a selection consequence. On WebArena, expert annotations rank the GPT-4o agent above Qwen2.5-VL, whereas rule-based scores reverse that pair on the same collected trajectories.
The size and direction of grading errors depend on the systems and tasks. In this study, rule-based scores were generally below expert-annotated success, with a zero tie for Llama 3.3 on VisualWebArena. The GPT-4o judge generally overestimated success, with two exceptions in WorkArena++. Neither pattern establishes a universal direction of grader bias.
The appropriate correction depends on the task. Answer equivalence can prevent a false rejection. Checking resulting records can reveal a failed action despite a convincing completion message. A grader that receives only that message cannot inspect a database change it never sees.
Read what the evaluator observes and how its decisions were checked. Expert-written tasks establish a different kind of evidence from expert validation of the grader. Agreement between two model judges also leaves open whether both accept the same incorrect answer. For local use, inspect false acceptances and false rejections on the failures and valid alternatives your work actually contains.
Repeated attempts improve discovery, but still require selection and payment
The original tau-bench paper considers independent trials of the same task under fixed conditions. It distinguishes pass@k, the chance that at least one of k trials succeeds on a task, from pass^k, the chance that all k trials succeed, averaged across tasks. Finding one successful answer and repeatedly resolving the same request are different uses. Adaptive retries that use feedback from earlier attempts change those conditions.
A system allowed several attempts also needs a way to choose its final answer. An evaluation that can identify the successful attempt from a hidden reference does not give the deployed system that reference. Measure the actual selection procedure, including its mistaken choices and checking effort.
The study AI Agents That Matter makes the case for comparing accuracy with inference cost and shows why developers’ procurement questions differ from model researchers’ questions. Its 2024 prices are not a current buying guide. Retain token usage and tool calls so costs can be recomputed for the configuration being considered.
For workflow selection, extend the accounting to failures, review and repair. Include unfinished requests as well as delivered answers. Otherwise a low cost per successful output can conceal work transferred to staff or left in a queue.
Read a published result in the order your decision needs
Before putting a system on your shortlist, record the following:
Work and inputs. What does each task require, and what evidence is supplied? Check the domain, language, document formats and how tasks were selected. A deliberately difficult test diagnoses limitations without estimating their frequency in ordinary intake.
Execution conditions. Identify the model and harness versions, tools, permissions, environment, resource limits and number of attempts. Distinguish controlled model comparisons from comparisons of different complete systems.
Success and scoring. Inspect valid alternatives, mandatory requirements, grader-visible evidence and checks against independent labels. Find the denominators for failures, refusals, invalid outputs and grader errors. Inspect task counts, repeated-run variation and uncertainty in shared-task score differences, including which sources of variation the estimate covers.
Transfer and cost. Name the conditions your workflow changes. Ask which outcomes remain untested and whether published costs include the effort needed to finish every request.
Missing information need not disqualify an otherwise promising candidate. It limits the decision the evidence supports. A score with an opaque harness might justify a small local trial, but not a conclusion about why the model wins.
Build that trial around representative requests, with held-out examples and independent references. Compare the actual configured systems under the same service requirements, including the existing process. Add targeted cases for consequential failures, keeping their results separate from ordinary-request rates.
The result of reading a benchmark should be a specific next test or selection decision. Broad knowledge evidence can narrow model candidates. A tool-use result can justify trying a system in a sandbox. Neither grants permission to act on business records. That permission needs evidence about the system, data and constraints that will actually be used.