AI agent benchmark

Agent

An agent is not a model, but an assembly of parts. An agent is a model combined with a harness. Changing the model or the harness can both change the benchmark results significantly. The exact prompt used for the agent can also have an effect.

The harness runs locally hosted, allowing complete control by the benchmarker. However, the model may not be locally hosted. Some are proprietary, while others may be open-weights, but too large to host for the benchmarker. Consequently, these models must be accessed via an API. Changing the API access point can change the benchmark results significantly.

Evaluating

An agency benchmark needs to have a way to score agent outputs. There are certain types of scorers, each having their own problems:

• Automatic verifier: a program that checks the output deterministically. It can be checking that the agent output is verbatim equal to the right answer, or running the agent output source code into a program test harness, or some other method. However, this cannot be done for many types of more qualitative evaluation, such as checking whether a trajectory cheated.
• AI verifier: the output is given to an AI to judge. May be not deterministic, thus less replicable. Normally, this is done by LLM-as-a-judge. The judge is usually a frontier model, since it is hard to qualitatively judge the work of someone with more capacity. However, the frontier moves, so the verifier moves, making past and future benchmark scores less comparable. Furthermore, the choice of the judge can significantly affect benchmark scores.
• Human verifier: the output is given to a human to judge. The most expensive and the least replicable. Rarely used.

An agent is defined by the ability to perform multi-step actions, and a complete multi-step action is called an episode, a rollout, or a trajectory. Because of this, the length of an episode is a critical measurement for agency. It is called the time horizon, a terminology that came from Markov decision process in reinforcement learnig. Time horizon is not a single measure, but a cluster of measures: the number of rounds, the number of tool-calls, the wallclock time, the number of tokens, and so on. Which time horizon to choose is a design question, so different benchmarks may choose differently.

Sometimes, not only the output, but also the agent episode itself is subject to scrutiny. For example, ExploitGym uses an LLM to check that the agent has obtained the answer in the intended way.

Construction

Agency benchmarks significantly more costly to construct.

Unlike standard benchmarks, in an agency benchmark, a trial runs in a loop of action-observation. The agent takes in an observation and produces an action, while an environment takes in an action and produces an observation. Thus, unlike all other benchmarks, agency benchmarks cannot be specified by static data (such as a textual description of the task), but require an interactive environment, which is specified programmatically. The environment usually consists of several parts. Common components of the environment include a code repository that the agent should work on, utility programs the agent might use, and a runtime environment for running the entire trial. Each task is often containerized to allow exact replicability.

It is difficult to programmatically evaluate performance on many agentic tasks. The agent might solve the task in an unintended way, or fail the evaluator despite actually solving it in an intended way. Consequently, the trial rollouts are often inspected for phenomena such as reward-hacking behavior or broken evaluator.

In some reported cases, complete mock-ups of user interfaces have been produced in order for both training and benchmarking. For example, it was reported in early 2026 that dedicated companies would produce such mock-up for $20k each, and up to $300k for highly complex ones such as a mock-up of Slack.

Some benchmark items have human baselines, and human baseline time horizons. They are usually interpreted as indicators of how difficult the item is. Measuring a benchmark item's human baseline is costly, and the harder the item is, the more costly.

• human baseline: How well humans performed on the same test item, under conditions close to the agent's test condition.
• human baseline time horizon: How long it would take a human to complete the same test item. Can be statistically estimated by measuring several human baselines. Some of the longest time-horizons are estimated using without measurement, since they are too expensive for the benchmark constructors to measure even once (on the order of weeks or more).

Running

Agency benchmarks are also significantly more costly to run.

As short-horizon tasks saturate, more difficult benchmarks are constructed that test for longer-horizon tasks. This means the agent-environment interaction takes longer. This has increased the time-per-trial from several minutes to several hours.

At the beginning, language model would take a query and produce an answer immediately. With the invention of chain-of-thought reasoning in language models, a language model can be prompted repeatedly for many tokens before producing a final answer. This means the cost per answer is variable. Thus, a higher cost tends to produce an answer more likely to be correct, and thus a higher benchmark score. This has two effects:

1. When running a benchmark on an agent, the task cost itself is recorded, and agents are compared for not just their benchmark performance, but on a cost-performance curve.
2. The same agent is benchmarked multiple times at different task cost levels. This makes running a benchmark more costly.

There are several types of "cost" usually used: the wallclock time, the output's token count, and the monetary cost.

Some of the benchmark items are simply difficult to run, since they may require extensive or difficult setup. For example, even OpenAI was only able to benchmark 477 of the 500 items in SWE-bench Verified when benchmarking o3 and o4-mini, due to infrastructure challenges.

As frontier model capability increases, agency benchmarks are increasingly subject to unique problems not found in other types of benchmarks.

• In certain areas, frontier agents have saturated essentially all benchmarks. This is particularly true of mathematics, where frontier progress is now measured by solving open problems.
• Frontier agents are increasingly evaluation-aware, i.e. aware that they are acting in an agency benchmark. This is similar to observer effect.
• Frontier agents increasingly reward-hack, acting to increase scores in a way that violates the task requirement. This is similar to Goodhart's law, but done by the agents rather than the researchers. Examples of reward-hacking range from simply web-searching the benchmark answers (if the agent harness runs in a container that can access the internet), using git to find the correct patch (if repository history has not been cleaned), all the way to the OpenAI–HuggingFace incident.

General

• AgentBench: 1014 test tasks and 269 development tasks in 8 interactive environments (operating system, database, web browsing, games, etc). Each tasks require multiple rounds (estimated 5–50) of interaction.
• Agents' Last Exam: long-horizon, professional computer tasks, currently containing 1500+ tasks collected toward a 5000-task target.
• ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence): Given three pairs of before-and-after diagrams of applying a rule, apply the same rule to the fourth before-diagram. It is similar to a Raven's Progressive Matrices test. Later versions explicitly test for agentic behavior in a game-like interactive environment.
• HCAST (Human-Calibrated Autonomy Software Tasks): 189 tasks in machine learning, cybersecurity, software engineering, and general reasoning. Each task has a "baseline", the measured average time required for a human skilled in the task domains, working under identical conditions as AI agents. The baseline ranges from 1 minute to 8+ hours. Produced by METR in 2025-03.

Tool use

Some agentic benchmarks are not particularly difficult in themselves, and test instead for proficiency in using tools that an AI agent would use, such as web browsing, using a command line interface, calling web APIs, etc.

• BrowseComp: 1,266 questions that requires internet browsing for finding a short factual answer. Adversarial against GPT-4o with and without browsing, OpenAI o1, and an early version of the Deep Research model.
• GAIA: 450 questions with unambiguous answers that require information that can be obtained by browsing the Internet, requiring different levels of tooling and autonomy to solve. Divided into 3 difficulty levels.
• Mind2Web: 2,350 tasks collected from 137 websites, and crowdsourced action sequences. The task is to reproduce the action sequence.
• SpreadsheetBench: 912 real-world spreadsheet manipulation tasks scraped from public Excel help forums, spanning formula writing, data cleaning, filtering and layout edits in diverse formatting. Scored automatically on 2729 test cases at cell-, sheet- and overall levels.
• WebArena: 241 mock-up websites based on real-world websites (Reddit, GitLab, Magento's admin portal, etc), and 812 tasks to be performed on the websites. The tasks include information-seeking, site navigation, and content and configuration operation.
• WebVoyager: 643 multimodal tasks based on 15 task environments, each environment a fully self-hosted popular website, such as GitLab and Wikipedia. Evaluation is by screenshotting the action sequence and asking a vision language model to judge.
• Terminal-bench: A collection of 66 complex tasks in the Linux terminal. Updated in versions.
• OSWorld: 369 multimodal computer-using tasks, involving multiple real web and desktop apps and OS file I/O, in Ubuntu. Each task includes an initial state setup configuration, and is tested by an execution-based evaluation script. 8 requires Google Docs access, and thus are impossible to run in an offline sandbox (necessary to prevent cheating by looking up OSWorld answers online). 30 tasks were designed to be impossible. For these, the success condition is for the agent must state that it is impossible and give up. 43 tasks were also translated to a Windows version (for example, converting LibreOffice to Microsoft Office, etc). No tasks in macOS due to copyright. Cleaned up to OSWorld-Verified in 2025-07. A new version, OSWorld 2.0, was released on 2026-06-26, containing 108 new tasks.
• Windows Agent Arena: 154 multimodal tasks with the same format as OSWorld. Only in Windows.

Economic

Some benchmarks are specifically designed to measure capability of AI agents to perform economically productive tasks that are currently performed by humans. These tasks are usually obtained by taking particular tasks in a job, and specifying them accurately enough so that they can be verified automatically. These benchmarks are usually developed in collaboration with companies in the industry that currently use humans to perform these tasks.

Economic tasks are often classified according to O*NET, a Department of Labor database of occupations in the US economy, including occupational characteristics and worker requirements. Some economic benchmarks include human baselines for not just time-horizon, but also monetary value. For example, a benchmark item for performing a freelance work unit might have a human baseline for how much it pays.

• AutomationBench: a benchmark developed by Zapier, on cross-application workflow orchestration via REST APIs, using 47 real tools across 6 business functions (Sales, Marketing, Operations, Support, Finance, and HR). Based on real monthly tasks executed by Zapier itself.
• APEX-agents: a benchmark showing how leading AI models hold up doing actual white-collar work tasks, such as consulting, investment banking, and law.
• Vals Index, run by Vals. Aggregation of benchmarks across 4 sectors (Finance, Coding, Legal, and Tax) weighted in proportion to its share of U.S. GDP. Intended to measure the potential economic impact.
  • Harvey's Legal Agent Benchmark: 1200 tasks for legal work, across 24 practice areas, evaluated by over 75000 expert-written rubric. The task is to produce legal work (such as legal discovery) based on instruction and a folder of legal documents. Part of the Vals Index
• Remote Labor Index, 240 items, each a self-contained work unit sourced directly from Upwork, a platform for experienced freelance professionals. Constructed by Scale AI labs.
• GDPval: 1,320 tasks evaluating AI agents on real-world knowledge-work deliverables across 44 occupations from the top 9 U.S. GDP-contributing industries. Tasks made by professionals averaging 14 years of experience. The full evaluation is by blinded expert pairwise comparison against human expert deliverables, each comparison costing 1 human hour on average. A gold subset of 220 tasks is publicly available. The public set can be graded by a finetuned version of GPT-5. The held-out set of 1,100 tasks is only available internally in OpenAI, and external models cannot be submitted for grading on this set.
• TAU-bench (Tool-Agent-User benchmark, also written as τ-bench): Two environments (retail, airline booking) that test for an agent to fulfill user instructions, interactively over multiple turns of dialogue. The user is simulated by a language model. Updated to TAU2-bench (τ²-bench), which focuses on telecom applications. Tasks are synthesized by LLM-generated product requirements document, agent database schema, agent tools, and user environments ("mocked phone"). Updated to TAU3-bench (τ³-bench), which adds tests for operate over large collections of internal company documents, and duplex voice.

Programming

SWE-bench is the prototypical agent programming benchmark, and many programming benchmarks follow its templates.

• ScienceAgentBench: 102 multimodal data science tasks, each being a real scientific data-driven discovery problem reframed as a code-generation task. Agents must produce a complete Python program file that implements the task, can run in isolation, and saves its outputs. In the domains of Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. Sourced from 44 peer-reviewed publications that have released their code and data under permissive licenses. Each task is validated by domain experts.
• SWE-bench: 2,294 tasks in software engineering. Each task is to fix well-defined, small issues with clear descriptions, drawn from real GitHub issues and corresponding pull requests, from 12 popular Python repositories. Given a codebase and an issue, the task is to edit the codebase to solve the issue. Most are from the 2018-2023 period. It has 2 subsets: Lite (300 problems that are faster to run), Verified (human-validated subset of 500 problems reviewed by software engineers). Each task is evaluated by checking that certain tests that pass pre-patch still pass post-patch ("pass-to-pass"), and certain other tests that fail pre-patch pass post-patch ("fail-to-pass").
  • The Verified version is deprecated by OpenAI in 2026-02, since many remaining failures reflected flawed or underspecified tests. They recommended the public split of SWE-bench Pro.
  • SWE-bench Pro was released later as an update to SWE-bench, containing 1,865 harder items. This has 3 splits: public, held-out, and proprietary (only available to a select set of early-stage startups).
• Multi-SWE-bench: 1,632 problems across 7 languages: Java, TypeScript, JavaScript, Go, Rust, C, and C++. Similar to SWE-bench.
• SWE-bench Multimodal: a variant of SWE-bench, with 619 task instances from 17 popular JavaScript repositories, each featuring images that are required for solving the task.
• SWE-Lancer: 1,488 freelance software engineering tasks from Upwork. Two types of tasks: implementation tasks (from $50 bug fixes to $32,000 feature implementations), called "IC" (for "Individual Contributor"); "Management" tasks, where the model must choose between technical implementation proposals. A subset of 502 are open-sourced as SWE-Lancer-Diamond. Produced by OpenAI.
• DeepSWE: 113 tasks spanning 91 active open-source repositories across 5 languages. Constructed to be a harder and contamination-free version of SWE-Bench. Every task is written from scratch to prevent contamination. Task prompts are shorter and solutions are longer.
• SWE-Marathon: 20 long-horizon (averaging 27.2M tokens) software engineering tasks.
• FrontierSWE: 17 long-horizon software engineering tasks. Each trial has a time limit of 20 hours. The wallclock run time of the submission is measured. FrontierSWE v2 adds 34 tasks and retires 4 tasks as being unscorable. Each task has a deterministic automated verifier. The wallclock time is replaced by CPU instruction count, heuristically weighted by how costly each instruction is.
• MirrorCode: 25 long-horizon software engineering tasks over 6 programming languages. Each task asks for a clean-room reimplementation of an entire program. The agent has access to a specification of the original program, some test-cases, and can send arbitrary inputs to the original program (black-box oracle). The reimplementation is tested against the original on whether they produce identical outputs. Each task would take a human engineer an estimated weeks to months to complete. 22 tasks were released, the other 3 heldout as private test set. Developed by EpochAI and METR.

Reinforcement learning

There are many robotics benchmarks, though they were normally used to benchmark performance of algorithms specifically designed for robotics. Relatedly, there are many reinforcement learning benchmarks, often game-like. Though they were normally used to benchmark performance of algorithms specifically designed for general reinforcement learning. Consequently, they had not been used to benchmark agents such as those of the GPT series or the Claude series, because these agents were originally descended from agents that could only take textual input. Other modalities were added later, creating agents based on vision-language models or multimodal models. As an example, classic games like Minecraft had been used as a benchmark in late 2010s for deep RL agents, and more recently they had been used to benchmark agents based on language models.

Such agents's capability of understanding visual modalities has continuously increased, and they have become increasingly capable and exhibited emergent capabilities in many fields, allowing them to be used even for robotics control. Consequently, such agents have been benchmarked on these benchmarks since 2026.

• NetHack Learning Environment: the task is to play the game NetHack (version 3.6.7). Performance is scored by various achievements in the game, all the way to beating the game.
• Robocurve: an organization that runs robotics benchmarks on primarily language-based multimodal agents, such as OpenAI Astra.
• CUA-Bench: 6 tasks, each task is to make progress along a commercially popular video game. The agent observes the video game environment only via screencap, and acts on the environment only via keyboard and mouse events.
• Vending-Bench: the agent runs a simulated vending machine with a given initial simulated budget of $500, and must attempt to make as much money as possible within 2000 turns. Agent can search the Internet and send emails. Emails are intercepted for a GPT-4o to write simulated replies. Sales are simulated by a programmed equation. Updated to Vending-Bench 2.

Others

• Continual Learning Bench: a benchmark for continual learning. Each task consists of several related subtasks. The agent performs the subtasks in sequence. Some subtasks are generated programmatically while others are written by hand. The sequence in which the subtasks appear may have some non-randomness in it. For example, in Exploitable Poker, the agent repeatedly play against an exploitable opponent program. In Database Exploration, the agent have to answer 20 questions by querying an SQLite database with obfuscated keys, then answer 20 questions on a similar but different database.