AgentBench Tasks and What They Reveal About Real-World Agent Reliability

AgentBench tasks reveal why a capable language model is not necessarily a reliable agent. Across eight interactive environments, success depends on sustaining a plan, following rules, and responding to feedback. Reliability is a profile of strengths and failures. For organizations deploying agents, that profile is more useful than a leaderboard rank, but it cannot replace runtime oversight.

Why static benchmarks cannot measure agent reliability

A single-turn question tests whether a model can produce the right answer with the information available. It does not test whether the model can keep pursuing a goal as its actions change the environment and new information arrives. That is a different kind of examination. An agent must choose a step, work out what the result means, and decide whether to go on or backtrack, while keeping earlier constraints in mind. Things can go wrong in that sequence even when the model scores well on language-understanding tests.

SAP Labs’ KDD’25 survey likens testing an LLM to testing an engine, while testing an agent means checking how the whole car works under different driving conditions. A fine engine is not a driving licence. The survey also lists what enterprises need, such as role-based access control, compliance, reliability guarantees, and long-horizon interactions. Static tests cannot capture those demands because they do not put the system through the sustained behavior being assessed.

How AgentBench's eight-environment design forces multi-step, interactive evaluation

Xiao Liu, Hao Yu, and colleagues at Tsinghua University’s THUDM lab introduced AgentBench to test precisely that sustained behavior. Published at ICLR 2024, the benchmark spans eight environments grouped into code-grounded, game-grounded, and web-grounded tasks. Breadth is part of the measurement design. A narrow test can make an agent look capable across the board because it happens to suit one strength, while different environments make it work with different rules, feedback, and kinds of uncertainty.

Operating System, Database, and Knowledge Graph are in the code-grounded group. Operating System tests how the agent handles OS operations, while Database tests how it handles database manipulation, including whether its SQL responses have the right form. Knowledge Graph requires interaction with large-scale APIs, such as Freebase, to answer multi-hop questions under partial observability. Its metric is answer F1 score. The agent therefore has to work toward an answer through successive interactions rather than receive all the relevant information in one prompt.

Digital Card Game tests strategic, turn-based card decisions in the Aquawar framework. Lateral Thinking Puzzles requires the agent to uncover concealed truths by asking binary yes/no questions, making indirect reasoning and uncertainty central to the task. House-Holding, drawn from ALFWorld, uses textual interaction for embodied household goals such as “put a pan on the dining table”. The request is simple. The sequence need not be.

Web Shopping and Web Browsing round out the suite with web-grounded interactions. They extend the assessment beyond code and games into shopping procedures and browsing actions. The infrastructure also separates tasks from one another. A decoupled Server-Client architecture allows agent–environment pairs to run optionally in dedicated Docker containers, supporting reproducibility and preventing cross-task contamination.

What the results across environments reveal about where agents actually break down

AgentBench’s original evaluation found that top commercial models, with GPT-4 as the frontier reference, performed substantially above the open-source field in aggregate. But the size of that advantage varied sharply across environments. The gap depended on what the agent was asked to do and how the environment required it to proceed.

That unevenness makes a reliability profile more informative than a single score. Strong performance in one environment predicted little about performance in another, while fluent chat did not transfer uniformly to interactive tasks. An aggregate compresses these differences into a convenient ranking. Convenience is not the same as diagnosis. Two systems can look broadly comparable at the top level while presenting very different risks for a particular deployment.

Practitioners should compare the target workflow with the environments being tested. Being good at procedural execution does not show that an agent can keep up an open-ended inquiry. Likewise, a broad lead does not establish that the model follows every environment’s action rules reliably.

The failure taxonomy: four categories that name the specific ways agents lose the thread

AgentBench’s failure analysis gives that profile a diagnostic vocabulary. It distinguishes successful completion from four named failure categories, each identifying a different way an interaction can go wrong. These are not interchangeable symptoms. A syntactically broken answer, an unauthorized move within a task, and a repetitive reasoning loop may all prevent completion, but they point toward different remedies.

Task Limit Exceeded, or TLE, means the agent runs out of interaction turns before it solves the task. Liu and colleagues’ paper found that over 90% of TLE trajectories had repetitive generation loops in which the agent made no progress. The agent was busy, but going nowhere. TLE was the dominant failure mode overall and reached especially high rates in Knowledge Graph and Lateral Thinking Puzzles, which demand sustained, iterative reasoning. The diagnosis points toward better planning, backtracking, and loop detection.

Invalid Format, or IF, means the response has the wrong syntactic form, such as missing SQL backticks or incorrect action labels. It was most prominent in Database tasks. Here, the reasoning may be sound while the interface contract is broken, which points toward stronger instruction-following fine-tuning. Invalid Action, or IA, instead means the agent attempts an action outside the environment’s defined action space. It appeared most often in House-Holding and Web Browsing, where the available actions are numerous and partly implicit, pointing toward tighter action-space grounding.

Context Limit Exceeded, or CLE, happens when the agent fills its context window before it finishes the task. This exposes weaknesses in summarization and state management across long interactions. More text is not necessarily more useful memory. Taken together, the categories show why the central breakdown was often poor long-term reasoning, decision-making, or instruction following. Each diagnosis directs attention to a different part of the system instead of treating every failure as evidence that the model needs to be larger.

Why code training produces uneven reliability gains across environments

AgentBench’s comparison of the CodeLlama and Llama-2 series complicates the idea that code training makes a model a better agent across the board. Code tuning helped on tasks with relatively static procedures, such as Web Shopping, where structured, step-by-step execution fits the work. But performance declined on Lateral Thinking Puzzles, which rewards creative, indirect reasoning. The benefit had a particular shape. Training that suited one kind of interaction did not reliably serve another.

WizardCoder supplied a related caution about training format. Its performance fell short of what its code-training pedigree might suggest, and the single-turn format of its training data appears to have hindered multi-turn performance. Knowing the domain was not enough. The form of interaction learned during training also influenced whether the agent could sustain a task beyond its first response.

These findings point toward alignment quality and data diversity as relevant levers alongside code exposure and model scale. They do not establish a universal training recipe. Instead, they show that a training intervention has to be understood through the kinds of behavior it improves or weakens. The reliability profile reflects the training regime as well as the task, so a gain in one environment cannot safely stand in for improvement across all eight.

How the AgentBench framework evolved: the move from scoring to training with AgentBench FC and AgentRL

THUDM introduced AgentBench FC on October 10, 2025, as a function-calling version integrated with AgentRL. Researchers from Tsinghua University and Z.AI developed AgentRL as a multi-turn, multi-task reinforcement learning framework. The change gave the existing infrastructure a second job. Containerized environments used to expose failures during evaluation could also become environments in which agents learned to complete tasks.

The AgentBench leaderboard as of October 2025 showed an RL-trained Qwen2.5-Instruct model scoring 62.0%, competitive with much larger commercial models. The smallest AgentRL model also scored competitively, indicating that the training benefit extended to very small models. Scale was not the only route to improvement. Reinforcement learning on the task environments directly addressed the TLE and long-horizon planning problems that the failure analysis had identified.

Once repetitive loops are distinguished from format errors or invalid actions, improvement can target the behavior behind the outcome. Evaluation can thus guide training as well as test the agent once training is done. These results support that use of the framework, though they do not establish broad adoption of AgentBench FC or guarantee transfer to every production environment.

Where AgentBench stands now and what benchmarks have moved to the frontier

AgentBench is no longer the leaderboard practitioners consult for frontier-model evaluation. Its original suite used GPT-4 as the frontier reference, and newer models have not been systematically re-evaluated against it. Comparisons therefore need care. Published scores also differ in their aggregation, with some averaging only environments where a model scored non-zero and others including the worst-environment scores in a strict mean. The former can inflate a headline result without improving the underlying agent.

Fudan NLP Lab’s AgentGym2, introduced in July 2026, extends the evaluation problem into “de-idealized” environments. It tests how agents handle noisy inputs, underspecified goals, and tools they must find for themselves. The working conditions are less accommodating. The framework finds that even current top systems, including Gemini and GPT-5, struggle under these demands. Its findings extend AgentBench’s argument by showing that reliability remains sensitive to how an environment presents the work.

Purdue’s SLBench, introduced in July 2026, tests a related but different way agents fail to follow skill files. Chen and colleagues found that agents could satisfy main action clauses while omitting postconditions, fallbacks, and constraints, with unsafe rates reaching up to 70% across LLM backbones. Visible completion concealed unfinished obligations. Plausible-looking results could leave sensitive artifacts on disk or omit required cleanup, a failure that AgentBench’s task design did not target.

What the reliability profile means for organizations deploying agents in production

A benchmark turn limit ends an unsuccessful test, but production looping consumes real resources. Detecting it requires runtime monitoring across extended interaction sequences. The unit of observation has to change. A single response may look reasonable even when the surrounding sequence shows that the agent is repeating itself without progress.

Invalid Action failures have a related production analogue in operations attempted outside an agent’s authorized scope. An environment’s action space and an organization’s permissions are not identical, but both require explicit boundaries. Enforcement needs to know what is permitted. Organizations need to flag deviations in real time, while also detecting format brittleness and context exhaustion as agents interact with tools and maintain state. SAP Labs’ survey explains why role-based access control, compliance, and long-horizon work make these failures more consequential in enterprises.

SLBench adds a sharper warning about what counts as success. An agent may finish the visible task but skip cleanup, leaving sensitive files on disk or sessions open. Artifact-level monitoring is needed to expose obligations that remain unmet after the main action succeeds. AgentGym2’s findings further undermine the idea that choosing a sufficiently advanced model will remove the need for this instrumentation.

Benchmark results serve as diagnostic evidence. Organizations need to discover every running agent, monitor all approved traffic, detect anomalous patterns, enforce rules on allowed actions, and produce audit logs of every action. Runtime observability and policy enforcement serve different purposes. The first makes behavior visible, while the second sets boundaries that a capable model cannot be assumed to respect unaided.