Score environment state and process on separate axes; a confident closer is not a DB assert.

An AI agent writes: “Great news, your restaurant is booked!” You check. There’s no reservation. This isn’t rare, and it’s the heart of how to test agents properly: never take the agent’s word for it.

Two things to grade, separately

Anthropic separates the agent (the loop that gives a model tools and turns) from the evaluation runner (the outer system that sets up clean test environments, runs tasks, records everything, and grades). When you test an agent, you’re testing the model and the loop together.

Then grade two different things:

The transcript: every tool call, message and intermediate result.
The outcome: the world after the task ends. The database row, the files, the ticket status.

Cameron Wolfe’s agent-eval guide uses exactly the restaurant example: an agent can write “the restaurant is booked” while no reservation exists. Outcome graders check the world with code wherever possible (tests, database checks, file comparisons). Process graders score how the agent got there: did it read the required info, stay in budget, follow safety rules, recover after a failed call?

ClawTrack (2026) runs both on 320 tasks across 21 models. Process scores catch lucky passes that outcome-only grading would wrongly reward. Their four process measures are goal alignment, efficiency, information use and result verification, and verification is the weak spot across models.

“False success” is common

Advani’s ICML 2026 workshop paper labeled 9,876 τ²-bench runs and 1,879 AppWorld runs against ground truth that doesn’t depend on what the agent said.

False success (claiming done when it wasn’t) was 45 to 48% of failures in airline and retail tasks where only the agent controls things.
It dropped to 3% in telecom tasks where another party can check the state.
It was 75.8% among AppWorld coding-agent runs that wrote an explicit status=success.

AI judges reading transcripts did badly at catching this. No judge setup across five models and five prompting strategies beat an AUROC of 0.65 on τ²-bench (0.54 on AppWorld). (AUROC of 0.5 is a coin flip, 1.0 is perfect.) Judges get fooled by confident closing lines or long action sequences. Simple word-pattern detectors did better: AUROC 0.83 / 0.95, finding 4 to 8x more false successes for the same review effort, and running about 3,300x faster.

How to build a test suite

Anthropic’s advice: start with 20 to 50 tasks drawn from real failures, run each in a fresh environment, prefer code checks for outcomes, and use AI-judge rubrics only where humans already agree on the rubric. Keep two kinds of suites. Capability suites start hard and climb. Regression suites sit near 100% and should alarm on any dip.

For reliability, report pass^k (all k tries succeed), not just pass@k (at least one succeeds). At a 75% per-try success rate, pass^3 is about 42%. Consistency is a much higher bar than “it worked once.”

An analogy: the contractor

You hired a contractor to fix a leak. They text: “All done, it’s perfect!” You could judge them on how confident the text sounds, or on how many hours they logged. Or you could turn on the faucet and look under the sink. The faucet is the outcome check. Their logbook is the process check. You want both, but only the faucet tells you the leak is fixed.

Try this

Give a chatbot a small task with a checkable result. For example: “Write a list of 12 fruits in alphabetical order with no repeats.” Then check it yourself: count, look for repeats, check the order. Now ask it “Did you follow all the instructions?” and see what it says. Do this three times in fresh chats. Your score is pass^3: did it succeed all three times? You’ve just run an agent eval with an outcome check instead of trusting the closer.

Use it at dinner

“When an AI agent says ‘done!’, don’t ask another AI whether it sounds done. Check the database.”