Production agent monitoring vs offline evals. The offline bank catches known breaks; step monitors, deterministic checks, human reads, and random A/B audits catch drift, and eval-only shipping is earned per change class

The last lesson built a test suite you run before shipping an AI agent. This one is about the day after, and the month after. Agents drift. Traffic changes. And the scariest failures don’t crash; they quietly produce confident, wrong output on schedule.

Offline versus online

Offline evals run a fixed set of test cases with known right answers on every change. Online evals score samples of real live traffic, where there’s no answer key, so they rely on code checks, judges that don’t need a reference, and user signals.

LangChain’s State of Agent Engineering survey: 89% of organizations have observability (they can see traces), 52% run offline evals, and only 37% run online evals. Most teams can watch their agent and still can’t say whether it’s getting worse.

Three layers of production signals

1. Step telemetry (cheapest). A 2026 paper on real-time failure detection trained tiny monitors on healthy runs only, using tool-call timing and step patterns. On 2,823 episodes they caught 71% of failures at a 5% false-alarm rate, at around 200 microseconds per step. The catch: moved cold to a new deployment, accuracy (AUROC) fell to 0.527, near a coin flip, versus 0.885 once retrained on that deployment’s own normal traffic. Every deployment needs its own baseline.

2. Deterministic checks (middle). The same paper recomputed each run’s stated total from the tool results it actually got and confirmed every required call happened. That caught 60% of failures with zero false positives in 63 tries and fired on none of 1,825 healthy runs. Re-running flagged runs lifted task success from 52% to 73% for about one extra model call. OnTrack does similar checks at about a millisecond per step and stops doomed runs early, saving about 18% of compute on SWE-bench, where 5 of every 6 stops were runs that would have failed anyway.

3. Humans and outcomes (slowest, catch the weirdest stuff). Wei Wu logged 22 silent-failure incidents over eight weeks in a production personal-agent system that had 4,286 unit tests and 827 governance checks. About 70% were caught by a person reading the agent’s actual output. The governance layer prevented 0 of 15 audited incidents beforehand and blocked 87% from recurring after. Silences lasted from 13 hours to 60 days. His name for the worst kind: “fail-plausible,” where an error leaks in and the model turns it into a confident, false summary that ships on time.

Why old test results expire

micro1 studied 2,611 human-scored production interviews from a voice agent. Two reviewers scoring the same batches differed by 0.79 standard deviations, and when the stricter one joined, a real improvement looked flat. Their estimate: after about five weeks without re-measuring, you can’t tell a major change from noise (a wide interval, roughly 1 to 21 weeks). Support tickets falling about 10% a month is how they knew the improvement was real.

When can a test score replace a live experiment?

Mårten Schultzberg at Spotify argues you need a track record: past changes where the offline score predicted the live result, within one type of change (prompt tweaks, retrieval changes, model swaps). On 183 real headline experiments, an AI judge measured precisely and still barely predicted clicks (corrected R² of 0.023), so every change went to an A/B test. Two numbers stick: if the judge is 3 points less sensitive to the new version, a real +2.0 point gain reads as -0.25. And in a simulated break, monitoring only the remaining A/B tests noticed it 1.7% of the time, while randomly A/B-auditing 40% of would-be launches caught it 96% of the time.

The counterexample is Nubank: across eleven versions of their support agents, offline improvements tracked live satisfaction, ending at +37 points tNPS and +29 points self-service on card delivery. One domain, one type of change, many paired readings.

An analogy: the restaurant inspection

A health inspection before opening is your offline test. It catches the problems inspectors know to look for. But a restaurant that passed in March can be a mess by June. Good owners also taste dishes every night, watch for slow tickets, read reviews, and send a mystery diner now and then. The mystery diner is the random audit.

Try this

Pick any AI tool you use weekly. For two weeks, save one real output per day in a note (a summary, an email draft, an answer). At the end, read them all in one sitting. Did quality drift? Were any confidently wrong? That’s a “human read,” the layer that caught 70% of silent failures in Wu’s study.

Use it at dinner

“Testing an AI agent before launch is like a health inspection. You still need someone tasting the food every night.”