89% watch agents fail. Only half test before shipping.

Most teams can see their agent failing in production. Fewer than half can catch it beforehand.

That gap has a number now. It also has a technical cause that is more specific than "evals are hard": the metric most teams use as a release gate is structurally bad at detecting the regressions they care about.

What the survey actually found

LangChain's State of Agent Engineering survey collected 1,340 responses between November 18 and December 2, 2025. Technology firms made up 63% of respondents, and 49% worked at companies with fewer than 100 employees (LangChain, 2026).

The headline split is the one worth quoting. 89% of organizations have implemented some form of observability for their agents, rising to 94% among teams already running agents in production (LangChain, 2026). Offline evaluation sits at 52.4%, and online evaluation at 37.3% in the same survey (LangChain, 2026).

Offline evaluation is testing an agent against a fixed test set before release. Online evaluation is scoring real traffic after release. Most teams have neither gate and a very good dashboard.

Quality was the top reported struggle at 32%, ahead of latency at 20% (LangChain, 2026). Cost concerns fell compared with previous years.

Why your end-to-end pass rate hides the bug

This is the finding that should change what you measure.

The layer-isolated evaluation paper, submitted in June 2026, splits a production agent into eight functional layers, such as routing, escalation, safety and memory. The researchers then broke one layer at a time and watched what each metric did.

The aggregate pass rate barely moved. It fell between 1.7 and 5.9 percentage points. The test slice matching the broken layer fell between 25 and 91 percentage points.

Think about what that means for a release gate. Ship a change that breaks your routing layer, and an end-to-end suite reports a couple of points of noise. You would approve that. The same run, sliced by layer, screams.

Fault localization held up too. The damaged layer's slice was the worst-affected slice in 5 of 7 cases and in the worst three in all 7, with a mean rank of 1.29 out of 19 slices.

Test the deterministic parts with no model at all

Here is the part teams skip because they assume every agent test needs an LLM call.

Much of an agent is ordinary software. Routing rules, schema validation, escalation thresholds, safety filters, memory writes. None of that is non-deterministic, and all of it can be tested the way you test any other code.

The same paper reports what that costs: 238 cases across 23 slices, of which 225 run in 2.39 seconds, at roughly 10 milliseconds per case. No model calls, no judge, no flake.

Consider how that changes your CI. A suite like that runs on every commit. An LLM-judged suite runs nightly at best, because it costs money and time per case. Build the cheap layer first and reserve the expensive one for what genuinely needs it.

Where an LLM judge earns its cost

LLM-as-a-judge is using a model to score another model's output against a rubric. It is the standard answer for anything you cannot check with an assertion.

It is much better than the lazy alternative. In an audit of tool-using agent evaluation, substring-matching heuristics scored a Cohen's kappa of 0.049 against each of two human annotators. A single GPT-4o-mini judge reached 0.567. Two human annotators agreed at 0.835.

Cohen's kappa is an agreement score that corrects for agreement you would get by chance, where 0 is chance-level and 1 is perfect. So substring matching was barely better than guessing, a cheap judge was usefully better, and humans were still clearly ahead.

The practical read: use assertions where an assertion works, a judge where it does not, and human labels to check the judge. That trace-level split is also how a practitioner guide to agent evaluation frames the same problem.

The judge failure that flips your conclusion

A judge that is merely noisy is manageable. A judge that is confidently backwards is not.

A May 2026 study of judge bias shows that raw judge outputs are biased, and that bias-correction methods have limits. Its sharpest finding concerns calibration shared across the models you are comparing. That can introduce severe bias, including cases where the comparison estimate points in the wrong direction.

The paper demonstrates this sign reversal on real MMLU-Pro data. The estimated direction of a model comparison flipped because calibration was unstable across models.

So if you use a judge to pick between two candidate agents, and you calibrate that judge once and reuse it for both, you can end up shipping the worse one while your report says otherwise. Validate the judge against human labels per comparison, not once at the start. 1,600 Court Cases of Fake AI Citations. One Cause. is the same shape of failure in a different setting: a trusted check that nobody checked.

One bad parameter usually becomes one wrong answer

This connects testing back to the most common agent bug: the model invents or mangles an argument to a tool.

The tool-using agent audit measured how often that turns into user-visible damage. A parameter-level error propagates to a wrong final answer with a human-calibrated probability of about 0.62, across both proprietary and open-weight models. The benchmark behind it covers 14,750 execution traces from 13 agents.

Roughly three in five bad parameters become a bad answer. Recovery is not free, and the paper notes that rejecting a corrupted input and recovering from one are separate abilities.

A lightweight runtime interceptor cut fabricated tool executions by up to 24 percentage points at a configurable operating point. That is a runtime guard, not a test, and it is worth having alongside one.

Layer	What it catches	Cost per run
Deterministic slice tests	Routing, schema, safety and memory regressions.	Milliseconds, no model call.
Assertion-based tool checks	Wrong tool, malformed arguments.	Cheap.
LLM judge on sampled cases	Answer quality and tone.	One model call per case.
Human labels on a calibration set	Whether the judge itself is trustworthy.	Slow, and unavoidable.

A test plan you can ship this week

1. Slice your suite by component, not only by task. Report per-slice pass rates next to the aggregate, and gate on the slices.
2. Write deterministic tests for everything that is not the model. Routing, schemas, thresholds and filters are ordinary code.
3. Assert on tool calls. Check the tool name and the argument shape before you judge the prose.
4. Sample for the judge. Full judged reruns on every commit are the reason teams stop running evals at all.
5. Keep a human-labelled calibration set and re-check the judge against it whenever you compare two agents.

Unbounded agents are expensive to get wrong. Can AI Agents Run a Business? It Lost $447 makes the point at product scale, and a pre-deploy gate is the cheapest place to catch what causes it.

The decision, in one line

If you have observability and no offline evals, do not start by building a judge. Start by slicing your suite so a broken component shows up as a broken slice, because your end-to-end number will move by three points when something real breaks.