Every trace looks green and the agent is still failing
Alex Li
|
TL;DR
A span's status records whether an operation threw an error. It can't tell you whether the answer was right.
The failures that cost trust don't get thrown. The agent succeeds at the wrong task or gives a confident wrong answer. Evals only check what you knew to assert, and in production they usually only run on a sample.
Label every span for intent, sentiment, failures and hallucinations, so a failure with a clean status still has a name you can count.
If every trace looks green and your agent is still failing, the failures are the kind that never throw exceptions. A span's status only records whether the operation raised an error, so a wrong answer from a call that completed looks healthy. To catch these failures, give every span a label that says what happened in it, then count those labels across all your runs.
Key facts
In OpenTelemetry, a span's status defaults to Unset, and instrumentation libraries "SHOULD leave the status code as Unset unless there is an error" OpenTelemetry trace API spec, 2026.
Databricks' production monitoring guide recommends sampling 5 to 20% of traces for expensive scorers such as complex LLM judges Databricks docs, 2026.
Snowflake's answer correctness metric compares the agent's final response with a ground truth answer you prepare in advance Snowflake docs, 2026.
Every span says OK
You open the trace. The planner ran, retrieval came back, every tool call returned, and latency was normal. Nothing set an error. By every measure the trace view shows, the run worked.
And the user still got the wrong answer.
The failures that cost trust don't throw exceptions
A status tells you whether an operation raised an error. It has no idea what the answer was supposed to be. So the failures that cost the most trust pass straight through:
The agent succeeds at the wrong thing. A refund request gets handled as an exchange. Every step completes.
It gives a confident wrong answer. Retrieval returned the wrong document, and the model summarized it fluently.
It does part of the job and stops. A tool returned an empty list, which is a valid response.
Engineers running agents in production describe it in their own words. One counted 0.4% of tool spans with an error status while about 70% of runs produced nothing useful: "all green, everything broken".
Evals catch what you knew to look for
Evals do their job well. A dataset of known cases with expected answers catches regressions on things you've seen before, and every team running agents should keep one. Snowflake, Databricks and the eval tools all support this.
The limit is in the design. An eval can only check what someone wrote down. As one engineer put it, "evals only catch what you already knew to assert. the ones that bite are the checks you never wrote".
In production there's a second limit. An LLM judge is a model call per judgment, so it usually runs on a slice of traffic. Databricks' own guide suggests 5 to 20% for the expensive ones. The failure nobody asserted, in the runs nobody sampled, gets found by a customer first.
So how do you catch a failure that returns a clean status?
Label every span
Classify and label every span, so a failure that never throws still has a name you can count.
A label sits on the span itself and describes what happened there: what the user was trying to do, how they felt about it, whether the step failed at its job, whether the answer was made up. It's separate from the status. A span can say OK and carry a failure label at the same time, and that pair is exactly the case your dashboards miss today.
Once every span has a label, the analytics questions open up. What share of runs carried a failure label this week? Did it jump on Tuesday? Which intent fails most? The whole point of analytics is that you can see it, and a status column only lets you see the crashes.
Span status: checks whether the operation raised an error. A confident wrong answer shows OK. Covers every span. You can count errors.
Evals: check the cases and criteria you wrote down. A confident wrong answer is caught if it matches a case you wrote. Covers your dataset, or a sample of production. You can count scores on what you tested.
A label on every span: checks what happened in the span: intent, sentiment, failures, hallucinations. A confident wrong answer can carry a failure or hallucination label. Covers every span. You can count every labeled failure, by intent, tool or day.
How Polaris labels spans
Polaris is the small language model we built in-house to classify and label agent traces. It labels every trace and span with intent, sentiment, failures and hallucinations. Kubit reads those traces in place, in your own Databricks, Snowflake, BigQuery or ClickHouse account.
One run, or the thousandth?
A label tells you a run failed. It doesn't tell you whether that's one odd run or the thousandth run of the same shape. That's next week's post: grouping every run by the shape of its execution path (which tools, in what order), so a failure that repeats a thousand times reads as one pattern with a count.
We're opening a first early-access cohort for teams running agents in production, with traces in Databricks, Snowflake, BigQuery or ClickHouse. Applications close October 26. Request early access, or book 20 minutes with me.
Previous post: Nobody should have to open hundreds of agent traces one at a time.
FAQ
Why do my agent traces show OK when the answer is wrong?
A span's status records whether the operation raised an error. In OpenTelemetry, instrumentation leaves the status unset unless there's an error, so a call that completed and returned a wrong answer still looks healthy.
Can evals catch silent agent failures?
Some of them. Evals check the cases and criteria you wrote down, on a dataset or on a sample of production traffic. A failure outside those criteria, or outside the sample, never gets checked.
What is span-level labeling?
Attaching labels to each span that describes what happened in it, such as the intent, the user's sentiment, a failure or a hallucination. Because the label sits on the span, you can count it across every run and slice it by tool, intent or time.
Do I have to move my traces to label them?
No. Kubit queries agent traces in place, inside your own Databricks, Snowflake, BigQuery or ClickHouse account. There's no SDK to install and no pipeline to build.
