Blog

Why we label every span instead of a sample

Alex Li

Oct 29, 2026

TL;DR

The standard advice is to run your LLM judge on a sample of production traffic (MLflow recommends 5% to 20% for expensive judges), because every judge call costs money and is slow.

A sample is good for estimating how often something happens. Grouping runs by pattern, or tracing a pattern back to a commit, needs a label on every run in the group.

We label every span with a small language model we built in-house. It takes under a second per span with no per-call bill, so your team decides how much traffic gets a label.

To evaluate every span instead of a sample, the labeling step has to be cheap and fast on every call. A large LLM judge is billed per call and is slow, so the standard guidance is to sample 5% to 20% of traffic and accept that most runs never get a label. A small language model trained for the labeling job takes under a second per span, with no per-call bill, and every run arrives with intent, sentiment, failure and hallucination labels attached.

Key facts

Databricks’ MLflow docs recommend sampling 5% to 20% of traffic (a sample rate of 0.05 to 0.2) for expensive scorers such as complex LLM judges, and 100% for critical checks like safety Databricks docs, 2026.

Depending on the judge model and the size of the input, Langfuse estimates an LLM-as-a-judge evaluation costs $0.01 to $0.10 per assessment, and lists sampling as the first way to manage the cost Langfuse FAQ, 2026.

In OpenTelemetry, a trace is a tree of spans, one per step, so a single agent run produces many spans to label OpenTelemetry docs, 2026.

Why teams sample

Say your agent produces 100,000 spans a day. Depending on the LLM model and your span size, Langfuse estimates the cost is $0.01 to $0.10 per assessment (Langfuse FAQ), so a judge on every span costs $1,000 to $10,000 a day. A 5% sample brings that down to $50 to $500. (Illustrative volume; the per-call range is Langfuse’s.)

Each judge call is also slow, since it is a full request to a large model, and 100,000 of them a day is a long queue.

At those prices and speeds, sampling is the right call, and it is what the tooling recommends. MLflow’s production monitoring guide says it directly: low rates for expensive judges, full coverage only for the checks you cannot afford to miss.

Sampling also works well for one job. If you want to know roughly what share of answers are unhelpful this week, a random 5% gives you a reasonable estimate.

What the sample costs

The trouble starts when the question changes from “how often” to “which runs”.

Take a failure that hits 1 in 1,000 runs, on an agent that handles 10,000 runs a day. That is 10 failing runs a day. A 5% sample labels about one of them every two days. On the dashboard it looks like noise, and each occurrence the sample does catch arrives on its own, with nothing linking it to the other nine. (Illustrative numbers.)

Three things follow:

Rare patterns stay invisible. The failure is real and repeats every day, but it never collects enough labels to show up as a pattern.
Grouping breaks. To group runs by execution path (which tools, in what order) and color each group by what went wrong, every run in the group needs a label. A group that is 95% unlabeled has no color.
Nobody chose the coverage. The sample rate was set to keep the judge bill reasonable. It was never a decision about which failures the team was willing to miss.

So the real question is: what would it take to label everything, without the judge bill?

Label every span with a small model

Our answer is Polaris, a small language model we built in-house to annotate agent spans. It takes under a second per span, and it is Kubit’s own model, so there is no third-party LLM credit bill behind it. Polaris annotates every span by default, so the sample rate stops being a line in the judge budget.

Polaris does three things:

Label. The Polaris Classifier labels every span with intent, sentiment, failure and hallucination, including the spans that return OK. It also annotates each span with four types: Action (an LLM, RAG or TOOL call), Exception (the call failed, such as a 500 or a timeout), Misstep (every call succeeded, but the agent did the wrong thing, such as passing a wrong parameter) and Warning (a loop, high cost, high latency or PII).
Fine-tune. Correct the labels that don’t fit your product, and Polaris Fine-tuning trains a custom classifier on your corrections.
Evaluate. Write an eval prompt for a check specific to your product, and Polaris scores every span against it as a number, a boolean or an enum. Evals run on the same small language model, not an LLM judge, at no extra cost.

	Large LLM judge on a sample	Small language model on every span
Coverage	5% to 20% of traffic, typically	Every span, including OK ones
Speed	A full LLM request per span	Under a second per span
What decides coverage	The per-call bill	A choice you make
Good for	Estimating a rate	Estimating a rate, grouping patterns, finding the cause
Rare failures	Mostly unlabeled	Labeled every time they occur

Why the rest of the product depends on it

Labeling every span is what makes the other parts of Kubit work.

Flows groups every run by its execution path and draws the population as one Sankey. The labels travel with each group, so you can see which paths fail, which loop, and which leave users unhappy. (That was last week’s post.)

Triggers looks for the signal behind a pattern. One of the signals it reads is which label cluster a pattern sits in, and that only works when every span carries a label.

All of it runs as queries inside your own Databricks, Snowflake, BigQuery or ClickHouse account, on the traces already there. There’s no SDK to install and no pipeline to build.

When sampling is still the right call

If all you need is a weekly quality number, a sampled judge is fine. The same goes for a judge you run during development, where you change the prompt and re-score a few hundred examples. Keep using it there.

Label every span when the job is finding the runs: grouping a pattern, sizing it, and tracing it back to what changed.

See it on your own traces

We’re opening Kubit to a first cohort of teams running agents in production, with traces in Databricks, Snowflake, BigQuery or ClickHouse. Request early access, or book 20 minutes with me and I’ll walk through it on your own traces.

Previous post: Why we group agent runs by the shape of their execution path.

FAQ

How much does LLM-as-a-judge cost?

It depends on the judge model and the size of each input. Langfuse estimates a typical evaluation costs $0.01 to $0.10 per assessment (Langfuse FAQ). Multiply by every span your agent produces in a day and the full-coverage bill is why the standard advice is to sample.

What sample rate do teams use for LLM judges in production?

Databricks’ MLflow docs recommend sampling 5% to 20% of traffic for expensive judges (a sample rate of 0.05 to 0.2), 30% to 50% during development, and 100% for critical checks such as safety.

Why not just sample?

A sample estimates how often something happens. It doesn’t label most of the runs, so rare failures rarely collect enough labels to show up, and you can’t group runs by pattern when most of each group is unlabeled.

What does Polaris label?

Every span, including spans that return OK. The Polaris Classifier labels intent, sentiment, failure and hallucination, and annotates each span as an Action, Exception, Misstep or Warning. You can fine-tune the classifier on your corrections, and Polaris Evals can also score every span against your own eval prompts.