Green traces, broken agents: what we showed with Sparta

TL;DR

• On October 28 we ran a live session with Arya Kanna and Saad Asad from Sparta. We took one support agent from testing to production and back.

• The agent refunded the wrong customer. Every span in the trace returned OK, because the tool call succeeded with the IDs swapped.

• Kubit found the failing execution path and the release behind it. Sparta Worlds reproduced the bug against working copies of the tools and turned it into a test that runs on every release.

[VIDEO: session recording embed goes here once the unlisted YouTube link exists]

The refund that looked fine

The demo agent handles support tickets. A customer, 4101, asks for an approved $40 refund on order 9207. The agent calls refund_order, replies to the customer and closes the ticket.

All four spans in the trace came back OK. The model calls succeeded, the tool call succeeded, and the reply went out. What the trace doesn't show is that the agent put the order number where the customer ID goes, and the customer ID where the order number goes. Customer 4101 got nothing. Customer 9207 got a refund nobody asked for.

You would only catch this by opening the right trace and reading the tool arguments closely. There's no stack trace to follow, because the run looks normal from start to finish.

Finding it in production

We sent the demo agent's traces to Kubit. Kubit puts a label on every span when it comes in, from Polaris, our own small language model, or from an eval you write as a prompt.

Then Flows groups every run by its execution path (which tools, in what order) into one chart. In the demo most runs take the same path to a reply. One branch ended at a failed refund_order step, and it stood out from the rest straight away.

Clicking that branch opens Triggers, which compares the failing runs with everything else. The top signal was the release: all 43 failed runs came from release f0c39, at four times the failure rate of the other runs. That tells you where to look, a prompt or tool-description change that shipped in that release.

Reproducing it before it ships

This is where Arya took over. Sparta builds Worlds, working copies of tools like Stripe and Zendesk with mock records and no real customers or money. They pointed the same release at a World with a Stripe copy and a help desk, and gave it the same ticket.

The run reproduced the bug on demand. Worlds compares the records before and after the run, so the swapped refund shows up as a fail even though the agent's own reply reads fine.

One agent, one loop

Arya then turned the pattern into a test written in plain English: fail any refund that doesn't go to the customer and order on the ticket. It now runs on every release and blocks CI when it fails. The fixed release passes every scenario.

Saad also walked through a second World built on Okta, ServiceNow and SAP. You pick the systems, set conditions such as rate limits, and Sparta drafts test cases with success rules from a task description. Each run is saved as a structured log you can hand to your coding agent.

Checking the fix by release

Once the fix shipped, we split the same runs by release in Kubit. On f0c39 the failed refund branch is there. On the fixed release, across 210 runs, it's gone.

The Worlds test keeps checking it on every release after that.

What the audience asked

Does labelling miss things? It can. Flows is the backstop, because a run that went wrong usually takes a different execution path even when no label caught it. When you spot one, you correct the label, and a custom classifier trains on your corrections.

What about cost? Polaris labels every span with a small language model, so there's no reason to sample. Arya added that Sparta's main cost is generating the test cases once, and later runs reuse them.

Can it find the root cause? Triggers finds what the failing runs have in common, such as a release or a label like the payment method. Sparta adds the before-and-after records, so you can see where the agent's intent and its action split.

Do we need our own labels first? No. Polaris starts pretrained. If your domain needs more, Polaris Fine-tuning trains a custom classifier on the labels you correct, and you can write your own eval as a prompt.

Try it on your own agent

Sparta's Worlds are in closed beta at usesparta.co. For Kubit, join the waitlist for the next cohort, or book 20 minutes with me and I'll walk through it on your own traces.

Previous post: How to find what changed right before an agent pattern started.

FAQ

Why didn't the trace show an error?

The tool call succeeded. The agent passed valid IDs in the wrong fields, so the API returned OK and every span stayed green. The mistake only shows up in the records the call changed.

How is testing in Sparta Worlds different from an eval?

Worlds runs the agent against working copies of tools like Stripe and Zendesk and compares the records before and after the run. The grade comes from the records the agent changed.

How does Kubit find the failing runs?

Flows groups every labelled run by execution path. Triggers then compares the failing runs with the rest and shows what they share, such as a release.

Do I have to sample my traces?

No. Polaris labels every span. MLflow's own docs suggest sampling 5% to 20% of traffic for expensive judges, which is the trade-off a small model avoids.

Do I have to move my traces?

No. Kubit queries agent traces in place, inside your own Databricks, Snowflake, BigQuery or ClickHouse account.