Blog

How to find what changed right before an agent pattern started

Alex Li

Nov 5, 2026

TL;DR

When an agent's quality score drops, or a new shape shows up in your execution paths, finding the cause usually means lining up prompts, models, config and commits by hand.
One trace can show you what happened in that run. It can't show you what the failing runs have in common, and that's where the cause usually is.
We read four signals off the whole pattern: what changed, which property is overrepresented, which label cluster it sits in, and what happened earlier in the same runs. Each one comes with the runs behind it, so you can check it yourself.

To find what changed right before an agent pattern started, compare the runs inside the pattern with the rest of your traffic. Look for a change in the prompt, model, tool or config that lines up with when the pattern began; a property of the runs that shows up far more often inside the pattern than outside it; a cluster in the span labels; and a step earlier in the same runs that predicted the outcome. Those four checks narrow a quality drop to a cause you can test, instead of a theory built from the one trace you happened to open.

Key facts

In OpenTelemetry, each span in a trace can carry attributes, key-value pairs such as a user segment or a tool name, which is what makes a property comparison across runs possible. (OpenTelemetry docs, 2026)
MLflow can link every trace to the Git commit, and the diff, of the code that produced it. (MLflow docs, 2026)
The numbers in the worked example below come from the demo Shopping Agent on kubit.ai/agent-observability. They are demo data, not anyone's traffic.

The score drops

You see it in a dashboard, or a teammate pings you. The share of runs marked unhelpful went up last week. Or Flows shows a loop at checkout that wasn't there two weeks ago.

The next step is usually manual. You open the prompt history, check which model the agent was calling, read the config changes, and scroll the commits from the days before. If nothing obvious turns up, you open some failing traces and read them one by one.

Why it takes so long

Three things make this slow.

The cause often isn't in the code. A prompt edited in a config file, a slower response from a tool the agent calls, or a new kind of user request can all move quality with no commit at all. Scrolling the commit log finds nothing, and the search starts again.
One trace tells you about one run. You can read a failing trace end to end and still not know whether the thing you spotted is in every failing run or only this one.
The comparison is the hard part. To say a property matters, you need to know how often it appears inside the pattern and how often it appears in your traffic overall. Doing that by hand means a query per hypothesis.

So the question is: how do you get from "something changed" to what changed, without a new query for every guess?

Read the signal off the pattern

Once runs are grouped by execution path (which tools, in what order), each pattern is a set of runs you can compare against everything else. That's what Triggers does. When quality shifts or a new usage pattern shows up, it reads four signals off the pattern.

Four signals Triggers reads off a pattern: Delta, Property, Semantic and Upstream, each with an example from the demo Shopping Agent. Related commits is next on the roadmap.

Signal	What it checks	Demo Shopping Agent example
Delta	What changed in the prompt, model, tool or config right before the pattern began	Shown when a change lines up with the start of the pattern
Property	A property of the runs that shows up far more often inside the pattern than in your overall traffic	`payment_method: paypal` at 2.0x its baseline, in 74 of 124 runs
Semantic	A cluster in the span labels inside the pattern	"Mentions gift bundle or set" in 47% of runs (47 of 101)
Upstream	A step earlier in the same runs that predicted the outcome	`check_inventory` ran slower in runs that later looped: 410ms against a 180ms baseline, 2.3x

Each signal comes with the count behind it, so you can see how many runs it covers and how far above baseline it sits.

A worked example

The Flows view of the demo Shopping Agent: a Sankey of 3,207 sessions fanning out from classify_intent through checkout into retry loops and error steps like payment_failed and out_of_stock.

In the demo, Flows shows a retry loop at checkout. Triggers puts three findings next to it. PayPal is twice as common inside the loop as outside it. Almost half of the looping runs mention a gift bundle or set. And check_inventory, a step that runs before checkout, took more than twice as long in the runs that went on to loop.

Read together, the three findings give you something specific to test: gift bundles may need an inventory check across several items, the check is slow, and the PayPal path retries when it times out. That's a hypothesis you can confirm in an afternoon.

You can also check each finding by hand, which is the point of showing the counts:

Property: filter the looping runs by payment method and compare the PayPal share with your overall traffic.
Semantic: read ten looping runs that mention a gift bundle and ten that don't.
Upstream: compare check_inventory latency in looping runs against runs that completed.

Where commits come in

A commit is one more kind of change, and often the one engineers check first. Related commits is next on our roadmap: for a failing execution path, it will show the commit, the diff, and the jump in error rate after it shipped.

Plenty of tools already know which commit produced a trace. MLflow, for one, links every trace to its Git commit and diff. What we're adding is the link from a pattern to the commit that came right before it, so you start from the group of failing runs and see the change, instead of starting from the commit log and guessing which runs it touched.

When one trace is enough

If a run throws an error with a clear stack trace, open it and fix it. You don't need a pattern for that.

Read the signal off the pattern when the failures don't throw, when the same problem repeats across many runs, or when the score moved and nothing in the commit log explains it.

See it on your own traces

Our first early-access cohort is closed, and we're keeping a waitlist for the next one. Join the waitlist, or book 20 minutes with me and I'll walk through it on your own traces.

Previous post: Why we label every span instead of a sample

FAQ

Why did my AI agent's quality drop when nothing in the code changed?

Agent quality can move without a commit. A prompt edited in config, a slower or different response from a tool the agent calls, or a new kind of user request can each change outcomes. That's why it helps to compare the failing runs with the rest of your traffic rather than only reading the commit log.