Blog
Should you build your own agent analytics?
Alex Li
|
TL;DR
A script over your agent trace export is quick to write and answers the first questions well: runs by tool, errors by day, latency by channel.
It gets harder with each question after that. Failures that return OK need a label on every span, grouping runs by execution path needs a second piece of work, and joining runs to what users did next is a project of its own.
Build your own agent analytics if your volume is low enough to read the traces that matter, your failures mostly throw errors, and one person can own the script. Use what Databricks or Snowflake already give you if your traces are there and you mostly debug one run at a time. Use a product when you need every span labeled, runs grouped by the route they took, and the results joined to the user and business tables sitting next to your traces.
Key facts
On Databricks, MLflow traces stored in Unity Catalog can be queried from the MLflow UI and Python SDK, and also with SQL and AI/BI dashboards Databricks docs, 2026.
Databricks stores those traces in OpenTelemetry format, so each span keeps its attributes and parent-child structure Databricks docs, 2026.
On Snowflake, Cortex Agents log threads, turns and execution spans to
SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS, which you can view in the Observability tab or query with SQL Snowflake docs, 2026.In OpenTelemetry, each span can carry attributes such as a tool name or a user segment, which is what a script groups by OpenTelemetry docs, 2026.
What the script gets you
The first version is usually a notebook or a scheduled query over the trace table. It counts runs by tool, errors by day and latency by channel, and puts them on a chart. If your traces already sit in a warehouse, that's a few hundred lines and an afternoon.
It's a good afternoon's work. You learn which tools fail most, whether latency moved after a release, and how much traffic each channel sends. If your volume is low enough that one person still reads the failing traces, that may be all you need for a while.
Where the script stops
The script counts what the spans already say. That covers errors, timeouts and slow calls. It doesn't cover the runs where every call returned OK and the agent still did the wrong thing, because nothing in the span says so.
Question | Script over the export | What it takes |
|---|---|---|
Which tools error most? | Yes, a group-by on status | Nothing more |
Did latency move after Tuesday's commit? | Yes, a group-by on time | Nothing more |
How many runs failed without an error? | No | A label on every span, from a model that reads it |
Which route did the failing runs take? | No | Grouping runs by execution path (which tools, in what order) |
What did users do after a run looped? | No | A join from runs to session and outcome tables |
Each of the last three is its own piece of work. Labeling every span means running a model over all of them, and our previous blog covers why the usual answer is to judge a sample instead. Grouping by execution path means turning each run's sequence of tool calls into a shape you can count. The join needs agent runs and user sessions keyed the same way, and someone from the analytics side who knows the session tables.
Then there's ownership. The script belongs to whoever wrote it. Every new question adds a query, the queries depend on how the trace schema looked when they were written, and when the agent's tools change, some of them quietly stop matching. When the owner moves on, the next person usually starts again.
Use what your warehouse already gives you
If your traces are in Databricks or Snowflake, check what's built in before writing anything.
On Databricks, MLflow Tracing can store traces in Unity Catalog tables in OpenTelemetry format. You get the MLflow UI for opening a trace and its span tree, and the same tables are queryable with SQL and AI/BI dashboards. Your agent needs MLflow Tracing in it for this.
On Snowflake, Cortex Agents log every conversation turn as a trace, and the Observability tab shows planning, tool calls, responses and user feedback. Agents built outside Cortex can be traced with TruLens, and evaluations score runs with LLM judges.
Both open one run end to end and store traces in tables you can query, so the script in the first section runs on them too. Labeling every span and grouping runs by route are separate work on top of either.
When building your own is the right call
Build it yourself if your volume is low enough that one person still reads the failing traces, your failures mostly throw, and you have an engineer who wants to own it. Build it if your questions are stable and you know them in advance.
It stops being the right call when the questions keep changing, when failures return OK, or when someone outside the agent team asks what users did after a run. That last question usually comes from product or growth, and it means a join.
What we built instead
We spent years building product analytics inside our customers' warehouses before we built for agents. Kubit is that approach applied to agent traces.
Kubit queries your traces directly in your warehouse, and Polaris labels every span. Nothing is retained outside your account. Native query reads traces where they already sit in Databricks, Snowflake, BigQuery or ClickHouse, with nothing to install on Databricks. Polaris, our own small language model, labels each span for intent, sentiment, failure and hallucination, so a run that returned OK and did the wrong thing still gets counted. Flows draws every run as one Sankey grouped by execution path, where outliers can be spotted visually from the population, even if Polaris missed them. Because the traces stay in your warehouse, you can map agent actions to the user activity, A/B tests and campaigns already there.
See it on your own traces
Our first early-access cohort is closed, and we're keeping a waitlist for the next one. Join the waitlist, or book 20 minutes with me and I'll walk through it on your own traces.
Earlier in the series: How to find what changed right before an agent pattern started.
FAQ
Should I build my own agent analytics?
Build it if your volume is low, your failures mostly throw errors, your questions are stable and one engineer can own the script. Once you need failures without errors counted, runs grouped by execution path, or runs joined to what users did next, each of those is a separate project.
Can I use Databricks or Snowflake for agent traces?
Yes. Databricks stores MLflow traces in Unity Catalog tables you can open in the MLflow UI or query with SQL. Snowflake logs Cortex Agent traces to SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS, with an Observability tab and SQL access. Both work well for opening one run end to end.
Why can't a script count failures that don't throw an error?
A script can only group by what the span records, such as status, latency and tool name. When a call succeeds and the agent still does the wrong thing, nothing in the span says so. Counting those runs needs a label on every span from a model that reads what happened.
Do I have to move my traces to use Kubit?
No. Kubit queries your traces directly in your Databricks, Snowflake, BigQuery or ClickHouse account, and Polaris labels every span. Nothing is retained outside your account.
Can I join agent traces to user behavior?
Yes, when both sit in the same warehouse. Kubit maps agent actions to the user activity, A/B tests and campaigns in your warehouse, so you can see what users did after a run.
