Govern & operate

Logs & traces

Every run on the platform — playground chats, swarm nodes, notebook calls — is recorded as a trace. Reading traces is the core debugging skill in agentic systems, and the one course environments almost never let you practice.

The traces table

/traces lists every run with the agent name, provider and model, latency, tokens in and out, dollar cost, status, and timestamp. Sort or scan for the rows that look wrong — the red statuses, the latency outliers, the runs that cost ten times their neighbours.

What a trace contains

Selecting a run opens the full record:

Metrics: Latency, tokens in, tokens out, and cost computed from model pricing — attributed to the user, agent and credential that caused it.
Resolved system prompt: What the model was ACTUALLY told, after retrieval, memory and routing guidance were folded in — not the template you configured. Usually the most surprising part of the record, and the first thing to read.
Tool calls: Each tool the model called, with the arguments it chose and the result it got back. If a tool you expected is never called, check the request payload to confirm it was offered at all.
Request / response payloads: The raw provider request and response. This is the ground truth: the exact message array, parameters, and tool definitions the model actually received, and exactly what it returned.
Error: For failed runs, the error message the runtime captured.

The playground inspector

While chatting in the Playground, the inspector panel shows the same information live, in three tabs: the latest request/response exchange, the stream of tool events as they happen (with a running call count), and the trace for the current conversation. For swarm runs, the observability view adds the per-node timeline.

Why it works this way

An agent's explanation of its own reasoning is generated text — it is a plausible story about what happened, not a record of it. The trace is the record. When the two disagree, the trace is right. Debug in that order and you will stop chasing phantom problems.

Three habits

Reproduce, then read. Re-run the failing input, open the trace, and read the request payload before forming a theory. Most "the model is broken" reports turn out to be "the model was sent something other than what I assumed".
Change one thing. Adjust a single line of prompt, one parameter, or the model — then run the same input and compare the two traces. Keeping the old trace open in a second tab is the closest thing prompt engineering has to a scientific method.
Watch cost as a signal. A run whose cost jumps an order of magnitude usually means a loop, a context blow-up, or a tool feeding the model far more text than intended — the trace shows which.

The Failure Modes Lab notebook is guided practice for exactly this skill: it produces a broken trace and asks you to find the cause.

Symptom to cause

The same handful of failures account for most of them, and each has a signature in the trace that identifies it in seconds. Read the row that matches what you saw:

What you saw	What the trace shows	Cause
A confident answer that is simply untrue	The resolved system prompt contains no retrieved context, or a retrieval block with zero passages	Retrieval matched nothing and the model answered from training. Fix the collection or the prompt's refusal rule — not the model.
It ignored a tool you know it has	The request payload's tool list does not contain it	Not enabled on this agent, or not allow-listed. If it IS in the list, the description is too vague to match the question.
A number that is wrong but plausible	No `sql_query` call — the answer came from a document	It read a figure out of prose instead of counting. Attach the table and enable the tool.
The answer stops mid-sentence	Tokens out sits exactly at the configured maximum	max_tokens truncation. Common cause of JSON that will not parse.
One run cost 20× its neighbours	Tokens in is enormous while the question is short	Retrieval or conversation history is dominating the prompt — and being paid for every turn.
It says it did something it did not do	No tool call for the action it described	The model narrated an intention. Only a tool call in the trace is evidence that anything happened.
It worked yesterday, fails today	A tool call returning an error the answer never mentioned	An upstream change. Models rarely announce a failed tool; they answer around it.

An empty result and a failed call look identical in the answer

Both produce a fluent reply with no sign anything went wrong, which is why the tool-call result — not the answer — is the thing to read. This is the single highest-yield habit on the page.

A debugging order that works

1
Read the resolved system prompt first
Half of all surprises are here — a retrieval block that came back empty, memory that recalled something stale, or routing guidance that pushed the model at the wrong tool.
2
Then the tool calls, in order
Look for a call that returned an error or an empty result. Models rarely announce that a tool failed; they answer anyway.
3
Then the token counts
A large input with a small question means retrieval or conversation history is dominating — and paying for it every turn.
4
Only then change the prompt
Most prompt edits made before reading the trace fix the wrong thing.

Swarm traces

A swarm run records per-node steps, so you can see which branch a router chose, which nodes were skipped, where an approval waited, and what each node wrote to flow state. Runs triggered through the API are traced identically and attributed to the key that started them.

Reading a run: three views, then one step

Open a run from Observability → Swarm Traces. The run itself has three views, and they answer different questions — reaching for the wrong one is why a trace can feel unreadable.

View	Answers
Canvas	Which path did the run actually take? The graph you built, with the nodes that ran marked on it — the fastest way to see a branch not taken or a node skipped.
Timeline	What happened, in order, and what did each step cost? One row per step. Click a row to open it.
Data flow	Which edges fired, and what moved along them. The count in the tab label is the number of edges that carried anything; zero means nothing flowed, which is itself the answer when a downstream node saw no input.

Clicking a step opens it with its status, latency, tokens in/out and cost at the top, then seven tabs holding exactly what that step saw and produced:

Tab	Contents
Input	The step's input, as JSON — what it was actually given, not what you intended.
Output	What it returned.
Thinking	Reasoning text, where the model emitted any.
Tools	Tool calls with their arguments and results.
Memory	The memory read into this step.
RAG	The retrieved chunks — the ones in the prompt, so an answer citing nothing shows an empty list here.
Error	The failure message, when the step failed.

Start at Input, not Output

A step that produced something wrong is usually a step that was given something wrong, and the Input tab settles that in one look. Working backwards from Output invites you to fix the prompt of a node whose real problem is upstream — the node before it wrote the wrong thing to flow state, and every node after inherits it.

Retention, and what a regulated tenant can turn off

Everything on this page rests on storing what people typed and what models replied. That is what makes debugging good and what makes the trace store sensitive. Two controls, and they do different jobs:

Control	Where	Effect
`PERSIST_PROMPT_BODIES`	Environment (default ON)	Set it to `false` and free text is never written: prompts, model responses, node inputs and outputs, chain-of-thought.
Trace retention (`trace_retention_days`)	Admin → IAM → Settings	Deletes traces older than the window. 0 keeps them indefinitely.

Turning bodies off keeps the skeleton

You do not lose observability — model, provider, tokens, cost, latency, status, the node graph and the shape of each tool call are all still recorded. What you lose is the text inside them. That is usually enough to spot a loop, a cost blow-up or a failing tool, and not enough to see what the user actually asked.

It is not retroactive

The setting drops bodies at write time. Turning it off today does nothing about what was captured yesterday — that is what the retention window is for. If you are switching it off for a compliance reason, set a retention window in the same change, or the existing rows sit there indefinitely.

PreviousBudgets & cost Next Analytics & audit