Run trace

Run trace, defined

A run trace is a durable, structured record of a single execution of an agent. It begins with an envelope: the contact, the channel, the consent state, the purpose, the budget and the initiating request. It then records the plan the agent produced, each step in order, the model and version used at that step, every tool call with typed arguments and the returned result, the latency of each step, the tokens consumed and the rupee cost, each policy decision with a reason code, any human approval with the approver identity, and the final outcome. The run identifier ties the whole record together and appears in every downstream system the run touched.

What a run trace is not matters as much as what it is. It is not the application log, which is a stream of diagnostic lines written for engineers and usually discarded after a fortnight. It is not the conversation transcript, which holds what was said but not why the system chose to say it. It is not a metrics dashboard, which aggregates counters until the individual case disappears. And it is not an eval, which is a test run against expected outcomes before deployment. A trace is the evidence of one real event in production.

That distinction has a practical test. An application log helps you find a stack trace. A trace helps you answer a question from a customer, an auditor or a regulator: which model saw this record, what did it send, which policy allowed it, who approved it, and how much did it cost. If the answer requires reconstructing events from three systems and a memory of the deploy, the trace does not exist in any useful sense.

The fields a trace has to carry

A useful schema is boring and explicit. Identity fields come first: run identifier, parent run identifier for sub-agents, workspace, environment, agent name and version. Context fields come next: contact identifier, channel, purpose tag, consent artefact reference, locale and timezone. Execution fields follow: the plan as structured data, each step with a sequence number, the model identifier and version, the prompt template version, the tool name and arguments, the tool result or error, the retry count and the idempotency key. Evidence fields close the record: policy decisions with reason codes, approval events with approver identity, token counts, unit costs, total cost in rupees, latency per step and the terminal outcome.

Two fields are usually missing in first attempts. The first is the template version. A prompt edited in production changes behaviour, and without a version reference the trace cannot explain why the same input produced a different answer last month. The second is the idempotency key. It is the only way to prove that a retry did not create a second order, a second refund or a second WhatsApp message. A trace that records a successful tool call but not the key that made it safe is a narrative, not evidence.

Store structured values rather than prose where possible. A policy decision should be a code such as consent_missing or amount_above_gate, not the sentence the model wrote about it. Reason codes can be counted, alerted on and compared across releases. Free text cannot.

• Run and parent run identifiers, agent name and version.
• Envelope: contact, channel, purpose, consent reference, budget.
• Plan, step sequence, model and prompt template version.
• Tool name, typed arguments, result or error, retry count, idempotency key.
• Policy reason codes, approver identity, token and rupee cost, latency, outcome.

Trace, log, span and eval are different objects

Four artefacts get conflated in platform conversations and each has a different owner. A log line is a timestamped message from a process, optimised for volume and cheap storage. A span is a timed operation in a distributed trace, optimised for latency analysis across services. An audit log is an append-only security record of who accessed or changed what, optimised for compliance. A run trace is a domain record of an agent decision, optimised for reconstruction. A single run will produce many spans and log lines, and exactly one trace.

The eval is the odd one out because it happens before production. An eval is a repeatable test with expected outcomes, run against a fixed dataset. It tells you whether a change is likely to help. A trace tells you what actually happened to a specific customer. Teams that only have evals ship confidently and cannot explain incidents; teams that only have traces cannot tell whether the latest prompt change improved anything. You need both, joined by the prompt template version.

The comparison matters commercially. Observability vendors sell spans, log tools sell search, and compliance suites sell audit logs. An agent platform has to sell the domain object, because none of the three generic products knows what a consent artefact or a human gate is. Ask a vendor to show a trace for a failed run, not a screenshot of a latency chart.

Latency and cost, measured per step

A single end-to-end latency number hides the only information an operator can act on. A voice run that takes 1.4 seconds overall might spend 90 milliseconds on retrieval, 700 milliseconds waiting for a model and 500 milliseconds on a tool that should have been cached. Recording latency per step turns a complaint into a fix. Record p50 and p95 across runs as well, because the tail is what callers remember and the median is what dashboards celebrate.

Cost has the same shape. Token counts per step, multiplied by the model rate, give a rupee figure that can be attributed to a customer, a workflow or a release. Routing makes this interesting: a runtime that sends classification to a small model and negotiation to a large one will have a cost profile that changes with the mix of work, not with the price list. Cost per run is therefore a distribution, and the useful summary is the average alongside the expensive tail.

The number that belongs in a business review is cost per resolved task. Add model cost, channel cost, tool cost and human review time, then divide by the runs that reached a correct end state without rework. A run that costs forty paise and escalates is not cheaper than a run that costs three rupees and closes. The trace supplies the numerator; the system of record supplies the denominator.

Policy decisions and the human gate

A trace without policy decisions cannot explain a refusal, and a refusal without a reason code cannot be reviewed. Every check should produce one of a small set of outcomes: allow, deny, redact, cap or escalate. Each outcome carries the rule identifier that produced it and the inputs that mattered, such as the consent state, the amount, the contact tier or the channel window. The rule identifier should map to a versioned policy document, so a reviewer can see what the rule said on the day of the run rather than what it says today.

Human gates need more detail than a boolean. Record who was asked, when the request was created, when it was answered, what payload they saw, what they decided and what comment they left. If the gate times out, record the timeout and the default action that followed. A gate that is recorded only as approved by manager is indistinguishable from an automatic approval when someone reviews the run six months later.

The ordering is part of the evidence. A policy check that runs after the write is a comment, not a control. The trace should show the check before the tool call, with the tool call carrying the decision reference. That sequence is what makes the record defensible when a customer disputes a charge or a regulator asks who authorised an action.

Retention: how long, and what gets deleted

Retention is a schema decision, not a storage setting. Split the trace into classes with different lifetimes. The operational record, meaning the plan, step timings and cost, can often be kept for ninety days at full fidelity. The accountability record, meaning the envelope, the policy decisions, approvals and outcome, usually needs longer, driven by contract, tax or sector rules. The payload record, meaning message bodies and personal data, should have the shortest life that still allows dispute resolution. One retention period for everything is either wasteful or non-compliant, and usually both.

Under the DPDP Act, a data principal can ask what you hold and ask for erasure. A trace keyed by a contact identifier can honour that request, provided the identifier is present on every record that contains personal data. If traces are keyed only by a random run id and the contact is buried in a prompt string, erasure becomes a search problem across an unbounded store. Design the key first, then the retention sweep, then the deletion job, and make the sweep cover the trace store, the search index, the analytics copy and the backups.

Legal holds complicate the schedule in a healthy way. When a dispute is open, the relevant runs must be preserved beyond the normal window, and the hold itself must be auditable. The practical pattern is a hold flag on the run plus a job that excludes flagged records from the sweep, with a report of what is being held and why. Deleting evidence during a dispute is worse than keeping it too long.

Personal data inside the trace

A trace is full of personal data: names, phone numbers, email addresses, order details, sometimes health or financial context. The default posture should be to keep the minimum needed to reconstruct the decision and to redact the rest at write time, not at read time. Redaction after storage means the sensitive value existed in the store, the backup and probably the analytics pipeline. Redaction before storage means it never did.

Useful techniques are unglamorous. Field allowlists stop new personal data from arriving by accident when a tool result changes shape. Tokenisation replaces a phone number with a stable token that still supports correlation but not contact. Hashing works for identifiers you only need to match, never for values you need to display. Structured summaries work better than raw payloads for message content: store the template identifier, the variables that were rendered and a hash of the final body, not the body itself.

Access is the other half. Trace viewers should be role-limited, with a reason required to open a record that contains contact data, and every view should itself be logged. Support teams need enough to resolve a ticket; marketing does not need the tool arguments; auditors need the decisions and the approvals. One search box for everyone is how traces become a shadow customer database.

Instrumenting a run without breaking it

The trace writer must never be the reason a customer waits. Emit events to a durable buffer and let a separate consumer assemble the trace, so a slow trace store degrades observability rather than the product. Propagate the run identifier through every service, queue message and tool call, and generate it at the edge where the request enters. If the identifier is created halfway through, the first steps will be missing from the record and the reconstruction will start in the middle.

OpenTelemetry is a reasonable transport because it already solves context propagation and span export, and its generative AI conventions give a shared vocabulary for model, token and tool attributes. It is not a schema for accountability. Keep the domain fields in your own store, with the trace id linking to the spans, rather than trying to encode consent, policy and approval semantics into generic attributes that no auditor will understand.

Sampling needs care. Head sampling is fine for latency analysis and wrong for accountability: the one run that goes wrong is the one you cannot afford to drop. Use tail-based sampling for diagnostic spans, and keep the accountability record for every run that touches money, consent or a regulated action. Version the schema and treat a breaking change as a migration, because traces outlive the code that wrote them.

Worked example: a refund run, traced

A customer messages on WhatsApp about a damaged order and asks for a refund of eleven hundred rupees. The envelope records the contact, the channel, the service purpose, the consent state and a budget of one model call plus two tool calls. The plan reads the order, checks the return window, proposes a refund and asks for approval if the amount crosses the gate. The first tool call returns the order and the delivery date. The second returns the return policy, which allows a refund within seven days of delivery.

The policy check sees an amount above the five hundred rupee auto-approval threshold and escalates. The gate creates a task for the support lead, who approves it ninety seconds later with a comment about the photograph the customer sent. The refund tool writes to the payment system with an idempotency key derived from the run, and the trace records the payment reference. Total model cost is thirty-one paise, total latency is four seconds before the gate and two days including the human wait. The customer receives a WhatsApp confirmation from an approved utility template.

Six weeks later the customer disputes a second charge. The trace answers the question in under a minute: the first refund was approved by a named person, the amount matched, the payment reference exists and the message that confirmed it used the approved template. Without the trace, the team would be reading a support inbox, a payment dashboard and a WhatsApp log, and guessing at the order of events.

Measuring whether traces are good enough

Trace completeness is the first measure: the share of runs where every step has a tool result, a policy decision and a cost. It is usually lower than teams expect, because a tool that times out often returns no result object at all. The second measure is reconstructability, tested by asking someone who did not build the system to answer a fixed set of questions from a sample of twenty traces. If they need to open another tool, the trace has a gap.

Two operational measures follow. Time to answer a customer dispute, measured from the request to the evidence, falls sharply when traces are complete. Duplicate action rate should be zero by construction; any non-zero value points to a missing idempotency key rather than a model problem. Cost per resolved task, with human review time included, is the number that survives a finance review.

Review a small sample by hand every week. Dashboards show that runs are being traced; only reading traces shows whether they are worth reading. The habit is cheap and it catches schema drift, redaction failures and reason codes that have quietly stopped being emitted long before an audit does.