From a production trace to a failing test

Your agent did something wrong in production. A customer complained, or you saw it in your tracing tool. This guide turns that run into a test you run on your computer. Your agent runs for real, on the same input. Each of its tool calls gets the answer production got. The test checks for what went wrong and tells you whether the failure happens again. The test stays in your repository, so once you fix the agent the failure cannot quietly come back.

Before you start

The Ancha CLI: npm install -g @ancha-labs/cli.
An agent whose runs are traced: OpenTelemetry (the AI SDK, OpenLLMetry, any GenAI instrumentation), Langfuse or LangSmith.
For traces kept in your workspace: an account at app.ancha.io, then ancha login. To keep everything on your computer, use ancha import --offline with a trace file instead.

1. Wrap your agent’s tools

In a test, your agent’s tools must not reach production. Wrap them once. Outside a test the wrapper calls your function as before. In a test, each call is answered from production’s recorded calls.

import { wrapTools, entry } from '@ancha-labs/tools';

const tools = wrapTools({ lookupOrder, refundOrder }); // functions, or the AI SDK's tool() objects

await entry(async input => runAgent(input.task, tools)); // the command a test runs

import ancha

@ancha.tool
def lookup_order(order_id: str) -> dict: ...

tools = ancha.wrap_tools([search_tool, update_tool])  # LangChain tools, OpenAI Agents function tools

ancha.entry(lambda input: run_agent(input["task"]))  # the command a test runs

Install with npm install @ancha-labs/tools or pip install ancha. Tools your agent reaches over MCP need no wrapper: point the MCP client at {{tools.mcp_url}} in ancha.yaml. Tool wrappers has the details. If a tool the trace recorded is not wrapped, the test stops before your agent runs and names the tool, so a test never calls production by accident.

2. Bring in the trace

Pick the source your agent already uses.

Langfuse or LangSmith
OpenTelemetry
A trace file

Import one trace by its id, with your own keys from this shell. They never leave your computer.

export LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=... LANGFUSE_HOST=https://cloud.langfuse.com
ancha import --from langfuse 6ab868e05ac8c1e5910fdfc70d565af0 --expect "it refunded an order that belongs to another customer"

For LangSmith, set LANGSMITH_API_KEY and pass --from langsmith. Ancha reads the whole conversation of a thread and the feedback on every turn.

On the Traces page of your workspace, click Create ingest key, and copy the two lines it shows into your agent’s production environment:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.ancha.io/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer%20<your ingest key>

An ingest key can only send traces, so it is safe in production configuration. Each trace is ready about a minute after its last span. Find the run and pull its test:

ancha traces --failed
ancha pull fc7f4f10 --expect "it declared the incident resolved on a status measured before the rollback"

ancha import --offline trace.otlp.jsonl --expect "it refunded an order that belongs to another customer"

--offline makes the test on this computer and uploads nothing. The file is OTLP JSON, a Langfuse or LangSmith export, or Ancha Trace IR.

--expect says in one sentence what went wrong. The test gets a check for it, and Ancha judges whether production did it (it did). Without --expect, the checks come from the trace: a thumbs-down comment becomes a check that the agent does not do what the user complained about, and an error that ended the run becomes a check that the agent finishes. With neither, the test asks what the agent should have done. The test lands in scenes/from-traces/trace-<id>/:

File	What it holds
`scene.yaml`	What the agent is asked, its tools, and the checks.
`observations.jsonl`	Every tool call production made, with its answer.
`production.jsonl`	Production’s run: its input, its tool calls and its answer. Each check’s production result comes from it.
`questions.json`	What one trace cannot settle, each with the default the test uses.

3. Answer what one trace cannot settle

The import lists the questions. The two you meet most:

tools.mcp: which tools your agent reaches over MCP. A trace cannot tell. The test serves those at {{tools.mcp_url}}.
tools.simulate_writes: which writes may be simulated when the agent makes one production never made. By default such a write fails, so a test never invents a write’s success.

ancha import --offline trace.otlp.jsonl --force --answer tools.mcp=get_deploy_status,rollback --answer tools.simulate_writes=post_status_update

For a trace in your workspace, answer with ancha pull <id> --answer ..., which saves a new version. See ancha pull.

4. Reproduce it

Run the test against the version of the agent that failed in production:

ancha run scenes/from-traces/trace-3d4e261c/scene.yaml --trials 3

Output

computed production's result for what_went_wrong, what_the_user_complained_about from scenes/from-traces/trace-3d4e261c/production.jsonl
✗ trace-3d4e261c  0/3 (0%)
  no-deletes                      passed in production · failed 0/3 here
  what_went_wrong                 failed in production · failed 3/3 here
  what_the_user_complained_about  failed in production · failed 3/3 here
=> production's failure happens here too
   answers to the agent's tools: 9 replayed

The last line says how much of the run stood on production’s own answers. A call production never made to a tool that only reads is answered by a model shown the tool’s description, every answer production got, and the agent’s writes so far. A tool that changed something is never simulated unless you said so. ancha explain lists each answer that was not production’s. If production’s failure does not happen here, either the agent is already fixed or the test differs from production. questions.json says where it may differ.

5. Fix the agent and keep the test

Change your agent and run the test again until production’s failure no longer happens:

Output

✓ trace-3d4e261c  3/3 (100%)
  no-deletes                      passed in production · failed 0/3 here
  what_went_wrong                 failed in production · failed 0/3 here
  what_the_user_complained_about  failed in production · failed 0/3 here
=> production's failure did not happen here: the agent is fixed, or the world differs from production (see questions.json next to the scene)
   answers to the agent's tools: 3 replayed

Run it several times. An agent that passes once can still fail. Then commit scenes/from-traces/. Every ancha run, in CI too, keeps the failure fixed. See Run Ancha in CI. Add checks of your own in scene.yaml, such as refund_order {order_id: A-1002} never being called. Ancha computes production’s result for them too, and they stay when you pull a newer version. See Checks.

When the agent calls vendor APIs directly

An agent that calls Slack, GitHub, Datadog or another tool Ancha has a twin of, with the request and response bodies in its trace, gets that twin seeded from the calls. The test then answers new calls from the twin’s state, not from a recording. Add the bodies with the Ancha recorder. A tool of your own whose code called such a vendor runs against the twin for calls production never made.

What a trace can and cannot tell

It shows what your agent was asked and what each tool answered. It cannot show what a tool would have answered to a call production never made; that is replayed from a similar call, simulated, or failed, and each answer says which.
It cannot test a bug inside a tool’s own code. Ancha answers in the tool’s place; what is tested is what your agent decides.
Your trace data stays in your workspace, redacted before it is stored. See Trace privacy.

Get started

Guides

Concepts

Tools

CLI

Reference

Cloud

Help

From a production trace to a failing test

Before you start

1. Wrap your agent’s tools

2. Bring in the trace

3. Answer what one trace cannot settle

4. Reproduce it

5. Fix the agent and keep the test

When the agent calls vendor APIs directly

What a trace can and cannot tell

​Before you start

​1. Wrap your agent’s tools

​2. Bring in the trace

​3. Answer what one trace cannot settle

​4. Reproduce it

​5. Fix the agent and keep the test

​When the agent calls vendor APIs directly

​What a trace can and cannot tell

Before you start

1. Wrap your agent’s tools

2. Bring in the trace

3. Answer what one trace cannot settle

4. Reproduce it

5. Fix the agent and keep the test

When the agent calls vendor APIs directly

What a trace can and cannot tell