> ## Documentation Index > Fetch the complete documentation index at: https://docs.ancha.io/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > These docs describe the Ancha CLI, npm package @ancha-labs/cli version 0.1.4, and the hosted workspace at app.ancha.io. Coding agents should start at https://docs.ancha.io/coding-agents. Run the CLI as `ancha` after `npm install -g @ancha-labs/cli`, or as `npx -y @ancha-labs/cli@0.1.4` without installing. Pass --json for machine-readable output, or set ANCHA_JSON=1. Every command ends with a `next:` line naming the next command. Exit codes: 0 pass, 1 a check failed or a comparison regressed, 2 usage, config or authentication error, 3 infrastructure error, 4 unimplemented endpoint with --strict. # ancha run > Run the agent against every scene several times, each trial on a fresh copy of the tools, and report how often it passes. Compare with a baseline to prove a fix, and send the results to your workspace. `ancha run` runs your agent against each scene in `ancha.yaml`, `trials` times, on a fresh world each time, evaluates every scene's checks and exits with a code that says whether the agent passed. ```bash theme={null} ancha run [options] [scenes...] ``` | Argument | Description | | - | - | | `scenes` | Scene YAML files (default: the globs in scenes: of ancha.yaml) | | Option | Description | | - | - | | `--agent ` | Shell command that runs the agent (default: agent.command in ancha.yaml) | | `--agent-url ` | Endpoint URL (overrides agent.url; requires endpoint setup/start configuration) | | `-n, --trials ` | Trials per scene (default: trials in ancha.yaml, else 5) | | `--parallel ` | Trials running at once (default: parallel in ancha.yaml, else command 2 / endpoint 1) | | `--min-pass-rate ` | Fraction of trials that must pass, 0 to 1 (default: min\_pass\_rate in ancha.yaml, else 1) | | `--seed ` | Seed for ids and randomness | | `--out ` | Where run directories go (default: .ancha/runs) | | `--strict` | Exit 4 if the agent hit an endpoint Ancha does not implement | | `--detach` | Continue in a private worker after this command exits; observe with status/wait | | `--compare ` | Compare against a completed run id or saved baseline; any per-scene decrease fails | | `--pass-env ` | Give the agent this variable from this shell (repeatable); it otherwise gets only a base set (PATH, HOME, locale…) plus Ancha's and the tools' variables | | `--inherit-env` | Give the agent this shell's whole environment, including any real credentials (the behavior before 0.1.2) | | `--idempotency-key ` | Retry safely: the same key and the same run definition return the original run instead of starting another | | `--on ` | Where the agent runs: local (a process on this computer), docker (a sealed container; needs agent.execution with an image in ancha.yaml) or cloud (your workspace: uploads this project, connects it there and runs each test there). Default: what ancha.yaml says | | `--upload` | Send the finished run to your workspace, where the console shows it (also ANCHA\_UPLOAD=1); a failed upload is a warning | | `--upload-required` | Like --upload, but a failed upload fails the command | ## Example ```bash theme={null} ancha run ``` ```text Output theme={null} running locally (worlds in a server this command starts) run_started runId=run_20261031-234205_e9f7049f942a39d5 scenes=["starter"] trials=1 parallel=1 workspace=local:/path/to/project target=local trial_started scene=starter trial=1 envId=0ykgpn5i trial_finished scene=starter trial=1 passed=true valid=true warnings=[] exit=exited exitCode=0 realMs=79 failedChecks=[] ✓ starter 1/1 (100%) agent not sealed: network open; environment limited to 9 variables from this shell (pass more with --pass-env NAME) pass (minimum pass rate 100%) run run_20261031-234205_e9f7049f942a39d5 next: ancha baseline save before ``` The first line says where the worlds run. After `ancha login` it reads `running locally (worlds in a server this command starts; logged in to https://app.ancha.io/t/: --upload sends the results there)`: the login keeps the worlds on this computer and names the workspace that `--upload` sends results to. The last line names the next command. Pass scene files to run only those: `ancha run scenes/support/scene.yaml`. `--agent ""` overrides `agent.command` for one run. Without `ancha.yaml`, `run` exits `2` and tells you to run `ancha init`. ## How a run works 1. `ancha run` chooses a server for the worlds: the one `--server-url`, `server.url` or `ANCHA_SERVER_URL` names, else a private local server it starts for this run. A saved login does not move the worlds; it names the workspace for `--upload`. The first line names the server, for example `running on hosted https://app.ancha.io/t/ (from ANCHA_SERVER_URL; --local to run here)`. `--local` keeps the worlds on this computer whatever is configured. See [CLI overview](https://docs.ancha.io/cli/overview.md#which-server-a-command-uses). 2. For each scene and trial, it creates a fresh world from the scene's seed and starts your agent with the task and the tool variables. See [Environment variables](https://docs.ancha.io/reference/environment-variables.md). 3. When the agent exits or the scene's `timeout` passes, it captures the final state and every API call, then evaluates the checks. 4. A trial passes when all required checks pass. A scene passes when its pass rate reaches `min_pass_rate`. The run passes when every scene passes and nothing regressed. Results go to `.ancha/runs//`, and `.ancha/runs/latest` points at the newest completed run. See [Attempts and runs](https://docs.ancha.io/concepts/attempts-and-runs.md#what-ancha-run-records). ## What the agent gets The agent runs as a process on this computer, in the project directory. It is not sealed: it can reach this computer's network. Its environment is limited to a base set from your shell (`PATH`, `HOME`, `USER`, `LOGNAME`, `SHELL`, `LANG`, `LC_*`, `TERM`, `TMPDIR`, `TZ`, `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`, `VIRTUAL_ENV`, `CONDA_PREFIX` and `PYTHONPATH`), plus Ancha's variables and the tools' variables. Real credentials in your shell, such as `GITHUB_TOKEN` or `DD_API_KEY`, never reach it unless you name them: ```yaml ancha.yaml theme={null} agent: command: node agent.mjs pass_env: [OPENAI_API_KEY] # names only; values come from your shell at run time ``` `--pass-env NAME` adds a name for one run and can be repeated. `--inherit-env` passes the whole shell environment, as 0.1.1 did, and prints a warning. The run summary says what the agent got, and `ancha explain` says how many variables were withheld. With `agent.execution.mode: container`, the agent gets only the names in `agent.execution.env`. See [ancha.yaml](https://docs.ancha.io/reference/ancha-yaml.md#agent-execution). ## Exit codes | Code | When | | - | - | | `0` | Every scene met `min_pass_rate` and nothing regressed. | | `1` | A scene's pass rate fell below `min_pass_rate`, or `--compare` found a scene or a check that got worse. | | `2` | Usage, config or scene error, no agent command, or a hosted server that rejected the API key (`AUTH_INVALID`). | | `3` | Infrastructure failed, or a trial was invalid because Ancha itself failed. | | `4` | With `--strict`, the agent called an endpoint a replica does not implement. | When several apply, `3` wins over `4`, and `4` wins over `1`. ## JSON output `ancha run --json` prints NDJSON: `run_started`, then `trial_started` and `trial_finished` for each trial, then `run_finished` with the summary and the `next` commands. `run_started` and the summary carry `workspace` (the world server, or `local:`) and `target` (`local` for a process on this computer, `docker` for a container). See [For coding agents](https://docs.ancha.io/coding-agents.md#json-output) for the fields, and `ancha schema run_finished` for the schema. ## Compare with a baseline ```bash theme={null} ancha baseline save before # change the agent ancha run --compare before ``` ```text Output theme={null} running locally (worlds in a server this command starts) run_started runId=run_20261031-234206_09b422aebb589f2b scenes=["starter"] trials=1 parallel=1 workspace=local:/path/to/project target=local trial_started scene=starter trial=1 envId=qbw16ffq trial_finished scene=starter trial=1 passed=true valid=true warnings=[] exit=exited exitCode=0 realMs=77 failedChecks=[] ✓ starter 1/1 (100%) compared with run_20261031-234205_7570556079db03cd: no regression starter: 0% → 100% (+100 pp); 1 → 1 trials improved: pagerduty_access 0/1 → 1/1 agent not sealed: network open; environment limited to 9 variables from this shell (pass more with --pass-env NAME) pass (minimum pass rate 100%) run run_20261031-234206_09b422aebb589f2b ``` `--compare` takes a baseline name or a run ID. Every scene and every required check is compared. A scene whose pass rate drops fails the run with exit `1`, even when the overall pass rate is fine. So does a check whose pass rate drops, even when the scene's rate holds, for example when one check went from 3/3 to 0/3 while another went from 0/3 to 3/3. Dropped checks are listed first: ```text Output theme={null} running locally (worlds in a server this command starts) run_started runId=run_20261031-234925_9ba852ebe26b8eca scenes=["starter"] trials=1 parallel=1 workspace=local:/path/to/project target=local trial_started scene=starter trial=1 envId=k7grx7pe trial_finished scene=starter trial=1 passed=false valid=true warnings=[] exit=exited exitCode=0 realMs=76 failedChecks=["github_access"] ✗ starter 0/1 (0%) ✗ github_access failed in 1/1 trials trial 1: The agent never made a call matching "* /*repos*". Its calls were: GET /api/conversations.list, GET /api/v1/dashboard, GET /services. compared with run_20261031-234211_30f86efd56c0f25a: regressed ✗ dropped: starter github_access 1/1 → 0/1 starter: 100% → 0% (-100 pp); 1 → 1 trials agent not sealed: network open; environment limited to 9 variables from this shell (pass more with --pass-env NAME) fail (minimum pass rate 100%) run run_20261031-234925_9ba852ebe26b8eca next: ancha explain run_20261031-234925_9ba852ebe26b8eca Patterns are " " matched against the service-relative path, and * is a wildcard. ``` Comparison needs the same scenes with the same content and checks; trial counts may differ. The summary's `comparison` field holds `baselineRunId`, `regressed`, `droppedChecks` and a per-scene `before`, `after` and `delta`. ## Coverage gaps and `--strict` When the agent calls an endpoint a replica does not implement, the output lists it: ```text Output theme={null} running locally (worlds in a server this command starts) run_started runId=run_20261031-234925_79b4774ed98fa32d scenes=["starter"] trials=1 parallel=1 workspace=local:/path/to/project target=local trial_started scene=starter trial=1 envId=2uvxqv90 trial_finished scene=starter trial=1 passed=true valid=true warnings=[] exit=exited exitCode=0 realMs=73 failedChecks=[] ✓ starter 1/1 (100%) coverage gap: POST /api/chat.scheduleMessage (slack) x1 agent not sealed: network open; environment limited to 9 variables from this shell (pass more with --pass-env NAME) pass (minimum pass rate 100%) run run_20261031-234925_79b4774ed98fa32d next: ancha baseline save before ``` The call gets an error in the tool's own shape, `unknown_method` for Slack, so the agent sees a failure. Without `--strict`, a gap does not change the exit code. With `--strict`, any gap exits `4` and the output adds `--strict: the agent hit an endpoint Ancha does not implement (see coverage gaps)`. `ancha explain` lists the gaps either way. Check `ancha twins coverage ` for what each tool implements. ## Where the agent runs `--on` picks where the agent itself runs; the default is what `ancha.yaml` says. | `--on` | The agent runs | | - | - | | `local` | As a process on this computer, with the environment described above. | | `docker` | In the sealed container `agent.execution` describes, which needs an `image`. Never a fallback to a process. See [ancha.yaml](https://docs.ancha.io/reference/ancha-yaml.md#agent-execution). | | `cloud` | In your hosted workspace: `ancha run --on cloud` uploads this project, registers its agent under the project folder's name (or updates it), connects it, and runs each test there as its own run, streaming the progress. The exit code is the worst of the runs. Running your own agent in the workspace is in private beta; until your workspace has access, the command is refused with `hosted runs of your own agent are in private beta for this workspace`. See [Hosted workspace](https://docs.ancha.io/cloud/overview.md). | `--on cloud` and `--upload` do not combine: a run in the workspace already lives there. ## Results in your workspace `--upload`, or `ANCHA_UPLOAD=1` in the environment, sends the finished run to your workspace, where the console shows it at `https://app.ancha.io/runs/`. It needs a workspace: `ancha login`, or `ANCHA_SERVER_URL` and `ANCHA_API_KEY`. The run's own files stay in `.ancha/runs/`. ```text Output theme={null} pass (minimum pass rate 100%) run run_20261031-234205_e9f7049f942a39d5 uploaded: https://app.ancha.io/runs/run_20261031-234205_e9f7049f942a39d5 ``` A failed upload is a warning: the run keeps its own exit code, and `ancha runs upload ` tries again. `--upload-required` makes a failed upload fail the command instead. A run started with `--detach` uploads when it finishes, and `ancha wait` and `ancha status` say where the results went. With `--json`, `run_finished` carries `upload`: `url`, `bytes` and `files`, `replaced: true` when the run was uploaded before, or `error`. See [Results in your workspace](https://docs.ancha.io/cloud/results.md) for what the upload holds and what is redacted. ## Retrying safely A caller whose command can time out, such as a coding agent or a CI step, passes `--idempotency-key ` (1 to 200 visible ASCII characters). Running the same command again with the same key returns the original run instead of starting another: its summary and exit code, with `replayed: true` in the JSON. While the original is still running, the retry fails with `SESSION_CONFLICT` and a hint to `ancha wait `. The same key with a different scene set, agent or options is a usage error: ```text Output theme={null} error[USAGE]: this --idempotency-key was used for a different run hint: use a new key for a new run; reuse a key only to retry the same command ``` ## Detached runs `--detach` hands the run to a background worker and returns at once: ```bash theme={null} ancha run -n 3 --detach --json ``` ```json Output theme={null} {"event":"run_detached","runId":"run_20261031-234928_4ef35808ffd351f9"} ``` Follow it with [`ancha logs --follow`](https://docs.ancha.io/cli/logs), [`ancha wait`](https://docs.ancha.io/cli/wait) or [`ancha status`](https://docs.ancha.io/cli/status). `logs --follow` and `wait` exit with the run's own exit code. Pass the same `--out` to all of them if you changed it. A detached run survives the launching shell, but not a reboot or a killed worker. ## Timeouts and failures An agent that runs past the scene's `timeout` is stopped and the trial records `exit=timeout`. An agent that exits non-zero records its exit code. In both cases the checks still run against what the agent did, and `ancha explain` shows the end of its log. Ctrl-C stops the run the same way and keeps the evidence collected so far. ## Model-graded checks Judge checks need `OPENROUTER_API_KEY` in the environment of `ancha run`. Without it they are skipped, and a skipped required check fails its trial. The key is read by `ancha run` itself; the agent does not get it unless you list it in `agent.pass_env`. See [Checks](https://docs.ancha.io/concepts/checks.md#judge-checks). ## Your own harness To start the agent yourself and still grade it with the scene's checks, start a world with `ancha world up`, run the agent against it, and grade it with `ancha check`. See [World lifecycle](https://docs.ancha.io/cli/world-lifecycle.md). ## Related topics - [ancha runs](https://docs.ancha.io/cli/runs.md) - [Attempts and runs](https://docs.ancha.io/concepts/attempts-and-runs.md) - [ancha logs](https://docs.ancha.io/cli/logs.md) - [ancha wait](https://docs.ancha.io/cli/wait.md) - [ancha report](https://docs.ancha.io/cli/report.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.