> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ancha.io/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> These docs describe the Ancha CLI, npm package @ancha-labs/cli version 0.1.4, and the hosted workspace at app.ancha.io. Coding agents should start at https://docs.ancha.io/coding-agents. Run the CLI as `ancha` after `npm install -g @ancha-labs/cli`, or as `npx -y @ancha-labs/cli@0.1.4` without installing. Pass --json for machine-readable output, or set ANCHA_JSON=1. Every command ends with a `next:` line naming the next command. Exit codes: 0 pass, 1 a check failed or a comparison regressed, 2 usage, config or authentication error, 3 infrastructure error, 4 unimplemented endpoint with --strict.

# Tests

> A test is a world plus tasks and how many times to try each one, saved so the same test runs again against new agent code.

A test fixes everything except your agent: the world, the tasks, the checks and how many attempts each task gets. When you change the agent and run the test again, any difference in the results comes from the agent.

## What a test holds

| Part | Meaning |
| - | - |
| World | The tools and starter data each attempt begins with. See [Worlds](https://docs.ancha.io/concepts/worlds.md). |
| Tasks | What the agent should do, in plain language. |
| Attempts | How many independent tries each task gets. |
| Checks | Conditions that decide whether an attempt passed. Optional; without them attempts are ungraded. See [Checks](https://docs.ancha.io/concepts/checks.md). |
| Time limit | How long an attempt may run before Ancha stops the agent. |

## Tests in `ancha run`

`ancha run` reads its test from files in your repository, so the test is versioned with your code.

* `ancha.yaml` lists scene files with globs, sets how many trials each scene gets and the minimum pass rate. See [ancha.yaml](https://docs.ancha.io/reference/ancha-yaml.md).
* Each scene file holds one task with its world and checks. See [Test file](https://docs.ancha.io/reference/test-file.md).

```yaml ancha.yaml theme={null}
agent:
  command: node agent.mjs
scenes: [scenes/**/scene.yaml]
trials: 5
parallel: 2
min_pass_rate: 0.8
```

A run passes when every scene's pass rate reaches `min_pass_rate` and no comparison regressed.

## Tests in the hosted console

In the hosted console, open an agent and choose **Add tasks**. Pick a world, write the tasks, set how many times each repeats, add checks, then **Run tasks**. The test is saved, so you can run it again after you change the agent. See [Hosted workspace](https://docs.ancha.io/cloud/overview.md).

## Tests with `ancha playground`

`ancha playground test create` saves the same kind of test locally:

```bash theme={null}
ancha playground test create --name "Support checks" \
  --agent <agent-id> --world <world-id> --tasks tasks.json --concurrency 2 --json
```

`tasks.json` is an array of tasks. Each task has a `prompt`, `attempts`, and optionally a `title`, `timeoutMs` and `checks`:

```json tasks.json theme={null}
[
  {
    "title": "Answer the newest support question",
    "prompt": "Answer the newest unanswered question in Slack #support in its thread.",
    "attempts": 3,
    "timeoutMs": 120000,
    "checks": [
      { "id": "replied", "trace": { "called": "POST /api/chat.postMessage" } }
    ]
  }
]
```

A test holds up to 20 tasks and 100 attempts in total. A task holds up to 20 checks. `timeoutMs` ranges from 1,000 to 3,600,000 and defaults to 300,000 (five minutes). Task checks use the scene check format but only `state` and `trace` checks; `judge` checks run only in `ancha run`. See [`ancha playground`](https://docs.ancha.io/cli/playground).

## Pinned inputs

A saved test pins the world revision and its starter data. Starting it later uses the same data, with your current agent code. If you edit the world, the test keeps its revision until you choose the newer one.

`ancha test save <run-id>` saves the setup of a completed `ancha agent run` as a reusable test. `ancha test run <id>` refuses to start when the saved scenario or data definitions have changed. See [`ancha test`](https://docs.ancha.io/cli/test).

## Baselines

A baseline freezes the summary of a completed `ancha run` so later runs can be compared with it:

```bash theme={null}
ancha baseline save before
ancha run --compare before
```

A comparison fails when the pass rate of any scene or any required check drops. It requires the same set of scenes with the same content, including their checks. Trial counts may differ. See [`ancha baseline`](https://docs.ancha.io/cli/baseline).


## Related topics

- [ancha test](https://docs.ancha.io/cli/test.md)
- [Test a Slack bot](https://docs.ancha.io/guides/test-a-slack-bot.md)
- [Test file (scene.yaml)](https://docs.ancha.io/reference/test-file.md)
- [ancha playground](https://docs.ancha.io/cli/playground.md)
- [Quickstart](https://docs.ancha.io/quickstart.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.