Skip to main content
A test fixes everything except your agent: the world, the tasks, the checks and how many attempts each task gets. When you change the agent and run the test again, any difference in the results comes from the agent.

​
What a test holds

​
Tests in ancha run

ancha run reads its test from files in your repository, so the test is versioned with your code.
  • ancha.yaml lists scene files with globs, sets how many trials each scene gets and the minimum pass rate. See ancha.yaml.
  • Each scene file holds one task with its world and checks. See Test file.
ancha.yaml
A run passes when every scene’s pass rate reaches min_pass_rate and no comparison regressed.

​
Tests in the hosted console

In the hosted console, open an agent and choose Add tasks. Pick a world, write the tasks, set how many times each repeats, add checks, then Run tasks. The test is saved, so you can run it again after you change the agent. See Hosted workspace.

​
Tests with ancha playground

ancha playground test create saves the same kind of test locally:
tasks.json is an array of tasks. Each task has a prompt, attempts, and optionally a title, timeoutMs and checks:
tasks.json
A test holds up to 20 tasks and 100 attempts in total. A task holds up to 20 checks. timeoutMs ranges from 1,000 to 3,600,000 and defaults to 300,000 (five minutes). Task checks use the scene check format but only state and trace checks; judge checks run only in ancha run. See ancha playground.

​
Pinned inputs

A saved test pins the world revision and its starter data. Starting it later uses the same data, with your current agent code. If you edit the world, the test keeps its revision until you choose the newer one. ancha test save <run-id> saves the setup of a completed ancha agent run as a reusable test. ancha test run <id> refuses to start when the saved scenario or data definitions have changed. See ancha test.

​
Baselines

A baseline freezes the summary of a completed ancha run so later runs can be compared with it:
A comparison fails when the pass rate of any scene or any required check drops. It requires the same set of scenes with the same content, including their checks. Trial counts may differ. See ancha baseline.