Checks

A check is a named condition that Ancha evaluates when an attempt ends. An attempt passes when every required check passes. Without checks, an attempt is ungraded: it ran, but nothing says whether it did the right thing.

Four kinds

Kind	Reads	Use it for
`state`	The final state of the tools	Outcomes: a reply exists in the right thread, a file has the right content, an issue carries the right label, an incident is resolved.
`trace`	The agent’s API calls, in order	Behavior: the agent filed an issue, or never wrote to the CRM.
Process checks	What the world’s engine recorded between the agent and the people in it	Conversations: the agent asked the customer the right question, told the engineer the details, never leaked a confidential fact, and did things in the right order. `ancha run` and `ancha check` only.
`judge`	The agent’s API calls and final Slack messages, graded by a model	Qualities that exact conditions cannot express, such as whether a reply was correct and polite. `ancha run` and `ancha check` only.

Prefer state checks for outcomes. They pass however the agent got there, while trace checks depend on which endpoint it chose.

Examples

In a scene file, checks are YAML:

scene.yaml

checks:
  - id: answered
    state:
      slack.message:
        channel: $slack_support
        from: $slack_bot
        in_thread_of: $slack_question
        text_contains: "Reports"
  - id: filed
    trace: { called: "POST /repos/acme/orders/issues" }
  - id: read_only_crm
    trace: { not_called: ["POST /crm/*", "PATCH /crm/*", "DELETE /crm/*"] }
  - id: asked_first
    before: [{ revealed: [laura.workspace] }, { called: "POST /repos/*/issues" }]
  - id: helpful
    judge: "The reply tells the customer where to find the CSV export and how long it takes."
pass: [answered, filed, read_only_crm, asked_first]

In the hosted console and in playground task files, the same checks are a JSON list on each task:

[
  {"id": "filed", "trace": {"called": "POST /repos/acme/orders/issues"}},
  {"id": "answered", "state": {"slack.message": {"channel": "$slack_support", "from": "$slack_bot", "in_thread_of": "$slack_question", "text_contains": "issues/"}}}
]

Required checks

pass lists the IDs of the required checks. It defaults to every check. A check outside pass is still evaluated and reported, but it does not decide the outcome. Use that for advisory checks, such as a judge rubric you are still tuning.

Trace patterns

A trace pattern is "<METHOD> <PATH>", for example "POST /api/chat.postMessage".

The path is relative to the tool’s base URL and has no query string: /repos/acme/starter, not the full URL.
The method is compared without regard to case. * as the method accepts any method. The path must match exactly, including case.
* in the path matches any run of characters, including /. "POST /crm/*" matches every POST under /crm/. The pattern "*" matches any call.
Only the agent’s own calls count. Actions taken by the world, such as a person posting in Slack, never match.

Field	Passes when
`called`	Every listed pattern matches at least one call.
`not_called`	No call matches any listed pattern.

With both fields set, both must hold.

Use * as the method when the verb does not matter. Slack’s official SDKs call every Web API method with POST, while plain fetch code often uses GET, and Slack accepts both. The starter scene’s checks are written this way, for example * /api/*.*. Check the agent’s calls with ancha explain or in trace.jsonl when a trace check surprises you.

State checks

Check	Passes when
`slack.message`	At least `count_gte` messages (default 1) in a channel match all the conditions given: author, a thread, a case-insensitive substring.
`github.file_at`	A file at a branch, commit or `production` contains, or equals, the given text.
`github.pull`	A pull request is merged, or not.
`github.production`	Production is exactly a given commit.
`github.issue`	Issues in a repository match a title, body, labels, state and count.
`datadog.monitor`	A monitor’s state is `OK` or `Alert`.
`datadog.downtime`	An active downtime covers a monitor with enough time left and a bounded total length.
`pagerduty.incident`	An incident has a status, an exact set of assignees, or was acknowledged or resolved within a time.
`fault_resolved`	A living-world fault is resolved.

State checks count every record in the final state, including starter data. Set from on a slack.message check so a message from the starter history cannot satisfy it. See Checks reference for every field.

Process checks

A scene can hold people: a customer who holds back details until asked, an engineer who knows the workaround, a manager who decides the priority. A model writes what they say, but the world’s engine decides who receives a message, which facts a person may share, what they decide and when they answer, and it records every one of those events. Process checks read those records.

Check	Passes when
`revealed`	The agent asked, and the owner of each fact shared it.
`knows`, `told`	A person knows each fact, from the agent or from someone else.
`decided`	A person made a decision, with a given option.
`message`	At least one message matching a selector reached a person or a place.
`never`	No message matched the selector, for example a confidential fact never reached the customer.
`before`	Milestones happened in order: a call, a message, or the first time a predicate held.
`within`	A milestone happened within a time limit of the start or of another milestone.

Because the checks read the engine’s records and never the model’s text, a person who writes something unexpected cannot pass or fail an attempt by wording alone. A milestone that never happened fails before and within, so an attempt that did nothing cannot pass them. See Process checks.

Judge checks

A judge check is a rubric in plain language. After the attempt, Ancha sends a model a bounded copy of the evidence: the agent’s API calls to Slack, GitHub, Datadog and PagerDuty with their requests and responses, and the final Slack messages, channels and users. The model returns passed or failed with a reason and the trace entries it relied on. Credentials, the scene’s seed and the other checks are never sent.

Judge checks need OPENROUTER_API_KEY in the environment of the ancha run or ancha check command. Ancha calls the model through OpenRouter. The agent does not get the key unless you list it in agent.pass_env.
Set judge.model in ancha.yaml to the OpenRouter model you want the judge to use.
Without a key, judge checks are skipped. A skipped check never passes, so a required judge check fails the attempt. Keep advisory rubrics out of pass.
If the model call fails, the trial is marked invalid instead of failed.
Judge checks run in ancha run and ancha check only. The hosted console and playground tasks accept state and trace checks.

Refs

Checks name records with refs such as $slack_support, not with generated IDs. Refs come from the world’s starter data or from your scene’s seed, and people from its people. Saving a task in the hosted console checks every ref against its world. See Worlds.

Results

Every check result has the same fields:

Field	Meaning
`id`	The check’s ID.
`required`	Whether it is in `pass`.
`passed`	The result.
`expected`	What the check wanted, in words.
`actual`	What the world held or what the agent did.
`explanation`	One sentence combining the two.
`evidence`	Sequence numbers of the trace entries behind the result.

ancha explain prints these for failed checks, ancha check --json prints them all for a live world, and each trial’s checks.json holds them all.

Get started

Guides

Concepts

Tools

CLI

Reference

Cloud

Help

Four kinds

Examples

Required checks

Trace patterns

State checks

Process checks

Judge checks

Refs

Results

​Four kinds

​Examples

​Required checks

​Trace patterns

​State checks

​Process checks

​Judge checks

​Refs

​Results

Four kinds

Examples

Required checks

Trace patterns

State checks

Process checks

Judge checks

Refs

Results