Checks reference

A check is an object with an id and exactly one check kind: state, trace, judge, or one of the process checks revealed, knows, told, decided, message, never, before and within. Scene files write checks in YAML under checks:. The hosted console and playground task files write the same objects as a JSON list. For how checks are evaluated, see Checks.

scene.yaml

checks:
  - id: answered
    state:
      slack.message: { channel: $slack_support, from: $slack_bot, in_thread_of: $slack_question }
  - id: filed
    trace: { called: "POST /repos/acme/orders/issues" }
pass: [answered, filed]

Task checks (console and playgrounds)

[
  {"id": "answered", "state": {"slack.message": {"channel": "$slack_support", "from": "$slack_bot", "in_thread_of": "$slack_question"}}},
  {"id": "filed", "trace": {"called": "POST /repos/acme/orders/issues"}}
]

Where each kind works

Kind	Scene files (`ancha run`, `ancha check`)	Hosted console tasks	Playground task files
`state`	Yes	Yes	Yes
`trace`	Yes	Yes	Yes
`judge`	Yes	No	No
Process checks	Yes	No	No

A playground task holds at most 20 checks.

Common fields

Field	Type	Meaning
`id`	string	Unique name of the check. Results and `pass` refer to it.
`state`	object	A condition on the final world state.
`trace`	object	A condition on the agent’s API calls.
`judge`	string	A rubric for a model to grade.
`revealed`, `knows`, `told`, `decided`, `message`, `never`, `before`, `within`	varies	A condition on what happened between the agent and the people in the world. See Process checks.

pass is a scene-level list of required check IDs. It defaults to every check.

Trace checks

Field	Type	Passes when
`called`	string or list	Every pattern matches at least one call by the agent that worked: a status below 400, not a coverage gap, no Slack `ok: false` and no GraphQL answer with only `errors`.
`not_called`	string or list	No call by the agent matches any pattern, failed attempts included.
`status`	`any`	With `called`: count every attempt, failed calls included.

Set at least one. With both, both must hold. A pattern is "<METHOD> <PATH>", or "*" for any call. The method ignores case, and * as the method accepts any method. The path is exact, relative to the tool’s base URL, without the query string. * in the path matches any run of characters, including /. Only the agent’s own calls count.

- id: posted
  trace: { called: "* /api/chat.postMessage" }
- id: read_before_write
  trace: { called: ["GET /repos/acme/orders/issues/13", "POST /repos/acme/orders/issues/13/comments"] }
- id: no_crm_writes
  trace: { not_called: ["POST /crm/*", "PATCH /crm/*", "DELETE /crm/*"] }

called with a list does not check order. For order, use a before check with called milestones.

State checks

Slack message

slack.message counts the messages in a channel that match every condition given and passes when there are at least count_gte.

Field	Required	Meaning
`channel`	Yes	Channel ref.
`from`	No	Author ref. Set it so starter history cannot match.
`text_contains`	No	Substring the message must contain, ignoring case.
`in_thread_of`	No	Message ref. Only replies in that thread count, not the parent.
`count_gte`	No	Minimum number of matching messages. Default `1`.

state:
  slack.message: { channel: $slack_support, from: $slack_bot, in_thread_of: $slack_question, text_contains: "Exports" }

GitHub

Check	Fields	Passes when
`github.file_at`	`[repo, revision, path]` plus exactly one of `contains` or `equals`	The file exists at that revision and contains, or equals, the text. Comparison is exact and case-sensitive.
`github.pull`	`pull` (pull request ref), `merged` (boolean)	The pull request’s merged state equals `merged`.
`github.production`	`repo`, `commit` (refs)	Production is exactly that commit.
`github.issue`	`repo`, then any of `title_contains`, `body_contains`, `labels`, `state`, and a count	Issues in the repository (not pull requests) match. With no count, at least one must match.

| github.permission | { repo, user }, then is | The user’s effective access to the repository is is: read, write, admin or none. | revision is a branch name, a commit ref, default (the default branch) or production. github.issue fields:

Field	Meaning
`repo`	Repository ref.
`title_contains`, `body_contains`	A string, or a list of strings that must all appear. Case-insensitive.
`labels`	The issue has every one of these labels.
`state`	`open` or `closed`.
`count`	Exactly this many issues match.
`count_gte`, `count_lte`	At least, or at most, this many. Not together with `count`.

- id: runbook_updated
  state: { github.file_at: [$github_orders, main, docs/runbook.md], contains: "verified health check" }
- id: pr_merged
  state: { github.pull: { pull: $github_runbook_pr, merged: true } }
- id: issue_filed
  state: { github.issue: { repo: $github_orders, body_contains: [CSV, export], labels: [bug], state: open, count: 1 } }

Datadog

Check	Fields	Passes when
`datadog.monitor`	monitor ref, `is`: `OK` or `Alert`	The monitor’s current state is `is`.
`datadog.downtime`	`monitor`, `remaining_gte`, `duration_lte`, optional `reject_active_unsafe: true`	An active, uncanceled downtime targets exactly that monitor, has at least `remaining_gte` left and lasts at most `duration_lte` in total. With `reject_active_unsafe`, it also fails if any active all-monitor or overlong downtime exists.

- id: monitor_ok
  state: { datadog.monitor: $datadog_orders, is: OK }
- id: muted_briefly
  state: { datadog.downtime: { monitor: $datadog_orders, remaining_gte: 1m, duration_lte: 10m } }

PagerDuty

pagerduty.incident selects one incident and checks at least one property.

Field	Meaning
`pagerduty.incident`	`{ ref: <incident ref> }`, or `{ service: <service ref>, first: true }` for the earliest incident of a service, including ones created during the attempt.
`status`	`triggered`, `acknowledged` or `resolved`.
`assigned_to`	List of user refs: the exact final set of assignees, in any order.
`acknowledged_within`	Duration from creation to the first real acknowledgment.
`resolved_within`	Duration from creation to the first real resolution.

- id: resolved_fast
  state:
    pagerduty.incident: { service: $pagerduty_orders, first: true }
    status: resolved
    resolved_within: 5m

Fault resolved

fault_resolved: <fault ref> passes when a living-world fault declared in world_rules.faults is resolved. See Test file.

Okta

okta.user: <login or ref> with status, status_in or exists: false passes when the user has that status, has one of the statuses in status_in, or no longer exists.

Other tools

The other tools have no state checks. Check what the agent did with trace checks.

Process checks

Process checks read what happened between the agent and the people in the world: who learned which fact, who decided what, which messages reached whom, and in what order. The world’s engine records these as the attempt runs. A check reads the records, never the text a model wrote, and nothing a person writes can end an attempt.

checks:
  - { id: asked_laura, revealed: [laura.workspace, laura.beta_on] }
  - { id: ben_got_the_details, knows: [ben, laura.beta_on] }
  - { id: laura_got_the_workaround, told: [laura, ben.workaround] }
  - { id: priority_decided, decided: [mike, severity, p1] }
  - { id: answered_in_her_thread, message: { to: laura, in_thread_of: $slack_laura_question, from: $slack_bot } }
  - { id: no_root_cause_to_customer, never: { to: laura, fact: ben.root_cause } }
  - { id: priority_before_label, before: [{ decided: [mike, severity, p1] }, { called: "POST /repos/*/issues/*/labels" }] }
  - { id: answered_fast, within: { event: { message: { to: laura, from: $slack_bot } }, of: start, limit: 30m } }

Kind	Value	Passes when
`revealed`	`[<person>.<fact>, ...]`	The owner of each fact shared it with the agent because the agent asked for it.
`knows`	`[<person>, <fact>, ...]`	The person knows every listed fact, learned from the agent or from another person.
`told`	`[<person>, <fact>, ...]`	The same as `knows`, for checks that read better that way.
`decided`	`[<person>, <decision>]` or `[<person>, <decision>, <option>]`	The person recorded that decision, with that option when one is given.
`message`	A message selector plus `count_gte` (default `1`)	At least `count_gte` messages match.
`never`	A message selector	No message ever matched.
`before`	A list of at least two milestones	Every milestone happened, each one strictly earlier than the next in simulated time.
`within`	`{ event, of, limit }`	`event` happened no later than `limit` after `of` (`start` by default, or another milestone).

Facts and decisions are written <person>.<id>, for example laura.beta_on. A person is written as their bare ref (laura), $laura or $person.laura.

Message selectors

message, never and message milestones select Slack messages and GitHub issues and comments written during the attempt. The seed never counts. Give at least one field; every field given must hold.

Field	Meaning
`to`	A person the message reaches: a direct message to them, a mention, a thread they are in, or a channel they listen to.
`in`	Where: a Slack channel, or a GitHub issue or pull request.
`in_thread_of`	A reply in the thread of this Slack message.
`from`	The author: a user ref such as `$slack_bot`, which also names the agent in GitHub, or a person.
`text_contains`	Case-insensitive. A list means every string must appear.
`fact`	The message told its recipient this fact, by the match the engine recorded when the message reached them.

A failed message check names the closest miss, for example a reply in the thread of the wrong question.

Milestones

before and within compare the times of milestones. A milestone is one of:

Milestone	Time it happened
`{ called: "<METHOD> <PATH>" }`	The agent’s first call that matches the pattern. `*` is a wildcard.
`{ message: { ... } }`	The first message that matches the selector.
A world predicate, such as `{ decided: [mike, severity, p1] }` or `{ issue_labeled: { repo: $github_orders, labels: [p1] } }`	The first time it held. See Predicates.

A milestone that never happened fails both before and within, so an attempt that did nothing cannot pass them. { issue_labeled: { repo, labels } } covers labels given when an issue is created as well as labels added later, which a called milestone cannot.

Judge checks

- id: helpful_reply
  judge: "The reply tells the customer where to find the CSV export and that it can take two minutes to appear."

Needs OPENROUTER_API_KEY in the environment of ancha run or ancha check. Set judge.model in ancha.yaml to choose the OpenRouter model.
The model sees the agent’s API calls to Slack, GitHub, Datadog and PagerDuty and the final Slack messages, never credentials, the seed or other checks.
Without a key, the check is skipped. A skipped check never passes. Keep advisory rubrics out of pass.
A failed model call makes the trial invalid, not failed.

Refs

Fields that point at records take refs: $name, or $kind.name to also check the record’s kind. Kinds are user, channel, message, repo, commit, pull, issue, fault, monitor, service, escalation_policy, schedule, incident, dashboard, event, log and person. A ref must exist in the world: validation rejects a check that names an unknown ref.

Durations

Durations are strings such as 500ms, 30s, 5m, 1h or 1m30s.

Results

Field	Meaning
`id`, `required`	The check and whether it is in `pass`.
`passed`	`true` or `false`.
`status`	`passed`, `failed` or `skipped` (judge checks without a key).
`expected`, `actual`, `explanation`	What was wanted, what was found, and one sentence with both.
`evidence`	Sequence numbers of the trace entries behind the result.

Each trial’s checks.json holds every result. ancha explain prints the failed ones, and ancha check --json prints them all for a live world.

Get started

Guides

Concepts

Tools

CLI

Reference

Cloud

Help

Checks reference

Where each kind works

Common fields

Trace checks

State checks

Slack message

GitHub

Datadog

PagerDuty

Fault resolved

Okta

Other tools

Process checks

Message selectors

Milestones

Judge checks

Refs

Durations

Results

​Where each kind works

​Common fields

​Trace checks

​State checks

​Slack message

​GitHub

​Datadog

​PagerDuty

​Fault resolved

​Okta

​Other tools

​Process checks

​Message selectors

​Milestones

​Judge checks

​Refs

​Durations

​Results

Where each kind works

Common fields

Trace checks

State checks

Slack message

GitHub

Datadog

PagerDuty

Fault resolved

Okta

Other tools

Process checks

Message selectors

Milestones

Judge checks

Refs

Durations

Results