Test a Slack bot

This guide tests a small Slack support bot built with @slack/web-api. You write a scene with a #support channel, two questions and checks for the right reply, run the bot three times, find a bug from the evidence, and prove the fix. It takes about ten minutes and needs no Slack workspace and no API keys.

Before you start

Node.js 24 or later.
The Ancha CLI: npm install -g @ancha-labs/cli. See Quickstart.

1. Create the bot

mkdir slack-bot && cd slack-bot
npm init -y
npm pkg set type=module
npm install @slack/web-api

Save this as bot.mjs. It reads #support, picks a question, and replies in its thread with an answer from a small playbook. It has a bug that Ancha will find.

bot.mjs

// A small support bot: answers the newest question in #support, in its thread.
import { WebClient } from '@slack/web-api';

const slack = new WebClient(process.env.SLACK_BOT_TOKEN, { slackApiUrl: process.env.SLACK_API_URL });

const playbook = [
  { topic: /invite/i, answer: 'An admin can invite a teammate from Settings → Members → Invite.' },
  { topic: /csv|export/i, answer: 'Reports → Exports, choose the month, then Download CSV. It can take two minutes to appear.' },
];

const { channels } = await slack.conversations.list();
const support = channels.find((channel) => channel.name === 'support');
const { messages } = await slack.conversations.history({ channel: support.id }); // newest first

const question = messages.findLast((message) => message.text.endsWith('?'));
const entry = playbook.find((item) => item.topic.test(question.text));
const answer = entry ? entry.answer : 'Thanks, a teammate will follow up shortly.';

await slack.chat.postMessage({ channel: support.id, thread_ts: question.ts, text: answer });
console.log(`Answered: ${question.text}`);

The bot reads SLACK_API_URL and SLACK_BOT_TOKEN, so the same code runs against real Slack when you give it real values.

2. Set up Ancha

ancha init --agent "node bot.mjs"
rm -r scenes/starter

init writes ancha.yaml with your command, a starter scene and .ancha/ in .gitignore. Because package.json depends on @slack/web-api, the starter only checks that the bot reaches Slack. Remove it; you will write a scene for this bot.

3. Write the scene

Save this as scenes/support-answer/scene.yaml:

scenes/support-answer/scene.yaml

id: support-answer
title: Answer the newest question in #support
twins: [slack]
world: static
start_time: 2026-10-26T14:00:00Z
timeout: 1m
task: Answer the newest question in Slack #support, in its thread.
seed:
  slack:
    team: { name: Acme }
    bot: $bot
    users:
      - { ref: bot, name: ancha, real_name: Ancha assistant, is_bot: true }
      - { ref: ben, name: ben, real_name: Ben Carter }
      - { ref: mike, name: mike, real_name: Mike Turner }
    channels:
      - { ref: support, name: support, members: [$bot, $ben, $mike] }
    messages:
      - ref: csv_question
        channel: $support
        user: $mike
        at: -35m
        text: "Ridgeway Robotics cannot find last month's CSV export. Where should I point them?"
      - channel: $support
        user: $ben
        at: -32m
        thread_of: $csv_question
        text: "Reports → Exports. Choose the month, then Download CSV."
      - ref: invite_question
        channel: $support
        user: $mike
        at: -2m
        text: "Ridgeway Robotics wants to add a teammate. How do they invite someone?"
checks:
  - id: answered_newest
    state:
      slack.message:
        channel: $support
        from: $bot
        in_thread_of: $invite_question
        text_contains: "Settings → Members → Invite"
  - id: posted_a_reply
    trace: { called: "POST /api/chat.postMessage" }
pass: [answered_newest, posted_a_reply]

The world has an old question that Ben already answered and a new one from two minutes ago. The checks say what a good outcome looks like:

answered_newest looks at the final Slack state: the bot replied in the thread of the newest question, with the right instructions. from: $bot keeps Ben’s earlier answer from counting.
posted_a_reply looks at the bot’s calls. It uses POST because the SDK calls every Slack method with POST.

Check the file:

ancha scene validate scenes/support-answer/scene.yaml

Output

ok  scenes/support-answer/scene.yaml (support-answer)

4. Run it three times

ancha run -n 3

Output

running locally (worlds in a server this command starts)
run_started runId=run_20261031-103344_2137d5d4f255e3a3 scenes=["support-answer"] trials=3 parallel=1 workspace=local:/path/to/slack-bot target=local
trial_started scene=support-answer trial=1 envId=imz4odkx
trial_finished scene=support-answer trial=1 passed=false valid=true warnings=[] exit=exited exitCode=0 realMs=92 failedChecks=["answered_newest"]
trial_started scene=support-answer trial=2 envId=yusbkdtu
trial_finished scene=support-answer trial=2 passed=false valid=true warnings=[] exit=exited exitCode=0 realMs=86 failedChecks=["answered_newest"]
trial_started scene=support-answer trial=3 envId=jv8tr2mp
trial_finished scene=support-answer trial=3 passed=false valid=true warnings=[] exit=exited exitCode=0 realMs=82 failedChecks=["answered_newest"]

✗ support-answer  0/3 (0%)
    ✗ answered_newest  failed in 3/3 trials
      trial 1: The check wanted ≥1 message in #support from @ancha in reply to @mike: "Ridgeway Robotics wants to add a teammate. How…" containing "Settings → Members → Invite" but found 0 matching; @ancha has 1 message(s) in #support, 0 in that thread.

agent not sealed: network open; environment limited to 9 variables from this shell (pass more with --pass-env NAME)
fail (minimum pass rate 100%)  run run_20261031-103344_2137d5d4f255e3a3
next: ancha explain run_20261031-103344_2137d5d4f255e3a3

The command exits with code 1. Each trial ran on a fresh copy of the channel, and all three failed the same way. The bot got only Ancha’s variables and a small base set from your shell, which is all it needs.

5. Find out why

ancha explain latest --trial 1

Output

support-answer  trial 1  ✗ answered_newest
  expected: ≥1 message in #support from @ancha in reply to @mike: "Ridgeway Robotics wants to add a teammate. How…" containing "Settings → Members → Invite"
  actual:   0 matching; @ancha has 1 message(s) in #support, 0 in that thread
  why:      The check wanted ≥1 message in #support from @ancha in reply to @mike: "Ridgeway Robotics wants to add a teammate. How…" containing "Settings → Members → Invite" but found 0 matching; @ancha has 1 message(s) in #support, 0 in that thread.
  trace:
    #1 agent POST /api/conversations.list -> 200
    #2 agent POST /api/conversations.history -> 200
    #3 agent POST /api/chat.postMessage -> 200

environment: 70 variables from this shell were withheld from the agent. If it needs one (a provider key, say), pass it with --pass-env NAME or list it under agent.pass_env in ancha.yaml.

next: ancha baseline save before --from run_20261031-103344_2137d5d4f255e3a3, then fix the agent, then ancha run --compare before

Without --trial 1, explain prints all three trials. The bot did post one message, just not in the newest question’s thread. The trial’s files show where it went:

RUN=$(cat .ancha/runs/latest)
cat .ancha/runs/$RUN/trial-support-answer-1/agent.log
grep chat.postMessage .ancha/runs/$RUN/trial-support-answer-1/trace.jsonl

agent.log says Answered: Ridgeway Robotics cannot find last month's CSV export. Where should I point them?, and the request in trace.jsonl carries that question’s thread_ts. The bot answered the oldest question. conversations.history returns messages newest first, and findLast takes the last match, which is the oldest.

6. Fix it and prove the fix

Save the failing run as the reference, then fix the bot:

ancha baseline save before

In bot.mjs, change messages.findLast( to messages.find( so the bot takes the newest question. Then run again and compare:

ancha run -n 3 --compare before

Output

✓ support-answer  3/3 (100%)

compared with run_20261031-103344_2137d5d4f255e3a3: no regression
  support-answer: 0% → 100% (+100 pp); 3 → 3 trials
    improved: answered_newest  0/3 → 3/3
agent not sealed: network open; environment limited to 9 variables from this shell (pass more with --pass-env NAME)
pass (minimum pass rate 100%)  run run_20261031-103345_81849c0329dfa0fd

Exit code 0: every check passed and no scene and no check got worse than before.

Next steps

Grade the wording. Add a judge check, and keep it out of pass while you tune it. Without OPENROUTER_API_KEY it is skipped with a warning and does not change the verdict:
- id: helpful judge: "The reply tells the person exactly where in the product to invite a teammate."
Cover more cases. Add scenes for a question the playbook does not know, a question in a thread, or no question at all. Each scene is a file under scenes/.
Test a model-based bot. Its answers vary, so run more trials and set min_pass_rate in ancha.yaml. List its provider key under agent.pass_env; Ancha withholds the rest of your shell from the bot. See ancha.yaml.
React to events. A bot that listens for messages reads Slack events from the trial’s event feed, or by HTTP or Socket Mode. See Slack.
Run it in CI. See CI.
Hand it to a coding agent. See For coding agents.

Get started

Guides

Concepts

Tools

CLI

Reference

Cloud

Help

Test a Slack bot

Before you start

1. Create the bot

2. Set up Ancha

3. Write the scene

4. Run it three times

5. Find out why

6. Fix it and prove the fix

Next steps

​Before you start

​1. Create the bot

​2. Set up Ancha

​3. Write the scene

​4. Run it three times

​5. Find out why

​6. Fix it and prove the fix

​Next steps

Before you start

1. Create the bot

2. Set up Ancha

3. Write the scene

4. Run it three times

5. Find out why

6. Fix it and prove the fix

Next steps