ananke

Assertion-based testing tool for AG-UI applications.

Features

• Run scripted multi-turn conversations against AG-UI endpoints
• Observe tool calls and assistant responses
• Evaluate deterministic assertions (no LLM-as-judge)
• CI-friendly with exit codes and JSON output

Installation

npm install -g @scoutqa/ananke

Or run directly with npx:

npx @scoutqa/ananke run

Quick Start

1. Create a config file

Create ananke.config.yaml in your project root:

version: "1.0"

target:
  type: agui
  endpoint: "https://your-app.com/ag-ui"
  agentId: "my-agent"
  headers:
    Authorization: "Bearer ${ENV.AGUI_TOKEN}"

2. Write a test file

Create tests/example.test.yaml:

version: "1.0"

name: basic conversation

turns:
  - user: "Hello, can you help me?"
    assert:
      text:
        matches: "help|assist"

  - user: "What's 2 + 2?"
    assert:
      text:
        matches: "4"

3. Run tests

ananke run

CLI Usage

Usage: ananke run [options] [patterns...]

Run test files

Arguments:
  patterns             Test file patterns (glob)

Options:
  -c, --config <path>  Path to config file
  -d, --dry-run        Validate tests without executing
  --json               Output results as JSON
  -h, --help           display help for command

Environment Variables

Variable	Description	Default
ANANKE_LOG_LEVEL	Log level: silent, info, debug, trace	info
ANANKE_LOG_TIMESTAMP	Show timestamps in log output: 1 to enable	(off)

Configuration

Project Config (ananke.config.yaml)

version: "1.0"

target:
  type: agui
  endpoint: "https://app.example.com/ag-ui"
  agentId: "my-agent"
  headers:
    Authorization: "Bearer ${ENV.AGUI_TOKEN}"
    X-Custom-Header: "value"
  # Optional: default assertions for all tests
  assert:
    response:
      having:
        durationMs: { max: 60000 }
    text:
      not:
        matches: ["exception", "fatal"]

Variables:

• ${ENV.NAME} - Environment variable
• ${VAR.NAME} - Variable from hooks or script turns

Test File Format

version: "1.0"

name: test name

hooks: # Optional setup scripts
  - cmd: ["bash", "scripts/setup.sh"]
    timeout_ms: 10000

turns:
  - user: "User message"
    assert: # Turn-level assertions (inherits from target and test)
      tool_names:
        some: { equals: "tool_name" }
        none: { equals: "forbidden_tool" }
      tools:
        some:
          having:
            name: { equals: "tool_name" }
            args.arg_name: { matches: "regex" }
      response:
        having:
          durationMs: { max: 30000 }
      text:
        matches: "regex"
        not:
          matches: ["error", "failed"]

assert: # Test-level assertions (inherits from target)
  tool_names:
    none: { equals: "dangerous_tool" }
  response:
    having:
      durationMs: { max: 120000 }
      idleMs: { max: 60000 }

Assertions

See docs/assertions-v2.md for the full assertion reference.

Selectors

Selector	Type	Description
text	string	Assistant response text
tool_names	array	Tool call names
tools	array	Full tool call objects
response	object	Response metrics (use having to assert on fields)

Common Operators

Operator	Description
equals	Exact match (string, number, boolean, null)
contains	Substring (string) or element membership (array)
matches	Regex pattern(s) that must all match
min, max	Number comparisons
count	Number assertion on array length
some, every, none	Element-level array assertions
ordered	Elements appear in order
filter	Keep matching elements, assert on sub-array
having	Dot-path shorthand for object field assertions
json	Parse JSON string (or pass through pre-parsed values)
and, or, not	Logical combinators

Hooks

Hooks run before test execution. They must output JSON to stdout with a variables key:

#!/bin/bash
echo '{"variables": {"THREAD_ID": "th_123", "USER_ID": "user_456"}}'

Variables from hooks are available as ${VAR.NAME} in turns and config:

turns:
  - user: "Check thread ${VAR.THREAD_ID}"

Script Turns

Script turns dynamically generate user messages based on previous turn data:

turns:
  - user: "Show shipping options"
    assert:
      tool_names:
        some: { equals: "get_shipping_options" }

  - type: script
    script: "scripts/pick-cheapest.sh"
    assert:
      tool_names:
        some: { equals: "calculate_total" }

Scripts receive an ANANKE env var (and stdin) with JSON containing value, turns, variables, and turnIndex. They output JSON with message (required), variables (optional), and action (optional).

See docs/unified-script-contract.md for the full script contract.

CI Integration

GitHub Actions

- name: Run AG-UI tests
  run: npx @scoutqa/ananke run --json > results.json
  env:
    AGUI_TOKEN: ${{ secrets.AGUI_TOKEN }}

- name: Upload results
  uses: actions/upload-artifact@v4
  with:
    name: test-results
    path: results.json

Exit Codes

• 0 - All tests passed
• 1 - One or more tests failed

License

MIT