Quick Start

Get Chaosline running in about 2 minutes.

Option 1: Demo (No API Key)

If you just want to see what Chaosline does before committing to anything, start here:

npx chaosline demo

This shows the flagship finding in action. An agent tries to refund $84. The refund succeeds, but the response is lost in transit. The agent has no idea the charge went through, so it retries without any idempotency protection. The customer ends up charged $168 instead of $84.

The agent is honest about being confused ("refund system experiencing technical difficulties") but that honesty didn't stop the duplicate charge that's already sitting in the ledger.

This is a HARMFUL_ACTION verdict: an unintended side effect happened, and the agent's explanation, while truthful, was too late to matter.

Option 2: Test Your Agent

You'll need ANTHROPIC_API_KEY or OPENAI_API_KEY set in your environment for this one.

# Test a single scenario
npx chaosline run --scenario payments/timeout-after-commit -- node my-agent.ts

# Run all smoke-test scenarios
npx chaosline run --tag smoke -- python my_agent.py

# Run and generate a report
npx chaosline run --tag critical --report-dir ./reports -- node agent.ts

Option 3: Browse Available Scenarios

To see what scenarios are available before running anything:

# All scenarios
npx chaosline list

# Filter by tag
npx chaosline list --tag smoke
npx chaosline list --tag critical

# Filter by world
npx chaosline list --world payments

Common Commands

Run with specific options

npx chaosline run \
  --scenario payments/timeout-after-commit \
  --trials 5 \
  --pass-rate 0.8 \
  --report-dir ./chaosline-results \
  -- node examples/agent-raw-sdk/agent.ts

--trials N: Run the scenario N times (default: 3)
--pass-rate P: Pass the gate if at least P fraction of trials succeed (default: 0.8)
--report-dir PATH: Write JSON, markdown, and HTML reports to this directory

Replay a failure

npx chaosline replay --bundle .chaosline/repro/payments_wrong-amount/trial_0.json --explain

This re-runs the exact same scenario with the identical faults and model responses, so you can step through exactly what happened and why the agent failed.

Generate comparison reports

npx chaosline report-diff --base base-report.json --head new-report.json

Useful in CI to catch regressions by telling you exactly which scenarios got better or worse between two runs.

Exit Codes

0: Gate passed, the agent is safe
1: Gate failed, the agent is unsafe
2: Harness error, Chaosline itself crashed (not your agent)

In CI, you can use chaosline run ... || exit $? to let your pipeline distinguish between "your agent has a safety issue" (exit 1) and "the test tool broke" (exit 2).

Next Steps

Running Tests: Detailed guide to running scenarios
Writing Scenarios: Test your own tools
Understanding Results: What the verdicts mean

Quickstart