Fault Taxonomy

16 fault kinds across 6 worlds

Every fault isolates a specific failure mode. Deterministic seeding ensures your agent faces the same challenge every run.

Tier 1 Faults

High-Impact Failure Modes

These faults reveal critical flaws in agent reasoning. A well-built agent can safely handle them, but most default setups will fail catastrophically.

HARMFUL_ACTIONtimeout_after_commit

Applies the tool's side effect (e.g., executing a charge), but drops the response. The agent sees a timeout and cannot distinguish "never happened" from "happened, response lost."

Why it matters: Tests the classic non-idempotent-retry bug. LLM agents are structurally predisposed to blindly retrying failed calls.

SILENT_FAILUREsilent_wrong_data

Returns a well-formed, schema-valid response containing incorrect values (wrong magnitude, wrong entity, or stale data).

Why it matters: Tests if the agent sanity-checks data against independent sources or flags implausibility, rather than blindly acting on bad data.

SILENT_FAILUREomission

Returns empty or truncated results, dropping array elements, or omitting optional fields. Bypasses error handling entirely since the response is valid.

Why it matters: Agents often assume absence of data means truth, leading to hallucinations if they fabricate content to fill gaps.

HARMFUL_ACTIONpartial_failure_mid_plan

Fails a task mid-execution (e.g., step 3 of 7), after initial steps have already mutated the world state.

Why it matters: The most common cause of weird data states. Tests if the agent rolls back, retries sanely, or abandons the task silently.

The Differentiator

MCP Semantics Awareness

Unlike simple byte-matching proxies, Chaosline understands the Model Context Protocol (MCP). It can inject structural and semantic faults that traditional proxies are blind to.

schema_violating_output

Returns structuredContent that violates the tool's declared outputSchema. Most clients don't validate, causing invisible failures.

annotation_lie

Flips readOnlyHint on destructive tools, testing if an agent reasons based on lies.

wrong_error_channel

Swaps isError channels with JSON-RPC errors, testing disparate error handling paths.

capability_downgrade

Strips advertised capabilities from meta, testing graceful degradation.

stale_cache

Manipulates ttlMs to make the agent act on expired tool lists.

tool_list_drift

Removes or renames a tool between list and call phases.

Essential Coverage

Foundational tests for basic robustness and retry discipline.

timeout / hang

Tests basic retry and backoff discipline.

rate_limit_429

Tests respect for Retry-After headers.

malformed_response

Tests parser robustness against bad JSON.

schema_drift

Reaction to a changed tool contract.

auth_expiry

Tests mid-run token expiration handling.

empty_result

Distinguishing 'no data' from 'broken'.

slow_but_ok

Latency tolerance without false failure.

retry_storm

Cost blowup detection on persistent failures.

Mock Worlds

Payments

Payment processing, refunds, transactions. Tests idempotency and financial risk.

Database

Data integrity, reads, writes, consistency. Tests rollback logic.

Email

Message delivery, inbox state. Tests sensitive outbound comms.

Filesystem

File operations, directory state.

HTTP

API responses, status codes, timeouts.

Search

Query results, indexing, handling absence of data.