Your system handles 80,000 requests per second across 50 microservices. Your oncall runbooks say “if Payment Service goes down, Order Service degrades gracefully within 3 seconds.” But you have never actually verified that. A vendor outage at 2 AM becomes the first real test, and you discover that Order Service does not degrade gracefully - it hammers Payment Service with retries until its own connection pool exhausts, then it locks up entirely, cascading to six downstream services that were never in the blast radius of the original failure. The incident postmortem recommends chaos engineering. Six months later you have zero infrastructure for running it safely.

The problem with chaos engineering is not convincing engineers it is a good idea. It is the operational safety problem: how do you break things on purpose without accidentally triggering a real incident? The moment you introduce a tool that deliberately injects failures, every stakeholder in the room has the same objection - “what if it goes wrong?” The answer has to be architectural. Blast radius control, automatic halts, and ironclad rollback cannot be afterthoughts bolted on after the first accident. Think of it like controlled burns in forestry: the whole value proposition of a controlled burn depends on the firebreaks being in place before you light the match, not after you see the fire spreading.

The engineering challenge here is that we need to build a system that is simultaneously powerful enough to simulate realistic failure conditions and conservative enough to earn the trust of every team whose services are targets. That trust is built through three concrete mechanisms: a dependency graph that knows which services are safe to target and which are excluded, a steady-state hypothesis framework that verifies conditions are normal before and during every experiment, and an automatic rollback trigger that halts injection and reverts configuration within 30 seconds of any SLO degradation.

Requirements and Constraints

Functional Requirements

• Run controlled experiments that inject one or more fault types into target services: latency addition (P50/P99 configurable), packet loss percentage, CPU spike to configurable percentage, memory pressure, disk I/O throttling, and pod/instance kill
• Enforce blast radius limits: maximum N% of instances per service, maximum M services per experiment, explicit exclusion list for critical services
• Verify steady-state hypothesis before experiment start and continuously during injection
• Automatically halt and revert faults when any monitored SLO breaches its threshold
• Track a service dependency graph and use it to classify targets, control groups, and excluded services
• Store full experiment history with per-second metric snapshots for post-experiment analysis
• Support scheduled experiments (cron-style) and on-demand execution
• Emit events to PagerDuty, Slack, and audit logs for all experiment lifecycle transitions

Non-Functional Requirements

• Auto-halt latency: SLO breach detected and faults reverted within 30 seconds end-to-end
• Blast radius: never affect more than 5% of total service instances globally in any single experiment
• Concurrent experiment limit: max 20 experiments running simultaneously across all regions, enforced globally
• Availability of the chaos platform itself: 99.9% for the control plane, 99.99% for the rollback path (it must halt even if the rest of the platform is down)
• Audit trail: every experiment start, halt, and completion stored with the triggering SLO metric value
• Agent overhead on target hosts: under 0.5% CPU, under 50MB resident memory

Constraints and Assumptions

• We inject at the OS/kernel layer (tc netem for network faults, cgroups for resource faults) for accuracy, not at the application layer
• Kubernetes is the primary deployment target; we also support VM-based services via SSH-based agents
• Metrics are available via Prometheus/Grafana; SLO definitions are pre-existing PromQL expressions
• Production experiments require explicit approval from a service owner before the scheduler can queue them

High-Level Architecture

The platform has six major components working in two planes: a control plane that plans and approves experiments, and an execution plane that injects and monitors them.

The Experiment Planner takes a YAML experiment spec from an engineer or the scheduler, validates it against the policy engine, and hands it to the Dependency Graph Engine. The Dependency Graph Engine maintains a live picture of service relationships scraped from service mesh telemetry and service registries - it classifies every service in the target list as a viable target, a mandatory exclusion, or a suggested control group. The Blast Radius Calculator takes that classification and applies percentage limits: if you ask to kill 50% of Payment Service pods but the global limit is 5% of instances, it rejects the experiment before any fault fires.

The SLO Watchdog is the most important component in the platform. It runs continuously, independent of whether any experiment is active, polling Prometheus for the SLO metrics of every service that has ever been a target. When an experiment is live, it tightens its polling interval from 60 seconds to 15 seconds and evaluates SLO expressions against tighter thresholds. The Fault Injection Engine executes the actual fault primitives - it is intentionally stateless; it receives a signed experiment token, applies the fault, and does nothing else. If it crashes mid-experiment, the Rollback Controller detects the missing heartbeat and reverts. The Rollback Controller maintains a rollback manifest for every active experiment: a deterministic set of operations that restore pre-experiment state, independent of the Fault Engine’s liveness.

The Experiment Scheduler

Chaos experiments are not ad-hoc - mature implementations run them on a regular cadence, the way you run load tests or security scans. Netflix Chaos Monkey became famous precisely because it ran automatically on a schedule, not because engineers manually clicked “inject failure” when they felt like it.

The Scheduler solves two problems that interact in a non-obvious way: preventing experiment collisions and respecting maintenance windows. Two experiments running simultaneously on overlapping services invalidate each other’s baselines - if Payment Service is already experiencing injected latency, starting a second experiment on Order Service produces results that reflect the compound effect, not the isolated one.