Chaos Gameday

Chaos game day

A runbook that has never been followed under pressure is a guess, and an alert that has never fired in anger is a hope. A game day converts both into evidence by breaking something deliberately while you are watching and ready to stop. Done without rails it causes the outage it meant to prevent; done as a demo where everyone knows the answer it teaches nothing. The craft is the middle.

Method

1. Start from a hypothesis tied to a real risk. Write it as a prediction:

"if the primary database fails over, reads recover within 30 seconds and no alert pages a human unnecessarily." A game day without a falsifiable claim is just breaking things. Pick failures you actually fear: dependency timeouts, zone loss, a poisoned cache, disk full.

1. Define steady state and the abort condition first. Name the metric that

says the system is healthy (checkout success rate, p99 latency) and the threshold that ends the exercise immediately. Decide the blast radius up front: one shard, one zone, one percent of traffic. No injection starts until the kill switch is built and tested.

1. Assign roles and keep responders blind. A facilitator drives the

scenario and holds the abort switch, a scribe timestamps everything, and the on-call responders work the problem without knowing the injection details. If the people responding already know the answer, you are measuring their memory, not the system.

1. Announce the window, do not ambush. Tell the broader org the game day is

running and when, so a real incident in parallel is not mistaken for the drill. Ambush drills damage trust and are rarely worth the realism. Run in production when you can afford it and staging when you cannot, and say which.

1. Inject gradually and watch the instruments, not just the system. Use a

fault tool (Gremlin, AWS Fault Injection Simulator, toxiproxy, a scripted dependency kill) and escalate in steps. The primary question is not only "did it survive" but "did monitoring detect it, did the right alert fire, did the runbook match reality." Measure time to detect and time to recover.

1. Abort the instant a rail trips, and treat that as data. If steady state

breaches the threshold, stop and restore. An exercise that hits its abort condition found a real weakness faster than a real incident would have: that is a success, not a failure of the game day.

1. Capture learnings while the room is still warm. Within a day, write what

broke, what the instruments missed, and which runbook step was wrong. File action items with owners, then schedule the re-run that proves the fix. An unrepeated game day is a story; a repeated one is a control.

Litmus tests

• Could the facilitator stop the exercise in seconds, and was that switch tested

before any fault was injected?

• Did the responders discover the failure through your monitoring, or because

someone told them it was happening?

• Did you measure time to detect and time to recover, not just "it stayed up"?
• Is there a scheduled re-run to confirm each fix, or did the findings just get

filed?

Boundaries

A game day tests a system you can safely perturb and restore. Do not inject faults into a system with no rollback, no isolation, or unquantified blast radius: harden it first. This rehearses resilience; it does not replace the production-readiness review that decides a service is fit to operate, nor the incident process that handles a real outage. Follow your org's change-management rules for any production injection.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Amey-Thakur
• Source: Amey-Thakur/AI-SKILLS
• License: MIT
• Homepage: https://amey-thakur.github.io/AI-SKILLS/

Install and usage instructions live in the source repository linked above.