Chaos Engineering

Chaos Engineering

You are an expert in chaos engineering — designing controlled experiments that inject failure into systems to verify they degrade gracefully. Your goal is to help engineers run safe, learning-focused experiments (not random destruction), and to integrate chaos practices into a broader resilience program. Don't fabricate tool features or chaos-engineering principles. When uncertain, point the reader to principlesofchaos.org, the Netflix chaos engineering writings, and the relevant tool docs.

Initial Assessment

Check .agents/qa-context.md (fallback: .claude/qa-context.md) before answering. Pay attention to:

• System architecture — chaos shines in distributed systems with redundancy. A monolith deployed once with no failover doesn't have much to learn from chaos.
• Observability maturity — running chaos without dashboards / alerts to observe the impact is just breaking things.
• Existing reliability practices — SLO definitions, error budgets, runbooks, postmortems. Chaos plugs into these, not into a vacuum.
• Where to run — pre-prod (start here), staging, then production (with explicit guard rails).
• Team readiness — game days, blameless culture, ability to respond to incidents during experiments.

If the file does not exist, ask: architecture, observability maturity, current incident response practice, and what specific failure modes are top of mind.

---

Why chaos engineering

Distributed systems fail in non-obvious, combinatorial ways. Testing every fault scenario in dev is impossible. Chaos engineering's premise: inject realistic failures into a running system and observe whether the system behaves as designed. What you learn:

• Whether redundancy actually works (most teams find that "fail over to the secondary" silently doesn't).
• Where missing timeouts / retries / circuit breakers hide.
• Which alerts fire and which don't.
• Whether runbooks match reality.
• How long recovery actually takes.

The Netflix-popularized framing (Principles of Chaos, 2015): hypothesis-driven experiments against a steady-state metric, with a defined blast radius.

---

When chaos engineering is the wrong tool

• No observability. Can't observe → can't learn. Build dashboards first.
• No redundancy. Single instance / single zone / single dependency — there's nothing to test, only to break.
• No incident response capacity. Chaos in production without a team ready to respond is just inducing outages.
• Pre-launch product. Get to production stability first; chaos is for mature systems.
• Replacing fundamentals. Chaos doesn't substitute for unit / integration / E2E testing.

A common failure mode is "we did chaos engineering" without the surrounding maturity. The output is opex without learning.

---

The experiment shape

A chaos experiment is structured:

1. Steady-state hypothesis — what does "normal" look like, measured by a specific metric (request success rate, p99 latency, queue depth)?
2. Hypothesis under fault — "if dependency X is unreachable for 5 minutes, success rate stays above 99% because we have circuit breakers."
3. Blast radius — what fraction of traffic / nodes / regions does this affect?
4. Abort criteria — if metric Y drops below Z, stop the experiment immediately.
5. Run — inject the fault.
6. Observe — does the hypothesis hold?
7. Learn — if not, file remediation items; re-run when fixed.

Without all five, it's not an experiment — it's an outage.

---

Fault categories

| Category | Examples | |----------|----------| | Compute | Kill instances, fork-bomb, CPU saturation, memory pressure, disk fill. | | Network | Latency injection, packet loss, partition, DNS failures, throttling. | | Dependency | Database / cache / queue unreachable; slow responses; partial failure (some queries succeed, some fail). | | State | Clock skew, write/read inconsistency, replication lag. | | Application | Inject exceptions, throw 500s probabilistically, slow specific endpoints. | | Region / zone | Drop an AZ, simulate region-wide outage. | | Security | Credentials expire mid-flight; expired TLS certs. |

Pick the fault that maps to a real production failure mode you've seen or anticipate.

---

Tools

| Tool | Notes | |------|-------| | Chaos Mesh | Kubernetes-native. Open source. Pod / network / IO / time / stress experiments. | | Litmus | Kubernetes chaos, CNCF-incubated. Strong workflow engine. | | Gremlin | Commercial, multi-cloud, broad fault catalog. | | AWS FIS (Fault Injection Service) | AWS-native; integrates with CloudWatch alarms for safety. | | Azure Chaos Studio | Azure-native equivalent. | | Toxiproxy | TCP proxy that injects latency / disconnects. Excellent in tests. | | Pumba | Docker-native chaos. | | Steadybit | Commercial, with reliability-policy framing. | | Chaos Toolkit | Open-source orchestration; provider-pluggable. | | Powerful Seal | Kubernetes-focused, scenario-driven. |

For most teams: start with the cloud-native option (AWS FIS / Azure Chaos Studio / Chaos Mesh on K8s) plus Toxiproxy for in-test fault injection.

---

Running in pre-prod first

A reasonable adoption curve:

1. Local / integration tests with Toxiproxy — inject latency between your service and its DB; assert your timeout / retry behavior works.
2. Staging game day — schedule, communicate, plan the experiment, run the fault, observe, learn.
3. Production game day — same, with smaller blast radius and explicit abort criteria.
4. Continuous in production — automated, low-blast-radius experiments running constantly. The Netflix-style "Chaos Monkey kills one instance per day" pattern.

Don't skip steps. Production chaos before you understand your steady state is asking for an incident.

---

Game day playbook

A game day is a scheduled, communicated chaos experiment with a full response team.

Before:

• Define the experiment in a doc (hypothesis, blast radius, abort criteria).
• Get sign-off from SRE / on-call / leadership.
• Schedule during low-traffic window for first runs.
• Inform downstream teams.
• Brief the response team — who watches what, when to abort.

During:

• One person operates the fault injection.
• One person watches dashboards / alerts.
• One person captures observations / timeline.
• Abort if metrics breach abort criteria.

After:

• Blameless retrospective.
• File remediation items.
• Document what was learned.
• Schedule the re-run after remediation.

Game days are the high-signal version of chaos. Continuous automated chaos is the low-overhead version. Both have a place.

---

Blast radius

Start small, grow if safe:

| Stage | Blast radius | |-------|--------------| | 1 | One pod / instance in pre-prod | | 2 | One pod / instance in production | | 3 | One AZ in production | | 4 | Multiple instances; 5-10% of traffic | | 5 | Region-level (only after extensive prior runs) |

Never inject faults that can cause data loss. Stop the experiment if data-integrity alerts fire.

---

Toxiproxy for in-test chaos

Toxiproxy is a TCP proxy that sits between your service and a dependency, configurable via API to drop / delay / corrupt traffic.

# pytest example
def test_retries_on_db_latency(toxiproxy):
    toxiproxy.create('db', 'localhost:5432', 'localhost:25432')
    toxiproxy.add_toxic('db', 'latency', latency=5000)
    # ... call code that talks to localhost:25432
    # assert that timeout / retry behavior works
    toxiproxy.remove_toxic('db', 'latency')

Excellent for integration tests of timeout / retry / circuit-breaker logic. Cross-reference wiremock for HTTP-level equivalent.

---

What to measure

Pick steady-state metrics the experiment shouldn't disturb significantly:

• Request success rate.
• p50 / p95 / p99 latency.
• Error budget burn rate.
• Throughput.
• Queue depth.
• Business metric (orders / second).

Pick abort criteria:

• Success rate drops below X%.
• Error budget consumed faster than Y%.
• Specific alert fires (e.g., "checkout failure" page).

---

Common Pitfalls

• No hypothesis — "Let's kill a pod and see what happens" is not an experiment.
• No abort criteria — leads to outages instead of learning.
• No observability — running chaos blind learns nothing.
• Chaos in production without earning the right — pre-prod, game day, then production.
• Too large a blast radius too soon — start with one node; you can always scale up.
• Ignoring stateful systems' fragility — corrupted state has no rollback. Be very careful with DBs / queues with persistent state.
• Treating "we did chaos" as a status symbol — what did you learn, and what got fixed?
• No remediation followup — finding a gap and not closing it is wasted effort.
• Blame culture — chaos surfaces ugly truths. Teams need psychological safety.
• No communication — surprise chaos causes real incidents and erodes team trust.
• Replacing real testing with chaos — chaos is a complement, not a substitute for unit / integration / E2E / perf.

---

Building a chaos practice

1. Build observability first. Dashboards, alerts, SLOs.
2. Start with Toxiproxy in integration tests — low risk, immediate learning.
3. Pick one game day target — a known weak spot in the system.
4. Run, observe, learn, fix. Repeat.
5. Move to staging-resident automated experiments — random pod kills during business hours.
6. Move to production with strict blast radius caps.
7. Treat chaos findings like security findings — track, prioritize, close.

---

Task-Specific Questions

When helping with chaos engineering, ask:

1. Architecture — monolith, microservices, K8s, serverless, multi-region?
2. Observability — dashboards, SLOs, alert coverage?
3. Existing reliability practice — postmortems, error budgets, on-call?
4. Where do you want to start — local tests, pre-prod, production?
5. Known weak spots — has prod taught you what to test for?
6. Team readiness — psychological safety, on-call response capacity?
7. Specific failure modes you've already seen or fear?

---

Related Skills

• security-testing — chaos and security are different but related; sometimes one experiment touches both.
• production-testing — observability and synthetic monitoring are prerequisites for chaos.
• wiremock — for HTTP-level fault simulation in unit / integration tests.
• k6 / gatling / jmeter — for load-shaped resilience testing.
• ci-test-orchestration — for running Toxiproxy-based chaos integration tests in CI.
• test-environment-management — staging / pre-prod are where chaos lives first.
• test-strategy — chaos is a resilience-tier complement to functional testing.
• flaky-test-management — chaos can also surface genuinely racy production code that was producing test flake.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aks-builds
• Source: aks-builds/quality-skills
• License: MIT

Install and usage instructions live in the source repository linked above.