Resilience Audit

Resilience audit

A happy-path pass says nothing about failure behavior. Force each material failure and observe the invariant directly.

1. Map failure boundaries

Trace critical operations across process, network, storage, queue, cache, filesystem, browser, and third-party boundaries. For each, name the owner, timeout, cancellation path, retry policy, idempotency key, cleanup duty, durable state, and user-visible outcome.

Complete when: each critical path has explicit failure and recovery invariants.

2. Build a pressure matrix

Select applicable faults:

• timeout before, during, and after side effects;
• cancellation and shutdown while work is active;
• retry after ambiguous success or partial write;
• dependency refusal, delay, disconnect, or malformed response;
• queue, cache, memory, descriptor, or process growth;
• overload and backpressure;
• restart, reconnect, stale state, and recovery;
• concurrent check-then-act or conflicting transitions;
• supported operating-system and filesystem differences.

Define expected state, observable signal, cleanup deadline, and safe retry outcome for each cell.

Complete when: every selected fault has an oracle that can distinguish safe degradation from silent corruption.

3. Force one fault at a time

Use bounded fault injection, controlled clocks, disposable state, test doubles at real boundaries, process signals, resource limits, or platform matrices. Capture before, during, and after state. Do not wait for rare failures to occur naturally.

Complete when: each executed cell has a reproducible trace and cleanup observation.

4. Classify the result

Classify outcomes as preserved invariant, graceful rejection, recoverable degradation, corrupted or duplicated state, leak, unbounded growth, retry storm, deadlock, or verification gap. A missing environment or unavailable dependency is a gap, not a pass.

Complete when: every matrix cell has evidence or an explicit reason it remains unverified.

5. Fix narrowly when authorized

Choose the smallest change that restores the violated invariant. Preserve retry budgets, cancellation propagation, atomicity, bounded resource ownership, and idempotency. Add a deterministic regression test that forces the same path.

Complete when: the forced failure is red before the fix, green after it, and normal behavior remains green.

6. Re-run under repetition and load

Repeat the failure, combine it with realistic concurrency or load, and inspect process, descriptor, memory, queue, cache, and durable-state cleanup.

Complete when: resource use returns to its declared bound and recovery remains deterministic.

7. Report and gate

Report the pressure matrix, traces, violated invariants, fixes, remaining gaps, operational signals, and rollback. Run the repository review gate when available.

Complete when: the report separates forced evidence from inference and no unexecuted cell is described as safe.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Railly
• Source: Railly/skills
• License: MIT

Install and usage instructions live in the source repository linked above.