# Chaos Hypothesis Validation

> >

- **Type:** Skill
- **Install:** `agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s15-hypothesis-validation`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [dungnotnull](https://agentstack.voostack.com/s/dungnotnull)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** https://github.com/dungnotnull/hybrid-harness-chaos-process-prm/tree/main/skills/s15-hypothesis-validation

## Install

```sh
agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s15-hypothesis-validation
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Chaos Hypothesis Validation

## Purpose
Transform vague "what if X breaks?" questions into precise, falsifiable hypotheses with measurable success criteria — making chaos experiments scientifically rigorous rather than random fault injection.

---

## Prerequisites
- [ ] Chaos experiment designs from s14 (Experiment Design)
- [ ] Steady state baselines from s17 (Steady State Definition)
- [ ] Target service metrics identified
- [ ] Observability stack active (Prometheus, Grafana)
- [ ] Performance baselines from s13 (Performance Testing)

## Input Contract

| Input | Source | Required |
|---|---|---|
| Experiment designs (ChaosEngine YAML) | s12 (workflow_context.artifacts) | Yes |
| Steady state baselines | s15 output | Yes |
| Service SLIs/SLOs from PRD | s01 context | Yes |
| Blast radius constraints | s14 output | Yes |
| Observability metrics available | s20 output | No |

## Output Contract

| Output | Destination | Format |
|---|---|---|
| Hypothesis document per experiment | `.commandcode/artifacts/hypothesis-.md` | Markdown |
| Validation script (Python) | `.commandcode/artifacts/hypothesis-validator.py` | Python |
| Pre-experiment check script | `.commandcode/artifacts/pre-chaos-check.sh` | Bash |
| Hypothesis tracker table | s24, s25 (scoring + learning) | Markdown table |
| Acceptance criteria YAML | s18 (game day gates) | YAML |

---

## The Chaos Hypothesis Formula

```
HYPOTHESIS STATEMENT:
"When [FAULT DESCRIPTION] is applied to [TARGET SCOPE] for [DURATION],
the [SYSTEM/SERVICE] will [EXPECTED BEHAVIOR],
as evidenced by [MEASURABLE METRIC] remaining [CONDITION] (e.g., below 5%, above 99.9%)."

ACCEPTANCE CRITERIA:
- [METRIC_1]: [OPERATOR] [THRESHOLD] (e.g., error_rate 

**Fault**: Delete 50% of `` pods every 10 seconds for 60 seconds.

**Pre-conditions (Steady State)**:
- HTTP health check returns 200: ✓
- Error rate (5xx): ` pods are deleted every 10 seconds for 60 seconds,
the service will continue to serve requests successfully due to Kubernetes
self-healing and load balancer session persistence,
as evidenced by HTTP error rate staying ≤ 5% and P99 latency staying ≤ 500ms
throughout the experiment duration.

**Acceptance Criteria**:
| Metric | Threshold | Measurement Method |
|---|---|---|
| HTTP 5xx error rate | ≤ 5% | Prometheus `http_requests_total{status=~"5.."}` |
| P99 latency | ≤ 500ms | Prometheus histogram_quantile(0.99) |
| Health endpoint | Returns 200 | HTTP probe every 5s |
| Pod recovery time | ≤ 30s | Kubernetes event watcher |

**Null Hypothesis (failure condition)**:
The service will return 5xx errors > 5% OR P99 latency will exceed 500ms
during the pod deletion period.

**Expected Learning**:
- Kubernetes HPA / ReplicaSet guarantees self-healing within 30s
- Service mesh / kube-proxy routes away from terminating pods
- Any gap = finding requiring action (PodDisruptionBudget, readiness probe tuning)
```

### Template 2: Network Partition
```markdown
## Hypothesis: Network Latency Resilience —  → 

**Fault**: Inject 500ms network latency between  and  for 120s.

**Hypothesis Statement**:
When 500ms network latency is injected on all traffic from `` to ``,
`` will activate its circuit breaker after 3 consecutive timeouts (5s timeout threshold),
return a cached/fallback response to end users,
and recover automatically within 30 seconds of latency removal.

**Acceptance Criteria**:
| Metric | Threshold | Evidence |
|---|---|---|
| Circuit breaker opens | Within 15s of latency onset | Circuit breaker state metric |
| User-facing error rate | ≤ 1% (fallback serves) | Frontend error tracking |
| Fallback response correctness | Returns valid fallback JSON | Functional probe |
| Recovery time after fault removal | ≤ 30s | HTTP probe re-establishes 200 |

**System Assumptions Being Tested**:
-  has circuit breaker configured with 5s timeout, 3 failure threshold
- Fallback response is cached and not stale > 5 minutes
- Circuit breaker half-open probe interval is 10s
```

### Template 3: Infrastructure Failure
```markdown
## Hypothesis: Single Node Failure — 

**Fault**: Drain one Kubernetes worker node for 5 minutes.

**Hypothesis Statement**:
When one worker node is drained in ``,
all workloads will reschedule to remaining healthy nodes within 2 minutes,
stateful services will recover without data loss,
and the platform SLO (99.9% availability) will be maintained throughout.

**Acceptance Criteria**:
| Service | Availability Target | Recovery Time Target |
|---|---|---|
| `payment-service` | 100% (no downtime) | N/A (immediate reschedule) |
| `order-service` | ≥ 99% | ≤ 60s |
| `postgres` (via StatefulSet) | 100% (follower promotes) | ≤ 120s |
| Overall platform | ≥ 99.9% | N/A |

**Anti-requirements (must NOT happen)**:
- No data loss in PostgreSQL write-ahead log
- No orphaned PersistentVolumes
- No alert pages (error rate stays below PagerDuty threshold)
```

---

## Hypothesis Validation Workflow

### Phase 1: Pre-Experiment Validation (T-10 minutes)
```bash
# Verify steady state is healthy BEFORE starting chaos
echo "=== Pre-Experiment Steady State Check ==="
echo "1. HTTP Health:"
curl -f http://..svc.cluster.local/health && echo "PASS" || echo "FAIL"

echo "2. Error Rate (last 5m):"
curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100' \
  | jq '.data.result[0].value[1]'

echo "3. P99 Latency (last 5m, ms):"
curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode 'query=histogram_quantile(0.99, rate(http_request_duration_ms_bucket[5m])) * 1000' \
  | jq '.data.result[0].value[1]'

echo "4. Pod Count:"
kubectl get pods -n  -l app= --no-headers | wc -l
```

### Phase 2: During Experiment
```bash
# Poll metrics every 10 seconds during fault injection
watch -n 10 'curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode "query=sum(rate(http_requests_total{status=~\"5..\"}[1m])) / sum(rate(http_requests_total[1m])) * 100" \
  | jq ".data.result[0].value[1]"'
```

### Phase 3: Post-Experiment Validation
```python
# hypothesis_validator.py — run after experiment completes
import requests
import json
from datetime import datetime, timedelta

PROMETHEUS_URL = "http://prometheus:9090"
EXPERIMENT_START = "2025-01-15T10:00:00Z"
EXPERIMENT_END   = "2025-01-15T10:05:00Z"

def query_range(promql, start, end, step="15s"):
    r = requests.get(f"{PROMETHEUS_URL}/api/v1/query_range", params={
        "query": promql,
        "start": start,
        "end": end,
        "step": step,
    })
    return r.json()["data"]["result"]

# Evaluate acceptance criteria
results = {
    "error_rate_max": max(
        float(v[1]) for series in query_range(
            'sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m])) * 100',
            EXPERIMENT_START, EXPERIMENT_END
        ) for v in series["values"]
    ),
    "p99_latency_max": max(
        float(v[1]) for series in query_range(
            'histogram_quantile(0.99, rate(http_request_duration_ms_bucket[1m]))',
            EXPERIMENT_START, EXPERIMENT_END
        ) for v in series["values"]
    ),
}

THRESHOLDS = {
    "error_rate_max": 5.0,      # max 5% error rate
    "p99_latency_max": 500.0,   # max 500ms p99 latency
}

print("=== Hypothesis Validation Results ===")
passed = True
for metric, value in results.items():
    threshold = THRESHOLDS[metric]
    status = "✅ PASS" if value  threshold:
        passed = False

print(f"\n{'✅ HYPOTHESIS CONFIRMED' if passed else '❌ HYPOTHESIS REJECTED'}")
```

---

## Hypothesis Tracker (Markdown Table)

Maintain this in your chaos runbook wiki:

```markdown
| ID | Service | Fault | Hypothesis | Date | Result | Score | Finding | Ticket |
|---|---|---|---|---|---|---|---|---|
| CHX-001 | payment | pod-delete 50% | Service stays available | 2025-01-10 | ✅ PASS | 95 | Minor latency spike at t=30s | — |
| CHX-002 | checkout | network-latency 500ms | Circuit breaker activates | 2025-01-12 | ❌ FAIL | 45 | Circuit breaker not configured | ENG-4421 |
| CHX-003 | auth | pod-delete 100% | Auth tokens cached in Redis | 2025-01-15 | ✅ PASS | 88 | Redis TTL could be longer | ENG-4438 |
```

---

## AI Agent Integration

### Autonomy Level

| Aspect | Level | Description |
|---|---|---|
| Current | L1 | AI assists in hypothesis writing and validation |
| Target | L2 | AI generates validation scripts and steady-state probes |

### Harness AI Agent

**Agent**: Harness AI Reliability Agent
**Capabilities**:
- Steady-state validation script generation
- Probe configuration for HTTP/CMD/Prometheus
- Hypothesis structure suggestions

### Human Gates

- Hypothesis acceptance/rejection
- Validation criteria approval

### MCP

- LitmusChaos MCP (resilience probes)
- Prometheus

---

## Success Criteria
- [ ] Every chaos experiment has a written hypothesis before running
- [ ] Hypothesis includes specific measurable thresholds (not vague "stays healthy")
- [ ] Pre-experiment steady state check passes before fault injection
- [ ] Post-experiment validation script produces PASS/FAIL verdict
- [ ] Results recorded in hypothesis tracker with ticket for every FAIL
- [ ] FAIL hypotheses generate remediation tickets within 24h

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** [dungnotnull/hybrid-harness-chaos-process-prm](https://github.com/dungnotnull/hybrid-harness-chaos-process-prm)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-dungnotnull-hybrid-harness-chaos-process-prm-s15-hypothesis-validation
- Seller: https://agentstack.voostack.com/s/dungnotnull
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.