# Chaos Application Faults

> >

- **Type:** Skill
- **Install:** `agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s19-application-faults`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [dungnotnull](https://agentstack.voostack.com/s/dungnotnull)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** https://github.com/dungnotnull/hybrid-harness-chaos-process-prm/tree/main/skills/s19-application-faults

## Install

```sh
agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s19-application-faults
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Chaos Application Faults (s17)

## Purpose
Produce complete, safe, runnable application-level chaos experiments that test service resilience against pod failures, container crashes, network degradation, DNS misconfiguration, and dependency timeouts.

---

## Prerequisites
- [ ] Chaos experiment designs from s14 (Experiment Design)
- [ ] Target application deployed and accessible
- [ ] Steady state defined from s17 (Steady State)
- [ ] Blast radius approved from s16 (Blast Radius Control)
- [ ] Observability active and monitoring application metrics

## Input Contract

| Input | Source | Required |
|---|---|---|
| Experiment design framework | s12 (workflow_context.artifacts) | Yes |
| Service definitions + pod labels | s05 output | Yes |
| Blast radius constraints | s14 output | Yes |
| Steady state probes | s15 output | Yes |
| Risk tolerance preferences | s02 taste (risk_tolerance) | Yes |

## Output Contract

| Output | Destination | Format |
|---|---|---|
| App fault ChaosExperiments | `.commandcode/artifacts/experiment-app-.yaml` | Litmus YAML |
| App fault ChaosEngines | `.commandcode/artifacts/engine-app-.yaml` | Litmus YAML |
| Network fault configurations | `.commandcode/artifacts/experiment-network-.yaml` | Litmus YAML |
| Recovery runbook | s18 (game day), s25 (postmortem) | Markdown |

---

## Application Fault Catalog

| Fault | LitmusChaos Name | Risk | Scope | Prod Safe? |
|---|---|---|---|---|
| Pod Delete | pod-delete | Low | Namespaced | Yes (≤30%) |
| Pod CPU Hog | pod-cpu-hog | Low | Namespaced | Yes (≤30%) |
| Pod Memory Hog | pod-memory-hog | Medium | Namespaced | Yes (≤30%) |
| Container Kill | container-kill | Medium | Namespaced | Yes (≤30%) |
| Pod HTTP Latency | pod-http-latency | Low | Namespaced | Yes |
| Pod Network Loss | pod-network-loss | High | Namespaced | With approval |
| Pod Network Latency | pod-network-latency | Medium | Namespaced | Yes |
| Pod DNS Error | pod-dns-error | Medium | Namespaced | With approval |
| Pod DNS Spoof | pod-dns-spoof | High | Namespaced | No (staging only) |
| Pod IO Stress | pod-io-stress | Medium | Namespaced | With approval |

---

## Fault 1: Pod Delete (Baseline, Always Run First)

```yaml
# Generated by: hybrid-harness-chaos-process-prm
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: -pod-delete-engine
  namespace: 
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: pod-failure
spec:
  appinfo:
    appns: 
    applabel: "app="
    appkind: deployment
  chaosServiceAccount: litmus-admin
  monitoring: true
  jobCleanUpPolicy: retain
  annotationCheck: "true"    # Only target annotated pods
  engineState: active
  experiments:
    - name: pod-delete
      spec:
        probe:
          - name: application-health-check
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: "http://..svc.cluster.local/health"
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            runProperties:
              probeTimeout: 5s
              interval: 5s
              retry: 2
              probePollingInterval: 2s
              stopOnFailure: true
          - name: error-rate-check
            type: promProbe
            mode: Edge
            promProbe/inputs:
              endpoint: "http://prometheus.monitoring.svc.cluster.local:9090"
              query: |
                sum(rate(http_requests_total{
                  namespace="",
                  pod=~".*",
                  status=~"5.."
                }[2m])) /
                sum(rate(http_requests_total{
                  namespace="",
                  pod=~".*"
                }[2m])) * 100
              comparator:
                type: float
                criteria: ""   # Specific container in pod
      - name: CONTAINER_KILL_SIGNAL
        value: "SIGKILL"        # SIGTERM | SIGKILL
      - name: PODS_AFFECTED_PERC
        value: "50"
      - name: RAMP_TIME
        value: "15"
```

---

## Fault 3: Pod CPU Hog

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-cpu-hog
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: resource-exhaustion
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "90"
      - name: CPU_CORES
        value: "1"              # Cores to consume
      - name: CPU_LOAD
        value: "80"             # % utilization
      - name: PODS_AFFECTED_PERC
        value: "50"
      - name: CONTAINER_RUNTIME
        value: "containerd"
```

---

## Fault 4: Pod Memory Hog

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-memory-hog
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: resource-exhaustion
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "90"
      - name: MEMORY_CONSUMPTION_MEGABYTES
        value: "512"            # MB to consume
      - name: MEMORY_PERCENTAGE
        value: "80"             # % of pod memory limit
      - name: PODS_AFFECTED_PERC
        value: "50"
      - name: NUMBER_OF_WORKERS
        value: "4"
```

---

## Fault 5: Network Latency (Service-to-Service)

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: -to--latency-engine
  namespace: 
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: network-degradation
spec:
  appinfo:
    appns: 
    applabel: "app="  # Source service pods
    appkind: deployment
  chaosServiceAccount: litmus-admin
  monitoring: true
  engineState: active
  experiments:
    - name: pod-network-latency
      spec:
        probe:
          - name: circuit-breaker-check
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              endpoint: "http://prometheus.monitoring.svc.cluster.local:9090"
              query: |
                circuit_breaker_state{
                  service="",
                  dependency=""
                }
              comparator:
                type: string
                criteria: "=="
                value: "open"       # Expected: circuit breaker triggers
            runProperties:
              probeTimeout: 10s
              interval: 15s
              retry: 3
          - name: user-error-rate
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              endpoint: "http://prometheus.monitoring.svc.cluster.local:9090"
              query: |
                sum(rate(frontend_errors_total[1m])) /
                sum(rate(frontend_requests_total[1m])) * 100
              comparator:
                type: float
                criteria: ""   # Only latency to this service
            - name: NETWORK_INTERFACE
              value: "eth0"
```

---

## Fault 6: Network Loss (Complete Disconnection)

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-network-loss
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: network-partition
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "60"             # Keep short — total network loss is severe
      - name: NETWORK_PACKET_LOSS_PERCENTAGE
        value: "100"            # Complete loss
      - name: TARGET_PODS
        value: ""
      - name: PODS_AFFECTED_PERC
        value: "100"            # All pods in scope lose network
      - name: DESTINATION_IPS
        value: ""               # Empty = all traffic
      - name: NETWORK_INTERFACE
        value: "eth0"
```

---

## Fault 7: DNS Error Injection

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-dns-error
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: dns-failure
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "120"
      - name: TARGET_SERVICE_NAME
        value: ""     # Service whose DNS is poisoned
      - name: MATCH_SCHEME
        value: "exact"            # exact | prefix | suffix | regex
      - name: ERROR_TYPE
        value: "error"            # error | random
      - name: PODS_AFFECTED_PERC
        value: "100"
```

---

## Fault Escalation Ladder

Always follow this progression within the application fault phase:

```
1. Pod Delete (50%) → validate self-healing
    ↓ PASS
2. Container Kill → validate container restart
    ↓ PASS
3. Pod CPU Hog → validate resource throttling
    ↓ PASS
4. Pod Memory Hog → validate OOM handling
    ↓ PASS
5. Network Latency (100ms → 250ms → 500ms) → validate timeouts/circuit breakers
    ↓ PASS
6. Network Loss (scoped to single service) → validate graceful degradation
    ↓ PASS
7. DNS Error (scoped to non-critical dependency) → validate fallback
```

---

## Circuit Breaker Verification

During network chaos, validate circuit breaker behavior:

```typescript
// circuit-breaker-monitor.ts — Runs alongside chaos experiment
import { register, collectDefaultMetrics, Gauge } from 'prom-client';

const circuitBreakerState = new Gauge({
  name: 'circuit_breaker_state',
  help: '0=closed, 1=open, 2=half_open',
  labelNames: ['service', 'dependency'],
});

async function monitorCircuitBreaker(service: string, dependency: string) {
  const state = await fetchCircuitBreakerMetrics(service, dependency);

  // Expected state transitions during latency injection:
  // T0:     CLOSED (normal)
  // T0+15s: CLOSED (latency building up)
  // T0+25s: OPEN (threshold exceeded — circuit breaker trips)
  // T0+90s: HALF_OPEN (probe requests sent after cooldown)
  // T0+120s: CLOSED (fault removed, circuit recovers)

  if (state === 'open') {
    console.log('✅ Circuit breaker opened as expected — service is protected');
  } else {
    console.warn(`⚠️ Circuit breaker state: ${state} — expected "open"`);
  }
}
```

---

## Recovery Runbook

### Pod Delete
```bash
# Self-healing — Kubernetes restarts pods automatically
# Verify: kubectl get pods -n  -l app= -w
# Expected recovery: /health
# Expected: latency returns to  -n  -- nslookup 
# If persistent: kubectl rollout restart deployment/coredns -n kube-system
```

---

## AI Agent Integration

### Autonomy Level

| Aspect | Level | Description |
|---|---|---|
| Current | L1 | AI generates application fault manifests |
| Target | L2 | AI generates pod/container/network/DNS fault YAML |

### Harness AI Agent

**Agent**: Harness AI Reliability Agent
**Capabilities**:
- Application fault manifest generation (pod-delete, container-kill, network, DNS)
- Fault Flags for application-level chaos
- Target selection based on service topology

### Human Gates

- Production application fault approval
- Fault Flag activation

### MCP

- LitmusChaos MCP
- Harness Fault Flags

### Industry Context

Application-level chaos is critically underrepresented in practice (3.0% of experiments per GitHub study of 971 repos) -- this skill addresses a documented industry gap.

---

## Success Criteria
- [ ] All 7 fault types designed and validated in dev
- [ ] Pod Delete passes with 100% health probe success
- [ ] Circuit breakers verified for network faults
- [ ] OOM handling triggers graceful shutdown (not crash loop)
- [ ] Fault escalation ladder followed (never skip to network loss)
- [ ] Recovery runbooks tested for each fault type
- [ ] All experiments have `stopOnFailure: true` on at least one probe
- [ ] Staging results documented before considering production

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** [dungnotnull/hybrid-harness-chaos-process-prm](https://github.com/dungnotnull/hybrid-harness-chaos-process-prm)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-dungnotnull-hybrid-harness-chaos-process-prm-s19-application-faults
- Seller: https://agentstack.voostack.com/s/dungnotnull
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.