Chaos Application Faults

Chaos Application Faults (s17)

Purpose

Produce complete, safe, runnable application-level chaos experiments that test service resilience against pod failures, container crashes, network degradation, DNS misconfiguration, and dependency timeouts.

---

Prerequisites

• [ ] Chaos experiment designs from s14 (Experiment Design)
• [ ] Target application deployed and accessible
• [ ] Steady state defined from s17 (Steady State)
• [ ] Blast radius approved from s16 (Blast Radius Control)
• [ ] Observability active and monitoring application metrics

Input Contract

| Input | Source | Required | |---|---|---| | Experiment design framework | s12 (workflowcontext.artifacts) | Yes | | Service definitions + pod labels | s05 output | Yes | | Blast radius constraints | s14 output | Yes | | Steady state probes | s15 output | Yes | | Risk tolerance preferences | s02 taste (risktolerance) | Yes |

Output Contract

| Output | Destination | Format | |---|---|---| | App fault ChaosExperiments | .commandcode/artifacts/experiment-app-.yaml | Litmus YAML | | App fault ChaosEngines | .commandcode/artifacts/engine-app-.yaml | Litmus YAML | | Network fault configurations | .commandcode/artifacts/experiment-network-.yaml | Litmus YAML | | Recovery runbook | s18 (game day), s25 (postmortem) | Markdown |

---

Application Fault Catalog

| Fault | LitmusChaos Name | Risk | Scope | Prod Safe? | |---|---|---|---|---| | Pod Delete | pod-delete | Low | Namespaced | Yes (≤30%) | | Pod CPU Hog | pod-cpu-hog | Low | Namespaced | Yes (≤30%) | | Pod Memory Hog | pod-memory-hog | Medium | Namespaced | Yes (≤30%) | | Container Kill | container-kill | Medium | Namespaced | Yes (≤30%) | | Pod HTTP Latency | pod-http-latency | Low | Namespaced | Yes | | Pod Network Loss | pod-network-loss | High | Namespaced | With approval | | Pod Network Latency | pod-network-latency | Medium | Namespaced | Yes | | Pod DNS Error | pod-dns-error | Medium | Namespaced | With approval | | Pod DNS Spoof | pod-dns-spoof | High | Namespaced | No (staging only) | | Pod IO Stress | pod-io-stress | Medium | Namespaced | With approval |

---

Fault 1: Pod Delete (Baseline, Always Run First)

# Generated by: hybrid-harness-chaos-process-prm
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: -pod-delete-engine
  namespace: 
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: pod-failure
spec:
  appinfo:
    appns: 
    applabel: "app="
    appkind: deployment
  chaosServiceAccount: litmus-admin
  monitoring: true
  jobCleanUpPolicy: retain
  annotationCheck: "true"    # Only target annotated pods
  engineState: active
  experiments:
    - name: pod-delete
      spec:
        probe:
          - name: application-health-check
            type: httpProbe
            mode: Continuous
            httpProbe/inputs:
              url: "http://..svc.cluster.local/health"
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            runProperties:
              probeTimeout: 5s
              interval: 5s
              retry: 2
              probePollingInterval: 2s
              stopOnFailure: true
          - name: error-rate-check
            type: promProbe
            mode: Edge
            promProbe/inputs:
              endpoint: "http://prometheus.monitoring.svc.cluster.local:9090"
              query: |
                sum(rate(http_requests_total{
                  namespace="",
                  pod=~".*",
                  status=~"5.."
                }[2m])) /
                sum(rate(http_requests_total{
                  namespace="",
                  pod=~".*"
                }[2m])) * 100
              comparator:
                type: float
                criteria: ""   # Specific container in pod
      - name: CONTAINER_KILL_SIGNAL
        value: "SIGKILL"        # SIGTERM | SIGKILL
      - name: PODS_AFFECTED_PERC
        value: "50"
      - name: RAMP_TIME
        value: "15"

---

Fault 3: Pod CPU Hog

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-cpu-hog
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: resource-exhaustion
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "90"
      - name: CPU_CORES
        value: "1"              # Cores to consume
      - name: CPU_LOAD
        value: "80"             # % utilization
      - name: PODS_AFFECTED_PERC
        value: "50"
      - name: CONTAINER_RUNTIME
        value: "containerd"

---

Fault 4: Pod Memory Hog

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-memory-hog
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: resource-exhaustion
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "90"
      - name: MEMORY_CONSUMPTION_MEGABYTES
        value: "512"            # MB to consume
      - name: MEMORY_PERCENTAGE
        value: "80"             # % of pod memory limit
      - name: PODS_AFFECTED_PERC
        value: "50"
      - name: NUMBER_OF_WORKERS
        value: "4"

---

Fault 5: Network Latency (Service-to-Service)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: -to--latency-engine
  namespace: 
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: network-degradation
spec:
  appinfo:
    appns: 
    applabel: "app="  # Source service pods
    appkind: deployment
  chaosServiceAccount: litmus-admin
  monitoring: true
  engineState: active
  experiments:
    - name: pod-network-latency
      spec:
        probe:
          - name: circuit-breaker-check
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              endpoint: "http://prometheus.monitoring.svc.cluster.local:9090"
              query: |
                circuit_breaker_state{
                  service="",
                  dependency=""
                }
              comparator:
                type: string
                criteria: "=="
                value: "open"       # Expected: circuit breaker triggers
            runProperties:
              probeTimeout: 10s
              interval: 15s
              retry: 3
          - name: user-error-rate
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              endpoint: "http://prometheus.monitoring.svc.cluster.local:9090"
              query: |
                sum(rate(frontend_errors_total[1m])) /
                sum(rate(frontend_requests_total[1m])) * 100
              comparator:
                type: float
                criteria: ""   # Only latency to this service
            - name: NETWORK_INTERFACE
              value: "eth0"

---

Fault 6: Network Loss (Complete Disconnection)

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-network-loss
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: network-partition
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "60"             # Keep short — total network loss is severe
      - name: NETWORK_PACKET_LOSS_PERCENTAGE
        value: "100"            # Complete loss
      - name: TARGET_PODS
        value: ""
      - name: PODS_AFFECTED_PERC
        value: "100"            # All pods in scope lose network
      - name: DESTINATION_IPS
        value: ""               # Empty = all traffic
      - name: NETWORK_INTERFACE
        value: "eth0"

---

Fault 7: DNS Error Injection

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-dns-error
  namespace: litmus
  labels:
    managed-by: hcprm
    fault-category: application
    fault-type: dns-failure
spec:
  definition:
    scope: Namespaced
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "120"
      - name: TARGET_SERVICE_NAME
        value: ""     # Service whose DNS is poisoned
      - name: MATCH_SCHEME
        value: "exact"            # exact | prefix | suffix | regex
      - name: ERROR_TYPE
        value: "error"            # error | random
      - name: PODS_AFFECTED_PERC
        value: "100"

---

Fault Escalation Ladder

Always follow this progression within the application fault phase:

1. Pod Delete (50%) → validate self-healing
    ↓ PASS
2. Container Kill → validate container restart
    ↓ PASS
3. Pod CPU Hog → validate resource throttling
    ↓ PASS
4. Pod Memory Hog → validate OOM handling
    ↓ PASS
5. Network Latency (100ms → 250ms → 500ms) → validate timeouts/circuit breakers
    ↓ PASS
6. Network Loss (scoped to single service) → validate graceful degradation
    ↓ PASS
7. DNS Error (scoped to non-critical dependency) → validate fallback

---

Circuit Breaker Verification

During network chaos, validate circuit breaker behavior:

// circuit-breaker-monitor.ts — Runs alongside chaos experiment
import { register, collectDefaultMetrics, Gauge } from 'prom-client';

const circuitBreakerState = new Gauge({
  name: 'circuit_breaker_state',
  help: '0=closed, 1=open, 2=half_open',
  labelNames: ['service', 'dependency'],
});

async function monitorCircuitBreaker(service: string, dependency: string) {
  const state = await fetchCircuitBreakerMetrics(service, dependency);

  // Expected state transitions during latency injection:
  // T0:     CLOSED (normal)
  // T0+15s: CLOSED (latency building up)
  // T0+25s: OPEN (threshold exceeded — circuit breaker trips)
  // T0+90s: HALF_OPEN (probe requests sent after cooldown)
  // T0+120s: CLOSED (fault removed, circuit recovers)

  if (state === 'open') {
    console.log('✅ Circuit breaker opened as expected — service is protected');
  } else {
    console.warn(`⚠️ Circuit breaker state: ${state} — expected "open"`);
  }
}

---

Recovery Runbook

Pod Delete

# Self-healing — Kubernetes restarts pods automatically
# Verify: kubectl get pods -n  -l app= -w
# Expected recovery: /health
# Expected: latency returns to  -n  -- nslookup 
# If persistent: kubectl rollout restart deployment/coredns -n kube-system

---

AI Agent Integration

Autonomy Level

| Aspect | Level | Description | |---|---|---| | Current | L1 | AI generates application fault manifests | | Target | L2 | AI generates pod/container/network/DNS fault YAML |

Harness AI Agent

Agent: Harness AI Reliability Agent Capabilities:

• Application fault manifest generation (pod-delete, container-kill, network, DNS)
• Fault Flags for application-level chaos
• Target selection based on service topology

Human Gates

• Production application fault approval
• Fault Flag activation

MCP

• LitmusChaos MCP
• Harness Fault Flags

Industry Context

Application-level chaos is critically underrepresented in practice (3.0% of experiments per GitHub study of 971 repos) -- this skill addresses a documented industry gap.

---

Success Criteria

• [ ] All 7 fault types designed and validated in dev
• [ ] Pod Delete passes with 100% health probe success
• [ ] Circuit breakers verified for network faults
• [ ] OOM handling triggers graceful shutdown (not crash loop)
• [ ] Fault escalation ladder followed (never skip to network loss)
• [ ] Recovery runbooks tested for each fault type
• [ ] All experiments have stopOnFailure: true on at least one probe
• [ ] Staging results documented before considering production

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: dungnotnull
• Source: dungnotnull/hybrid-harness-chaos-process-prm
• License: MIT

Install and usage instructions live in the source repository linked above.