# Chaos Observability Integration

> >

- **Type:** Skill
- **Install:** `agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s22-observability-integration`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [dungnotnull](https://agentstack.voostack.com/s/dungnotnull)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** https://github.com/dungnotnull/hybrid-harness-chaos-process-prm/tree/main/skills/s22-observability-integration

## Install

```sh
agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s22-observability-integration
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Chaos Observability Integration (s20)

## Purpose
Establish comprehensive observability for chaos experiments — ensuring every fault is visible, every probe is connected to real metrics, and every team member can see what's happening in real-time during chaos runs.

---

## Prerequisites
- [ ] Continuous verification config from s21 (CV Verification)
- [ ] Chaos experiment results from s14-s19
- [ ] Prometheus and Grafana deployed and accessible
- [ ] Service metrics endpoints available
- [ ] Dashboard creation permissions in Grafana

## Input Contract

| Input | Source | Required |
|---|---|---|
| Steady state metrics definitions | s15 output | Yes |
| Experiment manifests (probe configs) | s12-s17 outputs | Yes |
| Observability tool preferences | s02 taste (observability) | Yes |
| Alert routing preferences | s02 taste or s01 | Yes |
| CV verification config | s19 output | No |

## Output Contract

| Output | Destination | Format |
|---|---|---|
| Chaos Grafana dashboard JSON | `.commandcode/artifacts/dashboard-chaos.json` | JSON |
| Prometheus alert rules for chaos | `.commandcode/artifacts/alerts-chaos.yaml` | YAML |
| Observability health check script | `.commandcode/artifacts/obs-check.sh` | Bash |
| Probe validation report | s18 (game day) | Markdown |
| Chaos metrics feed | s21 (alerting), s24 (scoring) | Prometheus queries |

---

## Chaos Observability Stack

```
Experiment Runner (LitmusChaos/HCE)
    │  Exposes chaos metrics on :8080/metrics
    ▼
Prometheus (scrapes every 15s)
    │
    ├── Grafana Dashboard (real-time chaos view)
    ├── AlertManager (chaos-specific alert rules)
    └── S24 Scoring Engine (resilience calculation)
```

---

## Step 1 — Chaos Grafana Dashboard

```json
{
  "dashboard": {
    "title": "Chaos Engineering — ",
    "uid": "chaos-",
    "tags": ["chaos", "resilience", ""],
    "timezone": "browser",
    "panels": [
      {
        "title": "Chaos Experiment Status",
        "type": "stat",
        "targets": [{
          "expr": "litmuschaos_experiment_verdict{app=''}",
          "legendFormat": "{{experiment_name}}"
        }],
        "fieldConfig": {
          "defaults": {
            "mappings": [
              {"type": "value", "value": "0", "text": "Passed"},
              {"type": "value", "value": "1", "text": "Failed"}
            ],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {"color": "green", "value": null},
                {"color": "red", "value": 1}
              ]
            }
          }
        }
      },
      {
        "title": "Resilience Score (Real-time)",
        "type": "gauge",
        "targets": [{
          "expr": "litmuschaos_resilience_score{app=''}"
        }],
        "fieldConfig": {
          "defaults": {
            "min": 0, "max": 100,
            "thresholds": {
              "steps": [
                {"color": "red", "value": null},
                {"color": "yellow", "value": 60},
                {"color": "green", "value": 80}
              ]
            }
          }
        }
      },
      {
        "title": "Error Rate During Chaos",
        "type": "timeseries",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{namespace='',status=~'5..'}[1m])) / sum(rate(http_requests_total{namespace=''}[1m])) * 100",
            "legendFormat": "Error Rate %"
          },
          {
            "expr": "5",
            "legendFormat": "Threshold (5%)"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "thresholds": {
              "steps": [
                {"color": "green", "value": null},
                {"color": "red", "value": 5}
              ]
            }
          }
        }
      },
      {
        "title": "P99 Latency During Chaos",
        "type": "timeseries",
        "targets": [{
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_ms_bucket{namespace=''}[1m])) by (le))",
          "legendFormat": "P99 (ms)"
        }]
      },
      {
        "title": "Pod Status During Chaos",
        "type": "stat",
        "targets": [
          {
            "expr": "count(kube_pod_status_ready{namespace='', condition='true'})",
            "legendFormat": "Ready"
          },
          {
            "expr": "count(kube_pod_status_ready{namespace='', condition='false'})",
            "legendFormat": "Not Ready"
          }
        ]
      },
      {
        "title": "Active Faults",
        "type": "table",
        "targets": [{
          "expr": "litmuschaos_active_experiments{namespace=''}",
          "format": "table"
        }]
      },
      {
        "title": "Probe Status",
        "type": "status-history",
        "targets": [{
          "expr": "litmuschaos_probe_success_total{namespace=''}",
          "legendFormat": "{{probe_name}}"
        }]
      }
    ]
  }
}
```

### Import Dashboard
```bash
curl -X POST http://grafana.company.com/api/dashboards/db \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d @.commandcode/artifacts/dashboard-chaos.json
```

---

## Step 2 — Chaos-Specific Prometheus Alert Rules

```yaml
groups:
  - name: chaos_engineering_alerts
    rules:
      - alert: ChaosProbeFailure
        expr: litmuschaos_probe_success_total == 0
        for: 30s
        labels:
          severity: critical
          context: chaos
        annotations:
          summary: "Chaos probe {{ $labels.probe_name }} failed"
          description: >
            Probe {{ $labels.probe_name }} for experiment
            {{ $labels.experiment_name }} has failed.
            The experiment may have been aborted automatically.
          runbook_url: "https://wiki.company.com/chaos/probe-failure"

      - alert: ChaosExperimentRunningTooLong
        expr: time() - litmuschaos_experiment_start_time_seconds > 600
        for: 1m
        labels:
          severity: warning
          context: chaos
        annotations:
          summary: "Chaos experiment running > 10 minutes"
          description: >
            Experiment {{ $labels.experiment_name }} has been running
            for over 10 minutes. Expected duration was shorter.
            Consider aborting if this is unexpected.

      - alert: ChaosErrorRateThresholdExceeded
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1m])) /
          sum(rate(http_requests_total[1m])) * 100 > 5
        for: 30s
        labels:
          severity: critical
          context: chaos
        annotations:
          summary: "Error rate exceeded 5% during chaos experiment"
          description: >
            Error rate is {{ $value }}% which exceeds the 5% chaos threshold.
            Experiments should be aborted immediately.

      - alert: ChaosNoObservability
        expr: absent(litmuschaos_experiment_verdict)
        for: 2m
        labels:
          severity: critical
          context: chaos
        annotations:
          summary: "Chaos observability metrics are missing"
          description: >
            LitmusChaos metrics are not being scraped by Prometheus.
            No chaos experiments should run without observability.

      - alert: ChaosResilienceScoreBelowThreshold
        expr: litmuschaos_resilience_score 
            Service {{ $labels.service }} resilience score is {{ $value }}.
            Do not promote to production until score ≥ 80.
```

---

## Step 3 — Observability Health Check

Run before any chaos experiment:

```bash
#!/bin/bash
# obs-health-check.sh — verify observability before chaos
set -euo pipefail

PROMETHEUS="http://prometheus.monitoring.svc.cluster.local:9090"
GRAFANA="http://grafana.company.com"

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo " Observability Health Check"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━"

# 1. Prometheus is reachable
if curl -sf "$PROMETHEUS/-/healthy" > /dev/null; then
  echo "✅ Prometheus: healthy"
else
  echo "❌ Prometheus: unreachable — ABORT"
  exit 1
fi

# 2. Chaos metrics are being scraped
if curl -sg "$PROMETHEUS/api/v1/query" \
  --data-urlencode 'query=litmuschaos_experiment_verdict' \
  | jq -e '.data.result | length > 0' > /dev/null; then
  echo "✅ Chaos metrics: being scraped"
else
  echo "⚠️  Chaos metrics: not detected — has LitmusChaos been installed?"
fi

# 3. Target service metrics available
SERVICE="${1:?Usage: $0  }"
NAMESPACE="${2:?}"

ERROR_QUERY=$(curl -sg "$PROMETHEUS/api/v1/query" \
  --data-urlencode "query=sum(rate(http_requests_total{namespace=\"$NAMESPACE\"}[1m]))" \
  | jq '.data.result | length')

if [ "$ERROR_QUERY" -gt 0 ]; then
  echo "✅ Service metrics: available for $SERVICE/$NAMESPACE"
else
  echo "❌ Service metrics: not found — is the service deployed?"
  exit 1
fi

# 4. Grafana dashboard accessible
DASHBOARD_UID="chaos-${SERVICE}"
if curl -sf "$GRAFANA/api/dashboards/uid/$DASHBOARD_UID" \
  -H "Authorization: Bearer ${GRAFANA_API_KEY}" > /dev/null; then
  echo "✅ Grafana dashboard: chaos-$SERVICE exists"
else
  echo "⚠️  Grafana dashboard: not found — create before game day"
fi

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo " Observability ready for chaos"
```

---

## Step 4 — LitmusChaos Metrics Scrape Config

```yaml
# prometheus-scrape-config.yaml
scrape_configs:
  - job_name: 'litmuschaos'
    scrape_interval: 15s
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [litmus, ]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: chaos-(runner|exporter|operator)
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: 'true'
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
```

---

## AI Agent Integration

### Autonomy Level

| Aspect | Level | Description |
|---|---|---|
| Current | L1 | AI generates dashboards and alert rules |
| Target | L2 | AI creates observability stack and correlates chaos metrics |

### Harness AI Agent

**Agent**: Harness AI SRE Agent
**Capabilities**:
- Dashboard generation
- Chaos metric correlation
- Alert rule creation and optimization
- Investigation dashboard (dynamic AI-powered single-pane-of-glass)

### Human Gates

- Dashboard publication
- Metric baseline acceptance
- Alert rule activation

---

## Success Criteria
- [ ] Chaos Grafana dashboard created with all 7 panels
- [ ] Prometheus scraping LitmusChaos metrics every 15s
- [ ] Alert rules configured for probe failure, error rate, and missing observability
- [ ] Observability health check script passes before any experiment
- [ ] Dashboard visible to all game day participants
- [ ] Alert routing verified (test alert fired and received)
- [ ] Metrics gap < 5% (continuous monitoring throughout experiment)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** [dungnotnull/hybrid-harness-chaos-process-prm](https://github.com/dungnotnull/hybrid-harness-chaos-process-prm)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-dungnotnull-hybrid-harness-chaos-process-prm-s22-observability-integration
- Seller: https://agentstack.voostack.com/s/dungnotnull
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.