# Disaster Recovery Business Continuity

> >

- **Type:** Skill
- **Install:** `agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s29-disaster-recovery`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [dungnotnull](https://agentstack.voostack.com/s/dungnotnull)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** https://github.com/dungnotnull/hybrid-harness-chaos-process-prm/tree/main/skills/s29-disaster-recovery

## Install

```sh
agentstack add skill-dungnotnull-hybrid-harness-chaos-process-prm-s29-disaster-recovery
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Disaster Recovery & Business Continuity (s29)

## Purpose
Move beyond component-level chaos into organization-level disaster resilience — defining RTO/RPO targets, implementing backup and restore procedures, orchestrating multi-region failover, and validating that the business can survive the worst-case scenario.

---

## Prerequisites
- [ ] Service topology and architecture diagrams
- [ ] Resilience scores from s26
- [ ] Compliance audit trail from s30 (if available)
- [ ] Cloud provider access for failover operations
- [ ] Velero or equivalent backup solution configured

## Input Contract

| Input | Source | Required |
|---|---|---|
| Infrastructure topology (regions, AZs, providers) | s01, s05 | Yes |
| Critical service inventory with dependencies | s01 (PRD component inventory) | Yes |
| Resilience scores (all services) | s26 output | Yes |
| Chaos experiment results (identifies single points of failure) | s14-s21 outputs | Yes |
| Backup infrastructure details | s05 (infra definitions) | Yes |
| RTO/RPO targets (business requirements) | s01 (PRD — NFR constraints) | Yes |
| Compliance requirements | s30 (audit/compliance) | No |

## Output Contract

| Output | Destination | Format |
|---|---|---|
| DR plan document | `.commandcode/artifacts/dr/dr-plan.md` | Markdown |
| RTO/RPO matrix per service | `.commandcode/artifacts/dr/rto-rpo-matrix.yaml` | YAML |
| Failover runbook | `.commandcode/artifacts/dr/failover-runbook.md` | Markdown |
| Backup configuration | `.commandcode/artifacts/dr/backup-config.yaml` | YAML |
| DR test schedule | `.commandcode/artifacts/dr/dr-test-schedule.yaml` | YAML |
| DR test results | s26 (resilience score contribution) | JSON |
| Recovery scripts | `.commandcode/artifacts/dr/scripts/` | Bash/Python |
| Business impact analysis | s30 (compliance evidence) | Markdown |

---

## DR Planning Framework

```
ASSESS → DESIGN → IMPLEMENT → TEST → MAINTAIN

Assess:   Business impact analysis, RTO/RPO targets, critical path mapping
Design:   Multi-region architecture, backup strategy, failover mechanism
Implement: Backup schedules, replication, DNS failover, infrastructure automation
Test:     DR drills, tabletop exercises, full region failover
Maintain: Quarterly review, update for new services, validate backups
```

---

## Step 1 — Business Impact Analysis (BIA)

```yaml
business_impact_analysis:
  services:
    - name: payment-service
      tier: 0  # Revenue-critical — down = lost money
      max_tolerable_downtime_minutes: 5
      rto_target_minutes: 5
      rpo_target_minutes: 1    # 50% of pods in us-east-1 are unhealthy
- [ ] AWS status page reports us-east-1 degradation
- [ ] PagerDuty alert: "multi-AZ failure us-east-1"
- [ ] On-call SRE declares emergency

## Phase 1: Detection (Auto — = 3

# Step 2: Scale up us-west-2 to handle full load
kubectl --context us-west-2 scale deployment/payment-service --replicas=8
kubectl --context us-west-2 scale deployment/checkout-api --replicas=8

# Step 3: Promote PostgreSQL read replica to primary
aws rds promote-read-replica \
  --db-instance-identifier postgres-payment-prod-us-west-2 \
  --region us-west-2

# Step 4: Failover DNS (Route53)
aws route53 update-traffic-policy-instance \
  --id  \
  --ttl 60 \
  --endpoint-type HTTP \
  --evaluate-target-health false

# Step 5: Shift traffic
# Route53 automatically routes to us-west-2 (health-based)
# Verify: dig payment.company.com → should resolve to us-west-2

# Step 6: Notify stakeholders
```

## Phase 4: Verification ( 6 hours old"
  exit 1
fi
echo "✅ Backup age OK"

# 2. Test restore (create temporary instance from latest backup)
echo "🔄 Testing restore..."
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier postgres-payment-restore-test \
  --db-snapshot-identifier "$(echo "$LATEST_BACKUP" | jq -r '.DBSnapshotIdentifier')" \
  --db-instance-class db.t3.medium \
  --region us-east-1

# Wait for restore to complete
aws rds wait db-instance-available \
  --db-instance-identifier postgres-payment-restore-test \
  --region us-east-1

# Verify data integrity
ENDPOINT=$(aws rds describe-db-instances \
  --db-instance-identifier postgres-payment-restore-test \
  --query 'DBInstances[0].Endpoint.Address' \
  --region us-east-1 \
  --output text)

ROW_COUNT=$(psql -h "$ENDPOINT" -U admin -d payments -tAc \
  "SELECT COUNT(*) FROM transactions WHERE created_at > NOW() - INTERVAL '24 hours'")

echo "📊 Recent transaction rows: $ROW_COUNT"

if [ "$ROW_COUNT" -lt 100 ]; then
  echo "⚠️  Low row count — possible data integrity issue"
fi

# Cleanup
aws rds delete-db-instance \
  --db-instance-identifier postgres-payment-restore-test \
  --skip-final-snapshot \
  --region us-east-1

echo "✅ Backup validation complete"
```

---

## DR Maturity Model

| Level | Name | Characteristics |
|---|---|---|
| 0 | None | No DR plan, no backups, hope-based resilience |
| 1 | Backup Only | Backups exist, never tested, RTO measured in days |
| 2 | Documented | DR plan exists, manual failover, tested annually |
| 3 | Automated | Scripted failover, quarterly testing, RTO < 1 hour |
| 4 | Active-Active | Multi-region serving, automated failover, RTO < 5 min |
| 5 | Chaos-Verified | DR tested under live chaos, continuous validation |

---

## AI Agent Integration

### Autonomy Level

| Aspect | Level | Description |
|---|---|---|
| Current | L1 | AI generates DR plans and failover runbooks |
| Target | L2 | AI creates DR plans and validates RTO/RPO, human executes failover |

### Harness AI Agent

**Agent**: Harness AI SRE Agent
**Capabilities**:
- DR plan generation
- Failover runbook creation
- RTO/RPO validation
- Backup verification scheduling

### Human Gates

- Failover execution
- Backup restoration
- DR drill approval
- RTO/RPO target changes

---

## Success Criteria
- [ ] Business impact analysis completed for all services
- [ ] RTO/RPO targets defined and documented per service tier
- [ ] DR architecture diagram documented (multi-region for Tier 0)
- [ ] Backup configuration validated with restore test
- [ ] Failover runbook written and accessible to all SREs
- [ ] Tabletop exercise completed this quarter
- [ ] Full region failover tested within last 6 months
- [ ] Backup validation automated (weekly restore test)
- [ ] DR maturity score ≥ 3 for Tier 0 services
- [ ] DR test results feed into resilience scoring (s26)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** [dungnotnull/hybrid-harness-chaos-process-prm](https://github.com/dungnotnull/hybrid-harness-chaos-process-prm)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-dungnotnull-hybrid-harness-chaos-process-prm-s29-disaster-recovery
- Seller: https://agentstack.voostack.com/s/dungnotnull
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.