Disaster Recovery Business Continuity

Disaster Recovery & Business Continuity (s29)

Purpose

Move beyond component-level chaos into organization-level disaster resilience — defining RTO/RPO targets, implementing backup and restore procedures, orchestrating multi-region failover, and validating that the business can survive the worst-case scenario.

---

Prerequisites

• [ ] Service topology and architecture diagrams
• [ ] Resilience scores from s26
• [ ] Compliance audit trail from s30 (if available)
• [ ] Cloud provider access for failover operations
• [ ] Velero or equivalent backup solution configured

Input Contract

| Input | Source | Required | |---|---|---| | Infrastructure topology (regions, AZs, providers) | s01, s05 | Yes | | Critical service inventory with dependencies | s01 (PRD component inventory) | Yes | | Resilience scores (all services) | s26 output | Yes | | Chaos experiment results (identifies single points of failure) | s14-s21 outputs | Yes | | Backup infrastructure details | s05 (infra definitions) | Yes | | RTO/RPO targets (business requirements) | s01 (PRD — NFR constraints) | Yes | | Compliance requirements | s30 (audit/compliance) | No |

Output Contract

| Output | Destination | Format | |---|---|---| | DR plan document | .commandcode/artifacts/dr/dr-plan.md | Markdown | | RTO/RPO matrix per service | .commandcode/artifacts/dr/rto-rpo-matrix.yaml | YAML | | Failover runbook | .commandcode/artifacts/dr/failover-runbook.md | Markdown | | Backup configuration | .commandcode/artifacts/dr/backup-config.yaml | YAML | | DR test schedule | .commandcode/artifacts/dr/dr-test-schedule.yaml | YAML | | DR test results | s26 (resilience score contribution) | JSON | | Recovery scripts | .commandcode/artifacts/dr/scripts/ | Bash/Python | | Business impact analysis | s30 (compliance evidence) | Markdown |

---

DR Planning Framework

ASSESS → DESIGN → IMPLEMENT → TEST → MAINTAIN

Assess:   Business impact analysis, RTO/RPO targets, critical path mapping
Design:   Multi-region architecture, backup strategy, failover mechanism
Implement: Backup schedules, replication, DNS failover, infrastructure automation
Test:     DR drills, tabletop exercises, full region failover
Maintain: Quarterly review, update for new services, validate backups

---

Step 1 — Business Impact Analysis (BIA)

business_impact_analysis:
  services:
    - name: payment-service
      tier: 0  # Revenue-critical — down = lost money
      max_tolerable_downtime_minutes: 5
      rto_target_minutes: 5
      rpo_target_minutes: 1    # 50% of pods in us-east-1 are unhealthy
- [ ] AWS status page reports us-east-1 degradation
- [ ] PagerDuty alert: "multi-AZ failure us-east-1"
- [ ] On-call SRE declares emergency

## Phase 1: Detection (Auto — = 3

# Step 2: Scale up us-west-2 to handle full load
kubectl --context us-west-2 scale deployment/payment-service --replicas=8
kubectl --context us-west-2 scale deployment/checkout-api --replicas=8

# Step 3: Promote PostgreSQL read replica to primary
aws rds promote-read-replica \
  --db-instance-identifier postgres-payment-prod-us-west-2 \
  --region us-west-2

# Step 4: Failover DNS (Route53)
aws route53 update-traffic-policy-instance \
  --id  \
  --ttl 60 \
  --endpoint-type HTTP \
  --evaluate-target-health false

# Step 5: Shift traffic
# Route53 automatically routes to us-west-2 (health-based)
# Verify: dig payment.company.com → should resolve to us-west-2

# Step 6: Notify stakeholders

Phase 4: Verification ( 6 hours old"

exit 1 fi echo "✅ Backup age OK"

2. Test restore (create temporary instance from latest backup)

echo "🔄 Testing restore..." aws rds restore-db-instance-from-db-snapshot \ --db-instance-identifier postgres-payment-restore-test \ --db-snapshot-identifier "$(echo "$LATEST_BACKUP" | jq -r '.DBSnapshotIdentifier')" \ --db-instance-class db.t3.medium \ --region us-east-1

Wait for restore to complete

aws rds wait db-instance-available \ --db-instance-identifier postgres-payment-restore-test \ --region us-east-1

Verify data integrity

ENDPOINT=$(aws rds describe-db-instances \ --db-instance-identifier postgres-payment-restore-test \ --query 'DBInstances[0].Endpoint.Address' \ --region us-east-1 \ --output text)

ROWCOUNT=$(psql -h "$ENDPOINT" -U admin -d payments -tAc \ "SELECT COUNT(*) FROM transactions WHERE createdat > NOW() - INTERVAL '24 hours'")

echo "📊 Recent transaction rows: $ROW_COUNT"

if [ "$ROW_COUNT" -lt 100 ]; then echo "⚠️ Low row count — possible data integrity issue" fi

Cleanup

aws rds delete-db-instance \ --db-instance-identifier postgres-payment-restore-test \ --skip-final-snapshot \ --region us-east-1

echo "✅ Backup validation complete"

---

## DR Maturity Model

| Level | Name | Characteristics |
|---|---|---|
| 0 | None | No DR plan, no backups, hope-based resilience |
| 1 | Backup Only | Backups exist, never tested, RTO measured in days |
| 2 | Documented | DR plan exists, manual failover, tested annually |
| 3 | Automated | Scripted failover, quarterly testing, RTO < 1 hour |
| 4 | Active-Active | Multi-region serving, automated failover, RTO < 5 min |
| 5 | Chaos-Verified | DR tested under live chaos, continuous validation |

---

## AI Agent Integration

### Autonomy Level

| Aspect | Level | Description |
|---|---|---|
| Current | L1 | AI generates DR plans and failover runbooks |
| Target | L2 | AI creates DR plans and validates RTO/RPO, human executes failover |

### Harness AI Agent

**Agent**: Harness AI SRE Agent
**Capabilities**:
- DR plan generation
- Failover runbook creation
- RTO/RPO validation
- Backup verification scheduling

### Human Gates

- Failover execution
- Backup restoration
- DR drill approval
- RTO/RPO target changes

---

## Success Criteria
- [ ] Business impact analysis completed for all services
- [ ] RTO/RPO targets defined and documented per service tier
- [ ] DR architecture diagram documented (multi-region for Tier 0)
- [ ] Backup configuration validated with restore test
- [ ] Failover runbook written and accessible to all SREs
- [ ] Tabletop exercise completed this quarter
- [ ] Full region failover tested within last 6 months
- [ ] Backup validation automated (weekly restore test)
- [ ] DR maturity score ≥ 3 for Tier 0 services
- [ ] DR test results feed into resilience scoring (s26)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [dungnotnull](https://github.com/dungnotnull)
- **Source:** [dungnotnull/hybrid-harness-chaos-process-prm](https://github.com/dungnotnull/hybrid-harness-chaos-process-prm)
- **License:** MIT

Install and usage instructions live in the source repository linked above.