Rollback

Rollback

The fastest way to end an incident is often to undo the change that caused it. But rollbacks have their own failure modes - especially when data has changed.

Decision: Rollback vs Fix Forward

Is the fix obvious and small ( 15 minutes
- Impact is severe and growing
- Multiple systems are affected

### When to Fix Forward
- Root cause is identified and fix is small
- Rollback would cause data issues
- The deployed migration can't be reversed
- The bad code hasn't reached all users yet (canary)

## Code Rollback

### Revert the Deployment
```bash
# Option 1: Redeploy the previous version
# (Best - deploys a known-good artifact)
deploy --version 

# Option 2: Git revert + fast deploy
git revert 
git push
# CI/CD deploys the revert

Verify After Rollback

• [ ] Health checks passing
• [ ] Error rate back to baseline
• [ ] Key user flows working (smoke test)
• [ ] No new error types appearing
• [ ] Latency within normal range
• [ ] Customer-facing issues resolved

Database Rollback

The hard one. Data changes aren't automatically reversible.

Schema Rollback

| Migration Type | Reversible? | How | |---------------|-------------|-----| | Add column (nullable) | Yes | Drop the column | | Add column (NOT NULL with default) | Yes | Drop the column | | Add index | Yes | Drop the index | | Add table | Yes | Drop the table | | Drop column | NO | Data is gone (restore from backup) | | Drop table | NO | Data is gone (restore from backup) | | Rename column | Depends | Rename back (if no code depends on new name yet) | | Change column type | Depends | Change back (if no data loss in conversion) | | Data backfill | Depends | Reverse backfill (if old values are known) |

Rules:

1. Always write down migrations. Test the down migration.
2. Never drop columns in the same deploy as the code change.
3. Separate schema migration from data migration.
4. Before deploying irreversible migrations: take a snapshot/backup.

Data Rollback

When bad code wrote wrong data:

1. STOP THE BLEEDING - disable the code path (feature flag, rollback)
2. ASSESS DAMAGE - what records are affected? Query for anomalies.
3. DETERMINE CORRECT STATE - can you derive correct values from logs, events, or audit trail?
4. FIX DATA - scripted repair (never manual edits)
5. VERIFY - query to confirm all affected records are fixed
6. PREVENT - add validation to prevent bad data from being written again

Config Rollback

# If config is in environment variables:
# Revert the config change in your config management system

# If config is in a config file:
git revert 

# If config is in a feature flag service:
# Toggle the flag off (instant, no deploy needed)

Config rollback should be the fastest of all - no build, no deploy.

Feature Flag Rollback

The safest rollback - no deployment needed.

1. Toggle the flag OFF in your flag management system
2. Effect is immediate for new requests
3. Verify: feature is disabled, baseline behavior restored
4. Investigate at leisure - the incident is over

This is why feature flags exist. If a feature is behind a flag, rollback is a button click.

Multi-Service Rollback

When multiple services were deployed together:

1. Identify which service(s) are actually broken
   (don't roll back everything - that causes more disruption)
2. Rollback in reverse deployment order
   (last deployed → first rolled back)
3. Check cross-service API compatibility at each step
   (old Service A + new Service B - does it work?)
4. Verify inter-service communication after each rollback

The safest approach: Services should be independently deployable. If they're not, that's tech debt to address.

Rollback Checklist

Before Deploying (Prepare)

• [ ] Know the previous good version/SHA
• [ ] Verify rollback mechanism works (have you tested it recently?)
• [ ] Understand what data changes the deploy includes
• [ ] Know if database migrations are reversible
• [ ] Feature flags in place for risky features

During Rollback (Execute)

• [ ] Communicate to team: "Rolling back deployment X"
• [ ] Execute rollback (redeploy previous version)
• [ ] Reverse database migration if applicable
• [ ] Toggle feature flags if applicable
• [ ] Monitor health checks and error rates

After Rollback (Verify)

• [ ] Health checks green
• [ ] Error rate at pre-deploy baseline
• [ ] Key user flows verified
• [ ] Data integrity checked (if data was affected)
• [ ] Communicate: "Rollback complete, service restored"
• [ ] Create post-mortem task

Anti-Patterns

| Anti-Pattern | Problem | Fix | |-------------|---------|-----| | No rollback plan | Panic during incident, slow recovery | Require rollback plan for every deploy | | Rollback untested | Rollback mechanism itself is broken | Test rollbacks regularly (in staging) | | Rolling back everything | Unnecessary disruption | Identify the broken service, roll back only that | | Ignoring data impact | Rollback code but bad data persists | Check for data side effects before rolling back | | Fix forward under pressure | Bad fix causes second incident | Default to rollback when stressed. Fix forward only when root cause is clear. |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pvnarp
• Source: pvnarp/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.