# Incident Triage And Pipeline Recovery

> Guides agents through production data incidents. Use when a pipeline fails, publishes bad data, misses an SLA, partially loads, corrupts state, or requires rollback, replay, or stakeholder communication.

- **Type:** Skill
- **Install:** `agentstack add skill-vaquarkhan-data-engineering-agent-skills-incident-triage-and-pipeline-recovery`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vaquarkhan](https://agentstack.voostack.com/s/vaquarkhan)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [vaquarkhan](https://github.com/vaquarkhan)
- **Source:** https://github.com/vaquarkhan/data-engineering-agent-skills/tree/main/skills/incident-triage-and-pipeline-recovery
- **Website:** https://vaquarkhan.github.io/data-engineering-agent-skills/

## Install

```sh
agentstack add skill-vaquarkhan-data-engineering-agent-skills-incident-triage-and-pipeline-recovery
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Incident Triage And Pipeline Recovery

## Overview

Use this skill when production behavior is already wrong and the team needs controlled recovery. It helps agents contain blast radius, diagnose quickly, restore trust, and avoid making the incident worse.

## When to Use

- failed production runs
- bad or partial publishes
- corrupted incremental state
- freshness SLA breaches
- emergency rollback or replay decisions

Do not jump to fixes before stabilizing the system and understanding impact.

## Workflow

1. Contain the incident.
   Decide whether to:
   - pause schedules
   - block downstream publish
   - isolate bad partitions
   - notify owners and consumers

2. Pull live signals before mutation when possible.
   Load `mcp-data-observability-integration` to inspect lag, run state, or Spark stage metrics before reruns or replays.

3. Classify impact.
   Identify:
   - affected datasets
   - time window
   - downstream consumers
   - whether data is late, missing, duplicated, or wrong

4. Determine the safest recovery path.
   Options include:
   - rerun
   - replay
   - rollback
   - partial correction
   - full backfill

   When replay or backfill is chosen, load `safe-backfill-and-replay-orchestration` and draft `templates/backfill-plan.yaml` before execution.

5. Validate recovery before reopening publish paths.

6. Record the incident and add a guardrail.
   A fixed incident with no prevention work is unfinished.

7. Turn high-value incidents into repeatable resilience tests.
   Load `skills/data-resiliency-testing-and-failure-injection/SKILL.md` or `references/data-resiliency-testing-patterns.md` when the same failure mode must be prevented from surprising the team again.

## Common Rationalizations

| Rationalization | Reality |
| --- | --- |
| "Let us rerun everything now." | Blind reruns can duplicate data or destroy evidence. |
| "We can clean up downstream later." | Downstream trust loss is often harder to recover than the pipeline itself. |
| "The root cause can wait." | Without a guardrail, the same incident often returns quickly. |

## Red Flags

- no attempt to contain blast radius
- reruns happen before impact is understood
- downstream consumers are not informed
- recovery succeeds but no new guardrail is added

## Verification

- [ ] Impact scope and affected consumers are identified
- [ ] The chosen recovery path is safer than the alternatives
- [ ] Publish is reopened only after validation
- [ ] Post-incident guardrails or checks are added

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vaquarkhan](https://github.com/vaquarkhan)
- **Source:** [vaquarkhan/data-engineering-agent-skills](https://github.com/vaquarkhan/data-engineering-agent-skills)
- **License:** MIT
- **Homepage:** https://vaquarkhan.github.io/data-engineering-agent-skills/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-vaquarkhan-data-engineering-agent-skills-incident-triage-and-pipeline-recovery
- Seller: https://agentstack.voostack.com/s/vaquarkhan
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.