# China Incident Mitigation

> Draft step-by-step mitigation CLI commands for a root-caused

- **Type:** Skill
- **Install:** `agentstack add skill-aws-samples-sample-skills-for-aws-devops-agent-china-incident-mitigation`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [aws-samples](https://agentstack.voostack.com/s/aws-samples)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [aws-samples](https://github.com/aws-samples)
- **Source:** https://github.com/aws-samples/sample-skills-for-AWS-Devops-agent/tree/main/devops-agent-cn-management/china-incident-mitigation

## Install

```sh
agentstack add skill-aws-samples-sample-skills-for-aws-devops-agent-china-incident-mitigation
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# China Incident Mitigation

Routing is governed by `china-region-multi-account-routing`.
RCA hand-off is governed by `china-incident-rca`. This skill produces
**command drafts for human approval**, not autonomous actions.

## Intended agent type

Upload with Agent Type **Incident Mitigation** selected.

## The approval contract

This skill **only produces command drafts for human review**. It never
executes write operations autonomously.

Enterprise deployments: route approvals through your change management
system (ServiceNow, Jira, OpsGenie, etc.). The agent outputs the command;
a human approves and executes it in the change window.

Operator/demo deployments: the agent will wait for explicit "approve step N"
before executing. But even then, treat this as a convenience shortcut —
never in production without a change record.

Every mitigation output in this skill uses this exact 4-field format.
No exceptions.

```
### Mitigation step N — 

Command:
  aws  --region  --profile   \
    

What it does:
  

Rollback:
  aws  --region  --profile   \
    

Approval required:
  This command changes production state in .
  Enterprise: submit to change management system before executing.
  Operator: reply "approve step N" to execute, "skip N" to skip,
  or "stop" to abort.
```

The agent **never** executes without explicit approval.
"yes", "do it", "go ahead" are NOT valid — must be "approve step N"
or a documented change record approval.

## Pattern library

Map RCA findings to known mitigation patterns. If the RCA does not match
any pattern, fall back to general investigation (do not improvise
mitigation).

### Pattern A — Credential failure (AuthFailure / ExpiredToken)

Root cause signal: MCP pod logs show `AuthFailure` or `ExpiredToken`.

Steps:
1. Verify credentials in Secrets Manager are current
   ```
   aws secretsmanager get-secret-value \
     --secret-id /mcp/aws-cn --region us-east-1
   ```
2. If stale, rotate via operator — **user provides the new AK/SK**, agent
   does not generate credentials
   ```
   aws secretsmanager put-secret-value \
     --secret-id /mcp/aws-cn --region us-east-1 \
     --secret-string '{"AK":"","SK":""}'
   ```
3. Force pod restart to pick up new secret
   ```
   kubectl -n mcp rollout restart deploy/aws-cn
   ```

### Pattern B — MCP pod crashloop

Root cause signal: pod RESTARTS > 0, recent image change.

Steps:
1. Inspect recent crash
   ```
   kubectl -n mcp logs deploy/aws-cn --previous
   ```
2. If image rollout is the cause, roll back
   ```
   kubectl -n mcp rollout undo deploy/aws-cn
   ```
3. Confirm healthy
   ```
   kubectl -n mcp rollout status deploy/aws-cn --timeout=2m
   ```

### Pattern C — ALB target unhealthy

Root cause signal: ALB 5xx spike, target group health check failing.

Steps:
1. Describe current target health
   ```
   aws elbv2 describe-target-health \
     --target-group-arn  --region cn-northwest-1
   ```
2. If pods are ready but ALB marks them unhealthy, check recent
   `ModifyTargetGroupAttributes` from RCA evidence and revert:
   ```
   aws elbv2 modify-target-group-attributes \
     --target-group-arn  --region cn-northwest-1 \
     --attributes Key=,Value=
   ```
3. Rollback command (if revert itself breaks):
   ```
   aws elbv2 modify-target-group-attributes \
     --target-group-arn  --region cn-northwest-1 \
     --attributes Key=,Value=
   ```

### Pattern D — Overly-permissive SG (security incident)

Root cause signal: SG rule `0.0.0.0/0` on sensitive port added
accidentally.

Steps:
1. Revoke the rule
   ```
   aws ec2 revoke-security-group-ingress \
     --group-id  --region  \
     --protocol tcp --port  --cidr 0.0.0.0/0
   ```
2. Rollback command (if this rule was intentional and you just revoked a
   production-needed rule):
   ```
   aws ec2 authorize-security-group-ingress \
     --group-id  --region  \
     --protocol tcp --port  --cidr 0.0.0.0/0
   ```
3. Verify
   ```
   aws ec2 describe-security-groups \
     --group-ids  --region 
   ```

### Pattern E — CloudFormation stack stuck

Root cause signal: CFN stack in `UPDATE_ROLLBACK_FAILED` or
`UPDATE_IN_PROGRESS` too long.

Steps:
1. Inspect
   ```
   aws cloudformation describe-stack-events \
     --stack-name  --region  --max-items 20
   ```
2. Continue rollback skipping failed resources (surgical):
   ```
   aws cloudformation continue-update-rollback \
     --stack-name  --region  \
     --resources-to-skip 
   ```
3. Rollback option: none — CFN rollback is itself the rollback. Do not
   delete-stack unless the user explicitly authorizes (data loss risk).

### Pattern F — Secrets Manager secret accidentally deleted

Root cause signal: `ResourceNotFoundException` when MCP pod reads secret.

Steps:
1. Check if recoverable (7-day window)
   ```
   aws secretsmanager restore-secret \
     --secret-id /mcp/aws-cn --region us-east-1
   ```
2. If past recovery window, recreate and repopulate (user provides
   AK/SK — agent does not generate)
   ```
   aws secretsmanager create-secret \
     --name /mcp/aws-cn --region us-east-1 \
     --secret-string '{"AK":"","SK":""}'
   ```

## Procedure

1. Read the RCA output
2. Match to a pattern in the library
3. Produce each mitigation step in the 4-field format
4. Wait for explicit "approve step N" before executing
5. After each step, report result and wait for next approval

## Things not to do

- **Do not** execute any command without the exact phrase "approve step N".
  Ambiguous responses ("sure", "yes", "ok") → ask again, do not execute.
- **Do not** generate AWS credentials (AK/SK). The agent cannot see
  customer root-level credential creation. User must provide.
- **Do not** bundle multiple steps into a single approval. Each step is
  approved individually so the user can abort at any point.
- **Do not** improvise a mitigation if the RCA does not match a library
  pattern. Instead, state: "This pattern is not in the mitigation library.
  Recommend manual operator action." Then offer to help draft a one-off
  command with the user's confirmation of each field.
- **Do not** run `aws cloudformation delete-stack` as a mitigation —
  too often used to "fix" stuck stacks when `continue-update-rollback`
  is safer.
- **Do not** use `kubectl delete pod` when `kubectl rollout restart deploy`
  does the same thing with deployment-managed replacement.
- **Do not** silently retry failed commands. If step N fails, report the
  failure verbatim and wait for user decision.

## Examples

**Input**: RCA hypothesis = "Pattern A — Credential failure in aws-cn"

**Output**:
```
### Mitigation step 1 — Verify current credentials

Command:
  aws secretsmanager get-secret-value \
    --secret-id /mcp/aws-cn --region us-east-1

What it does:
  Reads the current AK/SK from Secrets Manager. Read-only.

Rollback:
  None needed (read-only).

Approval:
  This command reads production credentials in aws-cn. Reply
  "approve step 1" to execute, "skip 1" to move on, or "stop" to abort.
```

After approval + execution, the agent reports result and emits step 2.

**Input**: RCA hypothesis = "unknown — signal doesn't fit any pattern"

**Output**:
```
The RCA finding does not match any pattern in the mitigation library
(Patterns A–F). I will not auto-draft a mitigation. Options:

1. Describe the failure in more detail — I may be able to match a
   pattern I missed.
2. Draft a one-off mitigation together, field-by-field, with explicit
   confirmation at each step.
3. Escalate to a human operator.

Which would you like?
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [aws-samples](https://github.com/aws-samples)
- **Source:** [aws-samples/sample-skills-for-AWS-Devops-agent](https://github.com/aws-samples/sample-skills-for-AWS-Devops-agent)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-aws-samples-sample-skills-for-aws-devops-agent-china-incident-mitigation
- Seller: https://agentstack.voostack.com/s/aws-samples
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.