China Incident Mitigation

China Incident Mitigation

Routing is governed by china-region-multi-account-routing. RCA hand-off is governed by china-incident-rca. This skill produces command drafts for human approval, not autonomous actions.

Intended agent type

Upload with Agent Type Incident Mitigation selected.

The approval contract

This skill only produces command drafts for human review. It never executes write operations autonomously.

Enterprise deployments: route approvals through your change management system (ServiceNow, Jira, OpsGenie, etc.). The agent outputs the command; a human approves and executes it in the change window.

Operator/demo deployments: the agent will wait for explicit "approve step N" before executing. But even then, treat this as a convenience shortcut — never in production without a change record.

Every mitigation output in this skill uses this exact 4-field format. No exceptions.

### Mitigation step N — 

Command:
  aws  --region  --profile   \
    

What it does:
  

Rollback:
  aws  --region  --profile   \
    

Approval required:
  This command changes production state in .
  Enterprise: submit to change management system before executing.
  Operator: reply "approve step N" to execute, "skip N" to skip,
  or "stop" to abort.

The agent never executes without explicit approval. "yes", "do it", "go ahead" are NOT valid — must be "approve step N" or a documented change record approval.

Pattern library

Map RCA findings to known mitigation patterns. If the RCA does not match any pattern, fall back to general investigation (do not improvise mitigation).

Pattern A — Credential failure (AuthFailure / ExpiredToken)

Root cause signal: MCP pod logs show AuthFailure or ExpiredToken.

Steps:

1. Verify credentials in Secrets Manager are current

`` aws secretsmanager get-secret-value \ --secret-id /mcp/aws-cn --region us-east-1 ``

1. If stale, rotate via operator — user provides the new AK/SK, agent

does not generate credentials `` aws secretsmanager put-secret-value \ --secret-id /mcp/aws-cn --region us-east-1 \ --secret-string '{"AK":"","SK":""}' ``

1. Force pod restart to pick up new secret

`` kubectl -n mcp rollout restart deploy/aws-cn ``

Pattern B — MCP pod crashloop

Root cause signal: pod RESTARTS > 0, recent image change.

Steps:

1. Inspect recent crash

`` kubectl -n mcp logs deploy/aws-cn --previous ``

1. If image rollout is the cause, roll back

`` kubectl -n mcp rollout undo deploy/aws-cn ``

1. Confirm healthy

`` kubectl -n mcp rollout status deploy/aws-cn --timeout=2m ``

Pattern C — ALB target unhealthy

Root cause signal: ALB 5xx spike, target group health check failing.

Steps:

1. Describe current target health

`` aws elbv2 describe-target-health \ --target-group-arn --region cn-northwest-1 ``

1. If pods are ready but ALB marks them unhealthy, check recent

ModifyTargetGroupAttributes from RCA evidence and revert: `` aws elbv2 modify-target-group-attributes \ --target-group-arn --region cn-northwest-1 \ --attributes Key=,Value= ``

1. Rollback command (if revert itself breaks):

`` aws elbv2 modify-target-group-attributes \ --target-group-arn --region cn-northwest-1 \ --attributes Key=,Value= ``

Pattern D — Overly-permissive SG (security incident)

Root cause signal: SG rule 0.0.0.0/0 on sensitive port added accidentally.

Steps:

1. Revoke the rule

`` aws ec2 revoke-security-group-ingress \ --group-id --region \ --protocol tcp --port --cidr 0.0.0.0/0 ``

1. Rollback command (if this rule was intentional and you just revoked a

production-needed rule): `` aws ec2 authorize-security-group-ingress \ --group-id --region \ --protocol tcp --port --cidr 0.0.0.0/0 ``

1. Verify

`` aws ec2 describe-security-groups \ --group-ids --region ``

Pattern E — CloudFormation stack stuck

Root cause signal: CFN stack in UPDATE_ROLLBACK_FAILED or UPDATE_IN_PROGRESS too long.

Steps:

1. Inspect

`` aws cloudformation describe-stack-events \ --stack-name --region --max-items 20 ``

1. Continue rollback skipping failed resources (surgical):

`` aws cloudformation continue-update-rollback \ --stack-name --region \ --resources-to-skip ``

1. Rollback option: none — CFN rollback is itself the rollback. Do not

delete-stack unless the user explicitly authorizes (data loss risk).

Pattern F — Secrets Manager secret accidentally deleted

Root cause signal: ResourceNotFoundException when MCP pod reads secret.

Steps:

1. Check if recoverable (7-day window)

`` aws secretsmanager restore-secret \ --secret-id /mcp/aws-cn --region us-east-1 ``

1. If past recovery window, recreate and repopulate (user provides

AK/SK — agent does not generate) `` aws secretsmanager create-secret \ --name /mcp/aws-cn --region us-east-1 \ --secret-string '{"AK":"","SK":""}' ``

Procedure

1. Read the RCA output
2. Match to a pattern in the library
3. Produce each mitigation step in the 4-field format
4. Wait for explicit "approve step N" before executing
5. After each step, report result and wait for next approval

Things not to do

• Do not execute any command without the exact phrase "approve step N".

Ambiguous responses ("sure", "yes", "ok") → ask again, do not execute.

• Do not generate AWS credentials (AK/SK). The agent cannot see

customer root-level credential creation. User must provide.

• Do not bundle multiple steps into a single approval. Each step is

approved individually so the user can abort at any point.

• Do not improvise a mitigation if the RCA does not match a library

pattern. Instead, state: "This pattern is not in the mitigation library. Recommend manual operator action." Then offer to help draft a one-off command with the user's confirmation of each field.

• Do not run aws cloudformation delete-stack as a mitigation —

too often used to "fix" stuck stacks when continue-update-rollback is safer.

• Do not use kubectl delete pod when kubectl rollout restart deploy

does the same thing with deployment-managed replacement.

• Do not silently retry failed commands. If step N fails, report the

failure verbatim and wait for user decision.

Examples

Input: RCA hypothesis = "Pattern A — Credential failure in aws-cn"

Output:

### Mitigation step 1 — Verify current credentials

Command:
  aws secretsmanager get-secret-value \
    --secret-id /mcp/aws-cn --region us-east-1

What it does:
  Reads the current AK/SK from Secrets Manager. Read-only.

Rollback:
  None needed (read-only).

Approval:
  This command reads production credentials in aws-cn. Reply
  "approve step 1" to execute, "skip 1" to move on, or "stop" to abort.

After approval + execution, the agent reports result and emits step 2.

Input: RCA hypothesis = "unknown — signal doesn't fit any pattern"

Output:

The RCA finding does not match any pattern in the mitigation library
(Patterns A–F). I will not auto-draft a mitigation. Options:

1. Describe the failure in more detail — I may be able to match a
   pattern I missed.
2. Draft a one-off mitigation together, field-by-field, with explicit
   confirmation at each step.
3. Escalate to a human operator.

Which would you like?

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aws-samples
• Source: aws-samples/sample-skills-for-AWS-Devops-agent
• License: MIT

Install and usage instructions live in the source repository linked above.