Isolation Executor

Input Parameters

| Name | Type | Required | Description | |------|------|----------|-------------| | rundir | string | Yes | Workspace root directory for file passing between agents | | sshkeypath | string | No | SSH key path for cloud provider access | | batchplan | string | Yes | Path to analysis/isolationbatchplan.json (batch order, dependencies) | | isolationplans | string | Yes | Path to analysis/isolationplan{resourceid}.json files (glob pattern) | | environment | string | Yes | Environment tier for observation period matrix lookup |

Execution Flow

Task Context

Before starting execution, initialize task_context.json:

{
  "task_id": "",
  "current_step": 0,
  "current_step_id": null,
  "status": "running",
  "steps": {
    "phase_a_pre_isolation_setup": "pending",
    "phase_a_batch_isolation": "pending",
    "phase_b_observation_monitoring": "pending",
    "phase_c_anomaly_attribution": "pending",
    "phase_d_generate_observation_report": "pending",
    "phase_e_observation_batch_summary": "pending"
  },
  "updated_at": ""
}

Update this file after each step completes. On error, set step status to "failed" and overall status to "failed".

Step 1: phaseapreisolationsetup

Type: inline Description: Initialize observation directories and load isolation plans

Execution

Follow these instructions:

Initialize Phase 4 execution environment:

1. Create directory structure:

• {run_dir}/observe/ - for observation reports
• {run_dir}/observe/snapshots/ - for pre-isolation state snapshots
• {run_dir}/observe/logs/ - for execution logs

1. Load and validate inputs:

• Read {rundir}/analysis/isolationbatch_plan.json
• Load all {rundir}/analysis/isolationplan{resourceid}.json files
• Validate schema conformance

1. Determine observation period matrix:

• Read environment: {{ environment }}
• For each resource, determine observationperioddays:
• dev/test: 3 days (extend to 90 if quarterly task detected)
• staging: 7 days (extend to 35 if monthly task detected)
• production: 30 days (extend to 365 if annual task detected)
• Calculate observationend = isolationexecutedat + observationperiod_days

1. Write {rundir}/observe/executionplan.json with:

• batch_count: number of batches
• total_resources: count of all resources
• observationperiods: {resourceid: {start, end, days}}
• executionstartedat: ISO8601 timestamp

This file is required for subsequent steps.

Progress Tracking

After completing this step, update task_context.json:

• Set current_step_id to "phase_a_pre_isolation_setup"
• Set steps.phase_a_pre_isolation_setup to "completed"

Step 2: phaseabatch_isolation

Type: agent Description: Execute isolation for ALL batches sequentially

Input Files

• observe/execution_plan.json (from Step phaseapreisolationsetup, schema: schemas/execution-plan.schema.json)
• analysis/isolation_plan_{resource_id}.json

Execution

Launch an independent agent with the following prompt file:

Dispatch instruction:

IMPORTANT: The batch plan (isolationbatchplan.json) can contain N batches. You MUST iterate over ALL batches 1..N sequentially.

1. Read batchcount from {rundir}/analysis/isolationbatchplan.json

or {rundir}/observe/executionplan.json.

1. For batchnum in 1..batchcount (SEQUENTIAL, one batch at a time):

a. Identify all resources in batch batchnum from isolationbatchplan.json. b. Execute isolation for ALL resources in this batch (can run in PARALLEL within the batch, but must complete before starting next batch). c. Follow the per-resource isolation procedure below for each resource. d. Wait for every resource in this batch to finish (success or failure) before advancing to batchnum + 1.

1. After ALL batches complete, collect all isolationexecuted{resource_id}.json

files and proceed to phasebobservation_monitoring.

Per-resource isolation procedure:

For each resource:

1. Pre-isolation snapshot:

• Capture current state (VM: iptables rules, RDS: SG rules, Redis: config, K8s: replicas, etc.)
• Save to {rundir}/observe/snapshots/{resourceid}preisolation.json
• Include: timestamp, resourcetype, isolationmethod, current_state

1. Execute isolation steps:

• Read isolationplan{resource_id}.json
• Execute each isolation_step in sequence
• For each step: SSH exec / cloud API call / kubectl command (per resource type)
• Log each step: command, exit_code, stdout, stderr, timestamp

1. Verify isolation took effect:

• Re-check state to confirm isolation is active
• For iptables: verify rule exists in iptables -L -n
• For RDS SG: verify inbound rules count = 0
• For Redis: verify old password fails, new password works
• For K8s: verify replicas = 0 or selector matches no pods
• If verification fails: attempt immediate rollback, mark resource as failed

1. Record isolation execution:

• Write {rundir}/observe/isolationexecuted{resourceid}.json with status and verification

Error handling:

• If isolation step fails: log error, attempt rollback, mark as isolation_failed
• If verification fails: attempt rollback, mark as failed
• If rollback fails: P0 escalation, notify user, do NOT proceed to observation
• Partial batch failure: successfully isolated resources proceed to observation
• If an entire batch fails: do NOT skip remaining batches; continue to next batch
• Batch timeout: if a batch takes > 30min, log warning but continue to next batch

Proceed to observation only if isolationexecuted{resourceid}.json has status = "isolated" and verificationpassed = true.

Agent workflow:

1. Read input data from:

• Schema: schemas/execution-plan.schema.json
• Schema: schemas/isolation-plan.schema.json

1. Execute the agent with the prompt

1. Write results to:

• File: observe/isolation_executed_{resource_id}.json

Output

• Schema: schemas/isolation-execution-result.schema.json
• File: observe/isolationexecuted{resource_id}.json

Progress Tracking

After completing this step, update task_context.json:

• Set current_step_id to "phase_a_batch_isolation"
• Set steps.phase_a_batch_isolation to "completed"

Step 3: phasebobservation_monitoring

Type: agent Description: Monitor observation period, collect signals, detect anomalies, trigger rollback

Input Files

• observe/isolation_executed_{resource_id}.json (from Step phaseabatch_isolation, schema: schemas/isolation-execution-result.schema.json)
• analysis/isolation_plan_{resource_id}.json

Execution

Launch an independent agent with the following prompt file:

Dispatch instruction:

For each resource that successfully completed isolation (status = "isolated"), run observation monitoring in parallel.

Observation period: from isolationexecutedat to observation_end (per resource)

For each resource, run observation loop:

1. Initialize observation state:

• observation_id: obs-{uuid}
• observationstart: isolationexecuted_at
• observationend: isolationexecutedat + observationperiod_days
• signals: [] (list of collected signals)
• rollback_triggered: false

1. Monitoring loop (poll every 5 minutes until observation_end):

a. Collect metrics (5min window): Error rate, latency, CPU, memory, disk usage b. Fetch alerts: Query alert system for alerts touching blastradius.affectedservices c. Fetch complaints (poll q1h): Check IM channels and email/tickets d. Check auto-rollback triggers from isolationplan.autorollback_triggers

• If P0/P1: trigger rollback immediately
• If P2: pause observation, mark uncertain

1. P0/P1 Rollback execution (if triggered):

Retry strategy: max 3 attempts with exponential backoff (5s, 10s, 20s)

For each rollback attempt: a. Execute rollbackplan from isolationplan{resourceid}.json b. Verify rollback took effect c. Health check: wait for affected_services to recover (timeout 300s) d. If verification AND health pass: stop rollback, mark as completed e. If attempt 1440 min: score = 0.0 (negligible)

1. Topological Association:

• In blastradius.affectedservices: score = 1.0
• Within 2 hops: score = 0.6
• No match: score = 0.0

1. Directional Association:

• Isolated is upstream, anomaly in downstream: score = 1.0
• Isolated is downstream, anomaly in upstream: score = 0.0
• Peer: score = 0.5

Attribution decision:

• >= 2 dimensions scored > 0: CONFIRMED, high confidence
• 1 dimension scored > 0: UNCERTAIN, low confidence
• 0 dimensions scored > 0: DISMISS, no confidence

Write {rundir}/observe/attribution{resource_id}.json

Write the output to the specified output file.

Output

• Schema: schemas/attribution-result.schema.json
• File: observe/attribution{resourceid}.json

Progress Tracking

After completing this step, update task_context.json:

• Set current_step_id to "phase_c_anomaly_attribution"
• Set steps.phase_c_anomaly_attribution to "completed"

Step 5: phasedgenerateobservationreport

Type: inline Description: Generate final observation report per resource

Input Files

• observe/isolation_executed_{resource_id}.json
• observe/observation_monitoring_{resource_id}.json
• observe/attribution_{resource_id}.json

Execution

Follow these instructions:

For each resource, generate final observation report.

Read:

• {rundir}/observe/isolationexecuted{resourceid}.json
• {rundir}/observe/observationmonitoring{resourceid}.json
• {rundir}/observe/attribution{resource_id}.json (if exists)
• {rundir}/analysis/isolationplan{resourceid}.json

Determine observation_result:

• rollback completed: result = "failed"
• rollback failed: result = "failedrollbackincomplete"
• observation incomplete or has P2 anomaly: result = "uncertain"
• otherwise: result = "passed"

Summarize metrics: alertcount, complaintcount, anomalydates, maxerror_rate.

Write {rundir}/observe/observation{resource_id}.json per schema.

Write the output to the specified output file.

Output

• Schema: schemas/observation-report.schema.json
• File: observe/observation{resourceid}.json

Progress Tracking

After completing this step, update task_context.json:

• Set current_step_id to "phase_d_generate_observation_report"
• Set steps.phase_d_generate_observation_report to "completed"

Step 6: phaseeobservationbatchsummary

Type: inline Description: Aggregate all observation reports into batch summary

Input Files

• observe/observation_{resource_id}.json (from Step phasedgenerateobservationreport, schema: schemas/observation-report.schema.json)

Execution

Follow these instructions:

Aggregate all observation{resourceid}.json files into summary.

Read all {rundir}/observe/observation{resource_id}.json files.

Calculate aggregate metrics:

• total_resources, passed, failed, uncertain counts
• rollback_triggered count
• totalalerts, totalcomplaints

Write {rundir}/observe/observationbatch_summary.json per schema. This summary is required for the decision phase.

Write the output to the specified output file.

Output

• Schema: schemas/observation-batch-summary.schema.json
• File: observe/observationbatchsummary.json

Progress Tracking

After completing this step, update task_context.json:

• Set current_step_id to "phase_e_observation_batch_summary"
• Set steps.phase_e_observation_batch_summary to "completed"

Error Handling

Isolation execution fails for a resource

Log error, skip observation for that resource, mark as isolation_failed. Continue with remaining resources in batch.

Isolation verification fails

Attempt immediate rollback. If rollback succeeds, mark as failed. If rollback fails, P0 escalation, notify user, do NOT proceed.

Observation monitoring encounters metric backend unavailable

Graceful degradation: rely on alert/complaint channels. If critical data missing, mark observation as uncertain.

P0 anomaly detected during observation

Execute rollback immediately (no attribution needed). Mark observation_result = failed.

P1 anomaly detected during observation

Perform 3D attribution. If CONFIRMED, execute rollback. If UNCERTAIN, pause observation, notify user.

P2 anomaly detected during observation

Pause observation clock. Mark observation_result = uncertain. Notify user for manual judgment.

Rollback execution fails (after max 3 retries with backoff)

P0 Escalation protocol:

1. Mark resource: rollbackstatus = "rollbackfailed"
2. Generate incident report with all attempts, timestamps, errors
3. Immediate notifications to on-call engineer
4. Batch pause: STOP all observation loops for remaining resources
5. Save batch state, wait for manual intervention

Observation period expires without anomalies

Mark observation_result = passed. Proceed to deletion phase.

User interrupts observation period

Save current observation state. Can resume from checkpoint. Do not mark as complete until full period elapsed.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: KnoxOps
• Source: KnoxOps/open-devops-skills
• License: Apache-2.0
• Homepage: https://knoxops.app?invite_token=GITHUB26

Install and usage instructions live in the source repository linked above.