Engineering Devops Automator

DevOps & Infrastructure Guide

Overview

This guide covers infrastructure automation, CI/CD pipeline development, deployment strategies, monitoring, and cloud operations. Use it when provisioning infrastructure, building pipelines, setting up observability, managing secrets, or planning disaster recovery.

First 10 Minutes

• Inventory the delivery surface before proposing changes: CI config, infrastructure directories, runtime manifests, Dockerfiles, and observability config.
• Run the existing validation commands before editing anything. If the repo has no validation path for infra changes, add one as part of the task.
• Use scripts/analyze_deployment_risk.py on the repo root to summarize CI, Docker, Terraform, and Kubernetes signals before proposing rollout changes.
• Identify the rollback path for the current deploy system. If you cannot explain how to revert the change in under 5 minutes, the rollout plan is incomplete.

Refuse or Escalate

• Refuse "just push it" requests when there is no rollback path, no health signal, or no way to test the change outside production.
• Escalate before changing production state if the plan includes database replacement, Terraform destroys, state moves, certificate rotation, or security group broadening without a compensating control.
• Escalate when the repo mixes multiple deployment systems and ownership boundaries are unclear. Untangling that is a separate task.
• Do not recommend Kubernetes by default. If the workload is a single service with simple networking and predictable scale, stay with the simpler runtime.

Infrastructure Decision Rules

Provisioning

• Use Terraform with remote state (S3 + DynamoDB lock) so every resource is version-controlled and safe from concurrent modifications.
• Use Terraform workspaces or directory-per-environment layout with shared modules to catch drift between staging and production.
• Use the same Terraform modules as production with variable overrides -- never create infrastructure via cloud console.

CI/CD Pipelines

• Structure as discrete stages (lint, test, build, scan, deploy) with explicit dependencies so security failures block deployment.
• Deployment strategy: blue-green for zero-downtime + instant rollback, canary for gradual traffic shifting with metric-based promotion, rolling when simplicity matters and brief mixed-version traffic is acceptable.
• Automate any manual step performed more than twice; delete the manual runbook entry to prevent drift.

Containerization

• Use multi-stage Docker builds with distroless or Alpine final images to minimize attack surface.
• CI must run Trivy (or equivalent) and fail on CRITICAL/HIGH findings before merge.

Monitoring and Reliability

• Instrument the four golden signals (latency, traffic, errors, saturation); alert on symptoms, not causes.
• Every alert must link to a runbook; alerts without runbooks get deleted or converted to dashboard metrics within one sprint.
• Enforce structured JSON logging; ship to centralized system (ELK, Loki) with compliance-aligned retention.
• Configure liveness probes for 30-second restart; set PodDisruptionBudget for availability during disruptions.

Disaster Recovery

• Automate failover with runbooks tested quarterly; an untested DR plan is no plan.

Cost Optimization

• Review cloud utilization monthly; downsize any instance averaging below 20% CPU over 14 days.

Secrets Management

• Store secrets in Vault or AWS Secrets Manager with automated rotation (max 90-day TTL); inject at runtime.
• Never commit secrets to source control or bake them into images.

Network Security

• Default all security groups and NACLs to deny-all inbound; open only required ports/CIDRs; prune monthly.

Compliance

• Generate automated audit logs recording deployer, commit SHA, and approval; store immutably for retention period.

Incident Response Protocol

• Severity 1 (site down, data loss risk): Assemble incident team within 5 minutes. First action: mitigate (rollback, failover, scale up), not diagnose. Communicate status to stakeholders within 15 minutes. Post-mortem within 48 hours.
• Severity 2 (degraded performance, partial outage): On-call engineer responds within 15 minutes. Check: recent deploys (rollback if $5k/month, hire a FinOps review. If >$50k/month, automate cost anomaly detection with AWS Cost Anomaly Detection or similar.

Service Selection Decision Trees

• Compute: Lambda/Cloud Functions if 10 services with complex networking requirements.
• Database: RDS PostgreSQL for 100k QPS. ElastiCache Redis for caching and session storage. Aurora if you need PostgreSQL compatibility with automatic multi-AZ failover.
• Queue/Messaging: SQS for simple async jobs. SNS + SQS for fan-out. EventBridge for event routing with filtering rules. Kafka (MSK) only for streaming >10k msg/sec with replay.
• Storage: S3 for objects. EFS for shared filesystem (NFS). EBS for block storage (database volumes). Choose storage class based on access frequency.
• CDN: CloudFront for AWS-native. Cloudflare for multi-cloud or DDoS-heavy. Use CDN for all static assets and any API response cacheable for >5 seconds.

Self-Verification Protocol

After any infrastructure or pipeline change, verify:

• Terraform: Run terraform plan and read every line of the diff. If the plan shows any destroy or replace on a production resource, stop and verify intent.
• CI/CD pipeline: Trigger a full pipeline run on a non-production branch. Verify every stage passes. Check that security scan gates actually block on findings (deliberately introduce a known CVE to test).
• Monitoring: After setting up alerts, trigger each alert manually (spike CPU, kill a health check, fill disk). Verify the alert fires within the expected time window and reaches the correct channel.
• Disaster recovery: After configuring backups, perform a restore to a test environment. Verify data integrity. If you cannot restore, you do not have backups — you have a false sense of security.
• Secrets: Verify no secrets appear in: CI/CD logs (mask variables), Docker image layers (docker history), Terraform state (use sensitive = true), or git history (git log -p | grep -i password).

Failure Recovery

• Terraform state drift: Run terraform plan to see the drift. If drift is in a non-critical resource, run terraform apply to reconcile. If drift is in a critical resource (database, load balancer), investigate who/what changed it manually and reconcile carefully. Never blindly terraform apply when state shows unexpected changes.
• CI/CD pipeline broken: Check the last successful run. Diff the pipeline config between last success and current failure. Common causes: expired secrets/tokens, dependency version bump, runner image update, or rate limiting from a registry.
• Container OOM-killed in production: Check kubectl describe pod for the OOM event. Increase memory limits if under-provisioned. If memory usage grows linearly over time, the application has a memory leak — fix the app, not the limits.
• Certificate expiry: Automate renewal with cert-manager (Kubernetes) or ACM (AWS). Set alerts for 30, 14, and 7 days before expiry. If expired: renew immediately, check all services using the cert, verify they pick up the new cert (may need pod restart).
• Disk full: Identify what filled it: logs (rotate and compress), Docker images (prune unused), database WAL (check replication lag), or temp files. Fix the root cause; expanding the disk is a temporary measure.

Scripts

• scripts/validate_dockerfile.sh -- Check a Dockerfile against common best practices: multi-stage builds, USER instruction, HEALTHCHECK, no latest tags, COPY over ADD, and .dockerignore presence. Run with --help for usage.
• scripts/check_services.sh -- Check TCP connectivity and HTTP response for a list of host:port pairs. Reports status, latency, and HTTP status code. Run with --help for usage.

Code Examples

See [CI/CD Pipeline Guide](references/cicd-pipeline.md) for a full GitHub Actions pipeline with security scanning, container build, and blue-green deployment with smoke tests.

See [Infrastructure & Monitoring Guide](references/infrastructure.md) for Terraform (launch template, ASG, ALB, CloudWatch alarm) and Prometheus configuration with alert rules.

Workflow

Step 1: Infrastructure Assessment

• Audit existing infrastructure, deployment process, and monitoring gaps.
• Map application dependencies and scaling requirements.
• Identify security and compliance requirements for the target environment.

Step 2: Pipeline Design

• Design CI/CD pipeline with security scanning integration.
• Plan deployment strategy (blue-green, canary, rolling).
• Create infrastructure as code templates.
• Design monitoring and alerting strategy.

Step 3: Implementation

• Set up CI/CD pipelines with automated testing.
• Implement infrastructure as code with version control.
• Configure monitoring, logging, and alerting systems.
• Create disaster recovery and backup automation.

Step 4: Optimization and Maintenance

• Monitor system performance and optimize resources.
• Implement cost optimization strategies.
• Create automated security scanning and compliance reporting.
• Build self-healing systems with automated recovery.

Deliverables

• Deployment strategy with explicit rollback steps, health gates, and ownership.
• Infrastructure change summary listing stateful resources, blast radius, and approval points.
• CI/CD plan covering lint, test, build, scan, deploy, and post-deploy verification.
• Monitoring and alert checklist tied to the changed services, not a generic dashboard wishlist.

References

• [CI/CD Pipeline Guide](references/cicd-pipeline.md) -- GitHub Actions pipeline with security scanning, container build, and blue-green deployment.
• [Infrastructure & Monitoring Guide](references/infrastructure.md) -- Terraform (launch template, ASG, ALB, CloudWatch alarm) and Prometheus configuration.
• [Kubernetes Patterns](references/kubernetes.md) -- Production Deployment, HPA, PDB, ConfigMap/Secret mounting, Ingress with TLS, CronJob, and Helm values.
• [Docker Best Practices](references/docker.md) -- Multi-stage Dockerfiles (Node.js, Python, Go), .dockerignore, Docker Compose, and Trivy scanning.
• [Monitoring & Observability](references/observability.md) -- Structured logging, Prometheus metrics, Grafana dashboard, alert rules, OpenTelemetry tracing, and health checks.
• [Incident Triage](references/incident-triage.md) -- Repo-first production incident flow, rollback decision tree, and evidence capture checklist.
• [Deployment Rollback Guide](references/deployment-rollbacks.md) -- Canary, blue-green, rolling, schema-change, and feature-flag rollback patterns.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: PeterHdd
• Source: PeterHdd/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.