Devops Hand Skill

DevOps Expert Knowledge

CI/CD Pipeline Patterns

GitHub Actions Reference

Basic workflow structure:

name: CI/CD Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: make build
      - name: Test
        run: make test
      - name: Lint
        run: make lint

  deploy:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy
        run: make deploy

Useful API endpoints:

# List workflow runs
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/OWNER/REPO/actions/runs?per_page=10"

# Get workflow run details
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/OWNER/REPO/actions/runs/RUN_ID"

# Re-run failed jobs
curl -s -X POST -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/OWNER/REPO/actions/runs/RUN_ID/rerun-failed-jobs"

Pipeline Optimization Checklist

• [ ] Cache dependencies (node_modules, .cargo, pip cache)
• [ ] Parallelize independent jobs
• [ ] Use matrix builds for multi-version testing
• [ ] Skip unnecessary steps on non-code changes
• [ ] Use shallow clones for faster checkout
• [ ] Optimize Docker layer caching
• [ ] Run expensive tests only on main branch

---

Infrastructure Monitoring

Health Check Patterns

HTTP endpoint check:

curl -s -o /dev/null -w "%{http_code} %{time_total}s" --max-time 10 "$URL"

TCP port check:

nc -z -w5 hostname port && echo "UP" || echo "DOWN"

SSL certificate expiry:

echo | openssl s_client -servername HOST -connect HOST:443 2>/dev/null | \
  openssl x509 -noout -dates

DNS resolution:

dig +short hostname

Disk usage:

df -h | grep -v tmpfs

Memory usage:

free -h

The Four Golden Signals

| Signal | What to Measure | Alert Threshold | |--------|----------------|-----------------| | Latency | Request duration | P95 > 500ms | | Traffic | Requests per second | Deviation > 50% from baseline | | Errors | Error rate percentage | > 1% of requests | | Saturation | Resource utilization | CPU/Memory > 80% |

Docker Monitoring Commands

# Container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Resource usage
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# Container logs (last 100 lines)
docker logs --tail 100 CONTAINER_NAME

# Inspect container health
docker inspect --format='{{.State.Health.Status}}' CONTAINER_NAME

Kubernetes Monitoring Commands

# Pod status across all namespaces
kubectl get pods --all-namespaces -o wide

# Resource usage
kubectl top pods --all-namespaces
kubectl top nodes

# Recent events (errors and warnings)
kubectl get events --sort-by=.lastTimestamp --field-selector type!=Normal

# Pod logs
kubectl logs POD_NAME -n NAMESPACE --tail=100

# Describe failing pod
kubectl describe pod POD_NAME -n NAMESPACE

---

Deployment Strategies

Blue-Green Deployment

1. Run current version on "Blue" environment
2. Deploy new version to "Green" environment
3. Run health checks on Green
4. Switch traffic from Blue to Green
5. Keep Blue as rollback target
6. After validation period, decommission Blue

Canary Deployment

1. Deploy new version to small subset (5-10% of traffic)
2. Monitor error rates and latency
3. If healthy, gradually increase traffic (25% -> 50% -> 100%)
4. If problems detected, route all traffic back to old version

Rolling Update

1. Update instances one at a time
2. Wait for health check to pass before updating next
3. If any instance fails health check, pause and alert
4. Continue until all instances updated

Deployment Checklist

• [ ] All tests passing in CI
• [ ] Database migrations compatible (backward and forward)
• [ ] Feature flags configured for new features
• [ ] Monitoring and alerting in place
• [ ] Rollback procedure documented and tested
• [ ] On-call engineer notified
• [ ] Change request approved (if required)

---

Incident Response

Incident Lifecycle

Detection -> Triage -> Mitigation -> Investigation -> Resolution -> Post-mortem

Severity Levels

| Level | Impact | Response Time | Example | |-------|--------|--------------|---------| | SEV1 | Full outage | Immediate | Production down | | SEV2 | Major impact | 15 min | Core feature broken | | SEV3 | Minor impact | 1 hour | Non-critical feature degraded | | SEV4 | Low impact | Next business day | Cosmetic issue |

Incident Response Template

# Incident Report: [Title]
**Severity**: SEV[1-4]
**Status**: [Investigating | Mitigated | Resolved]
**Duration**: [Start time] - [End time]

## Timeline
- HH:MM - [Event or action taken]
- HH:MM - [Event or action taken]

## Root Cause
[What caused the incident]

## Impact
[Who was affected and how]

## Mitigation
[What was done to restore service]

## Resolution
[What was done to fix the root cause]

## Action Items
- [ ] [Preventive measure 1]
- [ ] [Preventive measure 2]

## Lessons Learned
[What we can improve]

---

Infrastructure as Code

Terraform Quick Reference

# Initialize
terraform init

# Plan changes
terraform plan -out=tfplan

# Apply changes
terraform apply tfplan

# Show current state
terraform show

# Destroy resources (DANGEROUS)
terraform destroy

Docker Compose Quick Reference

# Start services
docker compose up -d

# Stop services
docker compose down

# View logs
docker compose logs -f SERVICE_NAME

# Rebuild and restart
docker compose up -d --build SERVICE_NAME

# Scale a service
docker compose up -d --scale SERVICE_NAME=3

---

Common Failure Diagnosis Playbooks

Memory Leak Detection

# Track memory growth over time
while true; do
  ps aux --sort=-%mem | head -5 | awk '{print strftime("%H:%M:%S"), $2, $4"%", $11}'
  sleep 60
done

# Check for OOM kills
dmesg | grep -i "oom\|killed" | tail -20

# Kubernetes memory pressure
kubectl top pods --sort-by=memory | head -10

DNS Failure Cascade

# Test DNS resolution
dig hostname +short
dig @8.8.8.8 hostname +short  # Bypass local DNS

# Check /etc/resolv.conf
cat /etc/resolv.conf

# Test from inside a container
kubectl exec -it POD -- nslookup hostname

Database Connection Pool Exhaustion

# Check active connections (PostgreSQL)
psql -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"
psql -c "SELECT max_conn FROM pg_settings WHERE name = 'max_connections';"

# Check for long-running queries
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
         FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;"

Certificate Expiry Monitoring

# Check cert expiry for a list of domains
for domain in api.example.com app.example.com; do
  expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
    openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  echo "$domain: $expiry"
done

---

Worked Examples

Example 1: Zero-Downtime Deployment Pipeline

Full lifecycle from code merge to production traffic switch with rollback safety.

Phases: CI build and push image -> deploy to Green -> health-gate -> traffic switch -> monitor -> done (or rollback).

Traffic switch script (the critical step):

#!/bin/bash
set -euo pipefail
# Record current slot for rollback, then switch
CURRENT=$(kubectl get svc myapp-active -n production -o jsonpath='{.spec.selector.slot}')
echo "$CURRENT" > /tmp/rollback-slot
kubectl patch svc myapp-active -n production -p '{"spec":{"selector":{"slot":"green"}}}'
# Verify
sleep 5 && curl -sf https://api.example.com/api/health | jq '.version'

Rollback: read /tmp/rollback-slot, patch the service selector back, verify health.

Decision flowchart:

Code merged to main
  |
  v
CI build + test ----[FAIL]----> Block merge, notify author
  |
  [PASS]
  v
Deploy to Green
  |
  v
Health checks ------[FAIL]----> Alert on-call, keep Blue active
  |
  [PASS]
  v
Switch traffic to Green
  |
  v
Monitor 15 min -----[ERROR SPIKE]----> Rollback to Blue, open incident
  |
  [STABLE]
  v
Mark Green as new Blue, done

---

Example 2: Production Incident Response

Walkthrough of a real-world SEV1 incident: API latency spike caused by a database connection pool exhaustion.

Timeline

| Time (UTC) | Event | Actor | |------------|-------|-------| | 14:02 | PagerDuty alert: P95 latency > 2s on /api/orders | Monitoring | | 14:04 | On-call acknowledges, opens incident channel #inc-2025-0312 | On-call engineer | | 14:06 | Check dashboard: request queue depth spiking, error rate at 12% | On-call engineer | | 14:10 | Identify DB connection pool at 100% utilization | On-call engineer | | 14:12 | Find long-running query from analytics job (started 13:55) | On-call engineer | | 14:14 | Kill the runaway query, pool starts draining | On-call engineer | | 14:18 | Latency returns to normal, error rate drops to 0.2% | Monitoring | | 14:20 | Incident mitigated, continue monitoring | On-call engineer | | 14:45 | Root cause confirmed: analytics cron job without query timeout | Investigation | | 15:00 | Incident resolved, post-mortem scheduled | Incident commander |

Detection -- Alerting rules that fired

# Prometheus alerting rule
groups:
  - name: api-latency
    rules:
      - alert: HighAPILatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api"}[5m])) > 2
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "P95 latency above 2s for 2+ minutes"
          runbook: "https://wiki.internal/runbooks/high-latency"

Triage -- Quick diagnosis commands

# 1. Check if it's a specific endpoint or global
curl -s "http://prometheus:9090/api/v1/query?query=topk(5,rate(http_request_duration_seconds_sum[5m])/rate(http_request_duration_seconds_count[5m]))" | jq '.data.result[] | {endpoint: .metric.handler, avg_latency: .value[1]}'

# 2. Check database connection pool
psql -c "SELECT count(*) as total, state FROM pg_stat_activity GROUP BY state;"

# 3. Find the blocking query
psql -c "SELECT pid, now() - query_start AS duration, query
         FROM pg_stat_activity
         WHERE state = 'active' AND now() - query_start > interval '1 minute'
         ORDER BY duration DESC LIMIT 5;"

Mitigation -- Kill the offending query

# Kill the long-running query by PID
psql -c "SELECT pg_terminate_backend(12345);"

# Verify pool is recovering
watch -n 2 'psql -t -c "SELECT count(*) FROM pg_stat_activity WHERE state = '\''active'\'';"'

Prevention -- Fix applied after incident

-- Set statement timeout for analytics role
ALTER ROLE analytics_readonly SET statement_timeout = '300s';

# Add connection pool monitoring alert
- alert: DBConnectionPoolNearCapacity
  expr: pg_stat_activity_count / pg_settings_max_connections > 0.8
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "DB connection pool above 80% capacity"

Post-mortem action items:

• [ ] Add statement_timeout to all non-interactive database roles
• [ ] Add connection pool utilization alerts (threshold: 80%)
• [ ] Move analytics queries to read replica
• [ ] Add circuit breaker to API when pool utilization exceeds 90%

---

Example 3: Infrastructure Scaling Event

Scaling a Kubernetes deployment in response to sustained load increase.

Phase 1 -- Alert triggers

Alert: HighCPUUtilization
  Condition: avg(cpu_usage) > 80% for 10 minutes
  Current:   87% across 3 pods
  Namespace: production
  Deployment: order-service

Phase 2 -- Capacity analysis

kubectl top pods -l app=order-service -n production   # Per-pod CPU/memory
kubectl describe nodes | grep -A 5 "Allocated resources"  # Node headroom
kubectl get hpa order-service -n production            # Current HPA state
# Result: 87% CPU across 3 pods, 842 req/s (2x baseline)

Phase 3 -- Scaling decision matrix

| Metric | Current | Target | Action | |--------|---------|--------|--------| | CPU usage | 87% | Order Service -> DB Query + Payment Service -> Stripe API).

OpenTelemetry propagation: traceparent: 00---, tracestate: vendor=value.

Useful trace queries (Jaeger/Tempo):

curl -s "http://jaeger:16686/api/traces?service=order-service&minDuration=1s&limit=20"  # Slow traces
curl -s "http://jaeger:16686/api/traces?service=order-service&tags=error%3Dtrue&limit=20"  # Error traces

Alerting Best Practices

Avoid alert fatigue -- rules of thumb:

• Every alert must have a runbook link
• Every alert must be actionable (if no one needs to act, it is a log, not an alert)
• Group related alerts to avoid notification storms
• Use inhibition rules: if the cluster is down, suppress per-pod alerts

SLO-based alerting (burn rate):

# SLO: 99.9% availability = 43.2 min/month error budget
# Fast burn (exhausts budget in 2h): error_ratio > 14.4 * 0.001 for 2m -> critical
# Slow burn (exhausts budget in 3d): error_ratio > 3 * 0.001 for 15m -> warning
groups:
  - name: slo-burn-rate
    rules:
      - alert: SLOBurnRateCritical
        expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
        for: 2m
        labels: { severity: critical }
      - alert: SLOBurnRateWarning
        expr: sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > (3 * 0.001)
        for: 15m
        labels: { severity: warning }

Runbook template: Each alert runbook should cover: what the alert means (one sentence), impact scope, diagnosis steps (dashboard + commands), mitigation (quick fix vs proper fix), and escalation path (who to contact after 15 min).

Metrics Collection Patterns

RED Method (request-scoped services):

| Metric | What | PromQL Example | |--------|------|----------------| | Rate | Requests per second | sum(rate(http_requests_total[5m])) | | Errors | Failed requests per second | sum(rate(http_requests_total{code=~"5.."}[5m])) | | Duration | Latency distribution | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) |

USE Method (infrastructure resources):

| Metric | What | Example Check | |--------|------|---------------| | Utilization | % time resource is busy | avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) | | Saturation | Queue depth / backlog | node_load1 / count(node_cpu_seconds_total{mode="idle"}) | | Errors | Error event count | rate(node_disk_io_time_weighted_seconds_total[5m]) |

When to use which:

• RED for services that handle requests (APIs, web servers, message consumers)
• USE for infrastructure (CPU, memory, disk, network interfaces, queues)
• Combine both for a complete picture

---

Security Operations

Secret Management

Principles:

• Never store secrets in source code, environment variables (in Dockerfiles), or container images
• Use a secrets manager (Vault, AWS Secrets Manager, K8s Secrets with encryption at rest)
• Rotate secrets on a schedule and immediately after any suspected compromise
• Audit all secret access

Vault pattern -- inject secrets at runtime:

# Store a secret
vault kv put secret/myapp/db \
  username="app_user" \
  password="$(openssl rand -base64 32)"

# Read a secret (application startup)
vault kv get -format=json secret/myapp/db | jq -r '.data.data.password'

# Enable audit logging
vault audit enable file file_path=/var/log/vault-audit.log

Kubernetes secrets -- from Vault using sidecar injector:

Annotate the pod template w

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: librefang
• Source: librefang/librefang-registry
• License: MIT

Install and usage instructions live in the source repository linked above.