Sre Canary

SRE Canary — Post-Deploy Monitoring & Canary Verification

Approach every post-deploy window as the engineer who will be paged if something breaks. A deployment is not done when code ships — it is done when you have confirmed that real production traffic is behaving the same as or better than before the change. Every minute of a degraded canary that you do not catch is a minute of user-visible failure that compounds.

The two failure modes are symmetric and equally dangerous: rolling back too early wastes a deploy and erodes team confidence; rolling forward on a broken canary causes real outages. This skill teaches you to read the signals precisely enough to tell them apart.

---

Step 0: Before You Start Monitoring

Before watching any signals, establish your baseline. Monitoring without a baseline is just staring at numbers.

1. Record the pre-deploy steady state — p50/p95/p99 latency, error rate, request rate, and Core Web Vitals from the last 30 minutes before the deploy.
2. Note the deploy timestamp exactly — you will overlay this on every graph. The most common source of confusion in post-deploy review is uncertainty about whether a signal shift happened before or after the change.
3. Identify the blast radius — which endpoints, services, user segments, or geographic regions does this deploy touch? Monitor those first; do not drown in unrelated signal.
4. Know your rollback command before you deploy — not after. The worst time to find your rollback runbook is during an active incident.
5. Define success criteria in advance — "error rate stays below 0.5%, p99 latency stays below 800ms, no new JS exceptions" is a success criterion. "Seems fine" is not.

---

Post-Deploy Monitoring Loop

Run this loop continuously during the canary window. Do not walk away after pressing deploy.

Phase 1: Immediate (0–5 minutes post-deploy)

The first five minutes catch hard failures: startup crashes, broken health checks, misconfigured routing, missing environment variables, and database migration errors.

What to check:

# 1. Confirm new pods/instances are running and healthy
kubectl get pods -n production -l app= --watch

# 2. Check recent deployment events for errors
kubectl describe deployment  -n production | tail -30

# 3. Tail application logs for the new pods only
kubectl logs -n production -l app=,version= --since=5m -f

# 4. Watch the error rate in real time (adapt to your stack)
watch -n 5 'curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~\"5..\",service=\"\"}[1m])" | jq .data.result[0].value[1]'

# 5. Confirm health check is passing on new instances
kubectl exec -n production deploy/ -- curl -sf http://localhost:8080/health

Rollback immediately if:

• Any new pod enters CrashLoopBackOff or Error state
• Health check returns non-200 on new instances
• Error rate jumps above 5× baseline within 2 minutes
• Application logs show unhandled exceptions on startup path

Phase 2: Stabilization (5–15 minutes post-deploy)

The process is running but we are watching for regressions that only appear under real traffic: slow queries exposed by a schema change, memory growth from a leak, latency spikes from a missing cache warm-up.

What to check:

# HTTP error rate by status code — differentiate 4xx (client) from 5xx (server)
rate(http_requests_total{service="",status=~"5.."}[2m])
rate(http_requests_total{service="",status=~"4.."}[2m])

# Latency percentiles — watch p99 specifically; it moves first
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service=""}[2m]))
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service=""}[2m]))
histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{service=""}[2m]))

# Memory usage — a leak will show as monotonic growth here
container_memory_working_set_bytes{container="", namespace="production"}

# CPU — sudden sustained spike on new version is a regression signal
rate(container_cpu_usage_seconds_total{container="", namespace="production"}[2m])

# DB slow queries — a schema change or missing index will surface here
rate(db_query_duration_seconds_bucket{le="1",service=""}[2m])

Alert thresholds — stabilization phase:

| Signal | Warning | Critical | |---|---|---| | 5xx error rate | > 2× baseline | > 5× baseline or > 1% absolute | | p99 latency | > 1.5× baseline | > 2× baseline or > 2s absolute | | p95 latency | > 1.3× baseline | > 1.75× baseline | | Memory growth | > 20% above baseline | > 50% above baseline or OOM risk | | CPU | > 30% above baseline | > 60% above baseline, sustained | | New JS exceptions | Any new error type | Error rate > 1/min |

Phase 3: Confidence Window (15–60 minutes post-deploy)

Traffic has flowed through the new version. You are now watching for issues that only emerge over time: connection pool saturation, cache eviction patterns, slow memory leaks, and long-tail edge cases.

Extended checks:

# Connection pool saturation
db_connection_pool_used{service=""} / db_connection_pool_max{service=""}

# Cache hit rate — a regression here causes latency spikes at the DB
cache_hits_total{service=""} / (cache_hits_total + cache_misses_total)

# Queue depth — if the service processes async work, watch for backup
rabbitmq_queue_messages{queue="-queue"}
# or
aws_sqs_approximate_number_of_messages_visible{QueueName="-queue"}

# Downstream service health — your change may affect dependencies
rate(http_requests_total{client="",status=~"5.."}[5m])

# Apdex score — composite satisfaction metric
(
  rate(http_request_duration_seconds_bucket{le="0.3",service=""}[5m]) +
  rate(http_request_duration_seconds_bucket{le="1.2",service=""}[5m]) / 2
) / rate(http_request_duration_seconds_count{service=""}[5m])

---

Canary Deployment Verification

A canary routes a small percentage of production traffic to the new version while the old version handles the rest. You compare the two cohorts directly — same traffic, same load, same conditions — to detect regressions with statistical confidence before full rollout.

Traffic Split Configuration

Kubernetes with Argo Rollouts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-service
  namespace: production
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: my-service-canary
      stableService: my-service-stable
      trafficRouting:
        nginx:
          stableIngress: my-service-ingress
      steps:
        - setWeight: 5       # Step 1: 5% canary traffic
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 20      # Step 2: 20% canary traffic
        - pause: {duration: 15m}
        - analysis:
            templates:
              - templateName: success-rate
              - templateName: latency-p99
        - setWeight: 50      # Step 3: 50% canary traffic
        - pause: {duration: 20m}
        - analysis:
            templates:
              - templateName: success-rate
              - templateName: latency-p99
              - templateName: error-budget
        # Step 4: 100% — promoted if all analyses pass
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service

AnalysisTemplate — success rate:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.995
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="my-service-canary",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="my-service-canary"}[2m]))

AnalysisTemplate — p99 latency:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-p99
  namespace: production
spec:
  metrics:
    - name: latency-p99
      interval: 1m
      successCondition: result[0]  5× baseline for 2+ consecutive minutes
- [ ] p99 latency > 2× baseline for 3+ consecutive minutes
- [ ] Any canary pod in `CrashLoopBackOff`
- [ ] Health check failure rate > 0

#### Gate 2: 20% → 50% (after 15 minutes at 20%)

Pass conditions (all Gate 1 conditions, plus):
- [ ] Apdex score on canary ≥ Apdex score on stable × 0.98
- [ ] DB query p95 on canary ≤ baseline × 1.15
- [ ] Cache hit rate on canary ≥ baseline × 0.95
- [ ] No alerts firing on canary-specific dashboards
- [ ] Queue depth (if applicable) not growing on canary traffic

Fail conditions (auto-rollback):
- [ ] Apdex degradation > 5% from stable
- [ ] Any memory leak signal (sustained growth > 20% over 10 minutes)
- [ ] Error budget burn rate > 3× normal

#### Gate 3: 50% → 100% (after 20 minutes at 50%)

Pass conditions (all previous, plus):
- [ ] Error budget consumed during canary window  0.5% for 3 min |
| 99.5% | 3.65 hours | ~7.5 minutes | 5xx rate > 1.0% for 3 min |
| 99.0% | 7.3 hours | ~15 minutes | 5xx rate > 2.0% for 5 min |

**Canary-specific alert: error rate comparison**

The most actionable canary alert is not the absolute error rate — it is the ratio of canary error rate to stable error rate. A 0.5% error rate is fine if the stable version also has 0.5%. A 0.5% error rate is a regression if the stable version has 0.05%.

```promql
# Alert when canary error rate is more than 3× stable error rate
(
  rate(http_requests_total{service="my-service-canary",status=~"5.."}[5m])
  /
  rate(http_requests_total{service="my-service-canary"}[5m])
)
/
(
  rate(http_requests_total{service="my-service-stable",status=~"5.."}[5m])
  /
  rate(http_requests_total{service="my-service-stable"}[5m])
) > 3

Latency Thresholds

# Alert when canary p99 latency is 50% worse than stable p99
(
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="my-service-canary"}[5m]))
)
/
(
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="my-service-stable"}[5m]))
) > 1.5

Automatic Rollback Triggers

Define these before deploying. Automation is faster than humans when something breaks at 3am.

Hard rollback triggers (automated, immediate):

• 5xx error rate on canary > 5% absolute for 2 consecutive minutes
• p99 latency on canary > 3× baseline for 3 consecutive minutes
• Any canary pod in CrashLoopBackOff
• Health check failure rate on canary > 0 for 1 minute
• Error budget burn rate > 10× normal (consuming an hour's budget in 6 minutes)

Soft rollback triggers (alert + human decision within 5 minutes):

• 5xx error rate on canary > 2× stable for 5 minutes
• p99 latency on canary > 1.5× stable for 5 minutes
• New JavaScript exception type appearing at rate > 1/minute
• Cache hit rate drops > 10% on canary
• Apdex degradation > 3% vs. stable

# Manual rollback — always test this before you need it
kubectl argo rollouts abort my-service -n production && \
kubectl argo rollouts undo my-service -n production

# Verify traffic is back on stable
kubectl argo rollouts get rollout my-service -n production
kubectl get pods -n production -l app=my-service

# Confirm error rate is recovering
watch -n 10 'kubectl top pods -n production -l app=my-service'

---

SLO/SLA Monitoring During Canary

Error Budget Burn Rate

Burn rate is the rate at which you are consuming your monthly error budget. A burn rate of 1 means you are burning exactly at budget — you will use up the month's budget by end of month. A burn rate of 10 means you will exhaust the budget in 1/10 of the month (about 3 days).

During a canary, burn rate spikes are your highest-signal alert.

# 5-minute burn rate — short window, fast response
(
  1 - (
    rate(http_requests_total{service="my-service",status!~"5.."}[5m])
    /
    rate(http_requests_total{service="my-service"}[5m])
  )
) / (1 - 0.999)  # Replace 0.999 with your SLO target

# 1-hour burn rate — medium window, catches gradual degradation
(
  1 - (
    rate(http_requests_total{service="my-service",status!~"5.."}[1h])
    /
    rate(http_requests_total{service="my-service"}[1h])
  )
) / (1 - 0.999)

Burn rate alert thresholds (Google SRE Workbook recommendations):

| Window | Burn Rate | Action | |---|---|---| | 5 minutes | > 14.4× | Page immediately — critical | | 1 hour | > 14.4× | Page immediately — critical | | 6 hours | > 6× | Ticket + investigate | | 3 days | > 3× | Review and plan |

SLO Dashboard During Canary Window

A canary-window SLO dashboard should show, side-by-side:

1. Error rate: canary vs. stable — the comparison is more informative than the absolute value
2. Latency percentiles: canary vs. stable — p50, p95, p99, all on the same chart
3. Error budget remaining — how much of the monthly budget is left
4. Current burn rate — 5-minute and 1-hour windows
5. Apdex score: canary vs. stable
6. Request rate: canary vs. stable — to verify traffic is actually being split as configured

If your observability platform is Grafana, add a deploy marker annotation:

# Post a Grafana deploy annotation — creates a vertical line on all dashboards
curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
  "${GRAFANA_URL}/api/annotations" \
  -d "{
    \"dashboardId\": ${DASHBOARD_ID},
    \"time\": $(date +%s000),
    \"text\": \"Deploy: ${SERVICE} ${VERSION} — Canary ${CANARY_WEIGHT}%\",
    \"tags\": [\"deploy\", \"canary\", \"${SERVICE}\"]
  }"

---

Core Web Vitals Monitoring

For frontend deployments, Core Web Vitals are the highest-signal user-experience metrics. A canary that degrades CWV is hurting real users even if the API error rate is zero.

Thresholds

| Metric | Good | Needs Improvement | Poor (Rollback) | |---|---|---|---| | LCP (Largest Contentful Paint) | 4.0s | | INP (Interaction to Next Paint) | 500ms | | CLS (Cumulative Layout Shift) | 0.25 | | FCP (First Contentful Paint) | 3.0s | | TTFB (Time to First Byte) | 1800ms |

Real User Monitoring (RUM) During Canary

Collect Core Web Vitals from real users in the canary cohort using the web-vitals library:

// Install: npm install web-vitals
import { onCLS, onINP, onLCP, onFCP, onTTFB } from 'web-vitals';

function sendToAnalytics({ name, value, id, delta, rating }) {
  // Tag with canary version so you can segment in your analytics
  const payload = {
    metric: name,
    value: Math.round(name === 'CLS' ? value * 1000 : value),
    delta: Math.round(name === 'CLS' ? delta * 1000 : delta),
    id,
    rating,                             // 'good' | 'needs-improvement' | 'poor'
    app_version: window.__APP_VERSION__, // injected at build time
    canary: window.__IS_CANARY__,       // true for canary instances
    page: window.location.pathname,
  };

  // Send to your analytics endpoint
  if (navigator.sendBeacon) {
    navigator.sendBeacon('/api/vitals', JSON.stringify(payload));
  } else {
    fetch('/api/vitals', { method: 'POST', body: JSON.stringify(payload), keepalive: true });
  }
}

onCLS(sendToAnalytics);
onINP(sendToAnalytics);
onLCP(sendToAnalytics);
onFCP(sendToAnalytics);
onTTFB(sendToAnalytics);

Segment your CWV data by canary flag in your analytics platform. The rollback signal is not just "LCP got worse" — it is "LCP is worse for the canary cohort and stable for the stable cohort."

Lighthouse CI in the Canary Pipeline

# .github/workflows/canary-cwv-check.yml
name: Core Web Vitals — Canary Check

on:
  workflow_dispatch:
    inputs:
      canary_url:
        description: 'Canary URL to check'
        required: true
      stable_url:
        description: 'Stable URL to compare against'
        required: true

jobs:
  cwv-check:
    runs-on: ubuntu-latest
    steps:
      - uses

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [CODE-SAURABH](https://github.com/CODE-SAURABH)
- **Source:** [CODE-SAURABH/OpenSkills](https://github.com/CODE-SAURABH/OpenSkills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.