Synthetic Monitoring

Synthetic monitoring runs scripted tests against production on a schedule, 24/7. It catches outages, performance degradation, and broken flows before real users report them — at 3 AM when traffic is zero, probes are the only thing checking your app works. A login page that always returns 200 but never authenticates passes a naive uptime check; a synthetic probe that asserts the dashboard loads catches it. This skill covers probe design, alerting integration, SLA validation, multi-region execution, and the runbook discipline that keeps a 3 AM page actionable.

Quick Route

| Situation | Go to | |-----------|-------| | Picking a platform (Checkly, Datadog, Grafana, CloudWatch…) | Platform Options | | Writing a probe (login, API health, search) | Probe Design → references/probe-implementations.md | | Probe runs but users still report outages | Failure Modes | | Alerts too noisy or missing real outages | Alerting Integration | | Calculating downtime budget for an SLA | SLA Validation | | Probe failed at 3 AM and on-call is lost | runbook template in references/platforms-and-ci.md |

---

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math.

Critical flows (decides which probes you write):

• What are the 5-10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list.
• Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on-call.
• Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them.
• Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds.

Current monitoring (decides where the gaps are):

• What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap.
• Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep.
• How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix.
• What was the last outage, and how long before it was detected? Sets the detection-time target probes must beat.

Infrastructure (decides regions and CDN assertions):

• Is the app served from multiple regions or one? Multi-region serving needs multi-region probing.
• Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on x-cache/origin headers.
• Do third-party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks.
• What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config.

Test accounts (decides data safety):

• Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics.
• Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number.
• Is there API access for programmatic account setup and data cleanup? Decides whether create-then-delete probes are viable.

---

Core Principles

1. Synthetic tests validate the user experience continuously

RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things.

2. Keep probes simple and fast

A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per-probe wall-clock budget under 30 seconds), focused (one critical path each), and stable (retries: 0, zero flakiness tolerance). The 30-second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel.

3. Alert on trends, not single failures

A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive-failure and multi-region thresholds, or alert fatigue teaches the team to ignore the pager.

4. Probes must not affect production data

Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non-destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error-budget math itself, or probes inflate your own reliability numbers.

5. Assert on the goal, not the status code

A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx.

---

Probe Design

Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product.

| Probe | What It Validates | Frequency | Timeout | |-------|-------------------|-----------|---------| | Homepage load | DNS, CDN, server, basic rendering | 1 min | 10s | | Login flow | Authentication service, session management | 5 min | 15s | | Core workflow | Primary value-delivering action (create document, run report) | 5 min | 20s | | API health | Backend services, database connectivity | 1 min | 5s | | Search | Search index, query processing, result rendering | 5 min | 15s | | Checkout (if applicable) | Payment integration (sandbox mode), cart, order creation | 10 min | 25s | | Third-party integrations | OAuth providers, email delivery, file storage | 10 min | 15s |

Probe implementations

Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and retries: 0. See references/probe-implementations.md for runnable login-flow, API-health, and search probes plus the environment-aware config.

Non-destructive probe patterns

Safe patterns:
  - Read-only operations: GET requests, page loads, searches
  - Sandbox transactions: payment in test mode, email to internal addresses
  - Create-then-delete: create a draft, verify, delete immediately
  - Synthetic flag: operations tagged as synthetic, excluded from processing

Dangerous patterns (avoid):
  - Creating real orders, tickets, or user-facing records
  - Triggering real notifications (email, SMS, push)
  - Modifying shared resources (config, permissions, settings)
  - Operations that cannot be automatically cleaned up

Dedicated test accounts

Account requirements:
  - Clearly identifiable: email contains "synthetic" or "monitor"
  - Flagged in database with an is_synthetic flag (is_synthetic = true)
  - Excluded from analytics, billing, email campaigns, support queues
  - Excluded from RUM and SLO/error-budget pipelines (synthetic traffic is not real traffic)
  - Pre-populated with stable test data that probes can rely on
  - Credentials in a secrets manager (process.env), rotated quarterly
  - Separate accounts per concurrent probe (avoid state conflicts)

---

Platform Options

| Platform | Strengths | When to Use | |----------|-----------|-------------| | Checkly | Playwright-native, code-first, Git integration; Rocky AI agent (GA 2026) now does automated root-cause analysis across Playwright/API/Multistep/TCP/DNS/ICMP checks; CLI access from any AI agent; MCP server | Teams already using Playwright for E2E | | Datadog Synthetic | Deep APM integration, browser and API tests | Teams on the Datadog platform | | Grafana Synthetic Monitoring | Open source, integrates with Grafana dashboards; pairs with k6 2.x; pin a version channel (v1.x/v2.x) for reproducibility | Teams using the Grafana stack | | AWS CloudWatch Synthetics | Blueprints (heartbeat, API, broken-link, visual diff); Python or Node Puppeteer canaries | Teams already on AWS | | New Relic Synthetics | Full-stack observability integration | Teams on the New Relic platform | | Better Stack | Lightweight uptime + status pages + on-call | SMB-friendly, fast setup | | Uptime Kuma | OSS, self-hosted, lightweight | Self-hosting requirement, small surface area | | Custom (Playwright / k6 / Puppeteer + cron) | Full control, no vendor lock-in | Budget-constrained or custom requirements |

> Probes can be authored in Playwright (TS/JS), Puppeteer, k6 (JS — k6 2.0 shipped May 2026 with AI-assisted test authoring and a clearer Assertions API; first-class for synthetic), or Python (CloudWatch Synthetics, Checkly). Pick what your team already maintains. Avoid: Pingdom for new setups — it is a legacy uptime tool; prefer code-first alternatives (Checkly, Grafana, custom Playwright) that version-control probes alongside your app.

Custom and managed implementations

Self-managed: schedule Playwright probes with a GitHub Actions schedule cron (every 5 minutes), inject prod credentials via secrets, and report results to a monitoring webhook. Managed: Checkly is Playwright-native and runs from multiple locations on a fixed frequency. Either way, make probes environment-aware so the same code runs against staging and production with different base URLs and thresholds. See references/platforms-and-ci.md for the GitHub Actions workflow, the Checkly config, the alert-routing rules, and the runbook template.

---

Alerting Integration

Not every probe failure is an incident. Configure rules that cut noise while catching real problems.

Alerting rules:
  - Single failure: log, do not alert (transient network issue)
  - 2 consecutive failures from same region: warning (possible issue)
  - 2 consecutive failures from 2+ regions: alert on-call (confirmed outage)
  - Latency >2x baseline for 10 minutes: warning (performance degradation)
  - Latency >3x baseline for 5 minutes: alert on-call (severe degradation)
  - Any probe timeout: alert if 3 consecutive (service unresponsive)

Routing. Critical failures on revenue paths (login, checkout, api-health) page on-call (PagerDuty + Slack incidents) with a short repeat interval; warnings on secondary probes go to a Slack monitoring channel; info-level events use a long repeat interval. The probe must tag each result with a severity and probe label for these routes to match — see the routing note in references/platforms-and-ci.md for the full alerting-rules.yaml and the tagging step.

Suppress synthetic alerts during planned maintenance. A maintenance window should silence synthetic paging (the probes will fail by design) and exclude that window from error-budget math, or scheduled work burns budget and pages on-call for nothing.

Alert message format — include enough context to start investigating immediately:

Alert template:
  Title: [SYNTHETIC] {probe_name} failing from {region}
  Severity: {critical|warning|info}
  Consecutive failures: {count}
  Last success: {timestamp}
  Error: {error_message}
  Duration: {last_response_time_ms}ms (threshold: {threshold}ms)
  Dashboard: {link_to_dashboard}
  Runbook: {link_to_runbook}
  Regions affected: {list_of_failing_regions}

The {link_to_runbook} points at a per-probe runbook (six lines: what it tests, first checks, manual repro, escalation, dashboard, owner). See the template in references/platforms-and-ci.md.

---

SLA Validation

Availability calculation

Availability = (total_minutes - downtime_minutes) / total_minutes × 100

Where:
  - total_minutes = calendar month in minutes (43,200 for a 30-day month)
  - downtime_minutes = minutes where synthetic probes detected failure

SLA tiers (downtime/month, monthly basis on 43,200 min):
  99.0%  = 432 min   = 7h 12min     (basic web app)
  99.9%  = 43.2 min  = 43min 12s    (business application)
  99.95% = 21.6 min  = 21min 36s    (critical SaaS)
  99.99% = 4.32 min  = 4min 19s     (infrastructure/platform)

> These are common availability targets, not prescriptive tiers. Pick targets from a user-impact analysis, not by tier name. Modern practice (Google SRE Workbook, OpenSLO) favors explicit SLO + error-budget policies over labelled tiers — define what user-visible failure looks like, set the budget user impact tolerates, and let the SLO follow. References: https://sre.google/workbook/ ; https://openslo.com/

Response time percentiles

Track percentiles, not averages — averages hide the worst experiences.

SLA response time targets (example):
  Homepage load:  P50 50% remaining: normal operations, ship features
  25-50% remaining: caution, review recent changes
  <25% remaining: freeze non-critical deploys, focus on reliability
  Budget exhausted: incident mode, every deploy needs extra scrutiny

Exclude synthetic-probe downtime caused by your own maintenance windows from this calculation, or planned work shows as budget burn.

---

Multi-Region Monitoring

Run probes from regions where your users are. A service that works from us-east-1 but is broken from ap-southeast-1 is broken for APAC users.

Region selection strategy:
  - Minimum 3 regions for global services
  - Always include: closest to primary infrastructure, largest user base, farthest from primary
  - Example for US-primary service: us-east-1, eu-west-1, ap-southeast-1
  - Example for EU-primary service: eu-west-1, us-east-1, ap-northeast-1

Track regional latency separately — a global average hides regional degradation.

Regional latency dashboard:
  Region       | P50    | P95    | Status
  us-east-1    | 120ms  | 340ms  | healthy
  eu-west-1    | 280ms  | 620ms  | healthy
  ap-southeast | 450ms  | 1200ms | warning (P95 above threshold)

Alert when:
  - Any region's P95 exceeds its regional threshold
  - Latency difference between regions exceeds 5x (CDN or routing issue)
  - A region that was healthy becomes consistently degraded

CDN validation. Synthetic probes verify CDN caching by checking response headers (x-cache, cf-cache-status) for HIT and confirming the server header matches the expected provider. This catches CDN misconfigurations — and origin failures masked by a stale cache — before users hit slow uncached responses.

---

Anti-Patterns

Complex probes that break often

A probe that navigates 10 pages, fills 5 forms, and asserts on 20 elements is an E2E test, not a synthetic probe. When it breaks, you cannot tell if the app is down or the probe is flaky. Fix: one critical path per probe, under 30 seconds, under 5 assertions. A failure should make it immediately clear what is broken.

Alerting on every single failure

Network blips, DNS hiccups, and transient cloud issues cause occasional failures. Alerting on every one produces noise that teaches the team to ignore alerts. Fix: require 2-3 consecutive failures and failures from 2+ regions before paging. Escalating severity: first failure logs, second warns, third pages.

No test account isolation

Probes sharing one account interfere — one probe changes a setting, another fails because it expected the default. Fix: one dedicated synthetic account per concurrent probe, flagged synthetic, excluded from analytics and billing.

Monitoring only the happy path

Probes that only check "page loads, returns 200" miss broken functionality behind a loading page. A login that returns 200 but never authenticates is not working. Fix: assert on meaningful content

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: petrkindlmann
• Source: petrkindlmann/qa-skills
• License: MIT
• Homepage: https://qa-skills.com

Install and usage instructions live in the source repository linked above.