# Synthetic Monitoring > >- - **Type:** Skill - **Install:** `agentstack add skill-petrkindlmann-qa-skills-synthetic-monitoring` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [petrkindlmann](https://agentstack.voostack.com/s/petrkindlmann) - **Installs:** 0 - **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [petrkindlmann](https://github.com/petrkindlmann) - **Source:** https://github.com/petrkindlmann/qa-skills/tree/main/skills/synthetic-monitoring - **Website:** https://qa-skills.com ## Install ```sh agentstack add skill-petrkindlmann-qa-skills-synthetic-monitoring ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About Synthetic monitoring runs scripted tests against production on a schedule, 24/7. It catches outages, performance degradation, and broken flows before real users report them — at 3 AM when traffic is zero, probes are the only thing checking your app works. A login page that always returns 200 but never authenticates passes a naive uptime check; a synthetic probe that asserts the dashboard loads catches it. This skill covers probe design, alerting integration, SLA validation, multi-region execution, and the runbook discipline that keeps a 3 AM page actionable. ## Quick Route | Situation | Go to | |-----------|-------| | Picking a platform (Checkly, Datadog, Grafana, CloudWatch…) | Platform Options | | Writing a probe (login, API health, search) | Probe Design → `references/probe-implementations.md` | | Probe runs but users still report outages | Failure Modes | | Alerts too noisy or missing real outages | Alerting Integration | | Calculating downtime budget for an SLA | SLA Validation | | Probe failed at 3 AM and on-call is lost | runbook template in `references/platforms-and-ci.md` | --- ## Discovery Questions Check `.agents/qa-project-context.md` first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math. **Critical flows** (decides which probes you write): - What are the 5-10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list. - Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on-call. - Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them. - Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds. **Current monitoring** (decides where the gaps are): - What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap. - Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep. - How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix. - What was the last outage, and how long before it was detected? Sets the detection-time target probes must beat. **Infrastructure** (decides regions and CDN assertions): - Is the app served from multiple regions or one? Multi-region serving needs multi-region probing. - Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on `x-cache`/origin headers. - Do third-party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks. - What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config. **Test accounts** (decides data safety): - Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics. - Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number. - Is there API access for programmatic account setup and data cleanup? Decides whether create-then-delete probes are viable. --- ## Core Principles ### 1. Synthetic tests validate the user experience continuously RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things. ### 2. Keep probes simple and fast A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per-probe wall-clock budget under 30 seconds), focused (one critical path each), and stable (`retries: 0`, zero flakiness tolerance). The 30-second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel. ### 3. Alert on trends, not single failures A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive-failure and multi-region thresholds, or alert fatigue teaches the team to ignore the pager. ### 4. Probes must not affect production data Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non-destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error-budget math itself, or probes inflate your own reliability numbers. ### 5. Assert on the goal, not the status code A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx. --- ## Probe Design Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product. | Probe | What It Validates | Frequency | Timeout | |-------|-------------------|-----------|---------| | Homepage load | DNS, CDN, server, basic rendering | 1 min | 10s | | Login flow | Authentication service, session management | 5 min | 15s | | Core workflow | Primary value-delivering action (create document, run report) | 5 min | 20s | | API health | Backend services, database connectivity | 1 min | 5s | | Search | Search index, query processing, result rendering | 5 min | 15s | | Checkout (if applicable) | Payment integration (sandbox mode), cart, order creation | 10 min | 25s | | Third-party integrations | OAuth providers, email delivery, file storage | 10 min | 15s | ### Probe implementations Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and `retries: 0`. See `references/probe-implementations.md` for runnable login-flow, API-health, and search probes plus the environment-aware config. ### Non-destructive probe patterns ``` Safe patterns: - Read-only operations: GET requests, page loads, searches - Sandbox transactions: payment in test mode, email to internal addresses - Create-then-delete: create a draft, verify, delete immediately - Synthetic flag: operations tagged as synthetic, excluded from processing Dangerous patterns (avoid): - Creating real orders, tickets, or user-facing records - Triggering real notifications (email, SMS, push) - Modifying shared resources (config, permissions, settings) - Operations that cannot be automatically cleaned up ``` ### Dedicated test accounts ``` Account requirements: - Clearly identifiable: email contains "synthetic" or "monitor" - Flagged in database with an is_synthetic flag (is_synthetic = true) - Excluded from analytics, billing, email campaigns, support queues - Excluded from RUM and SLO/error-budget pipelines (synthetic traffic is not real traffic) - Pre-populated with stable test data that probes can rely on - Credentials in a secrets manager (process.env), rotated quarterly - Separate accounts per concurrent probe (avoid state conflicts) ``` --- ## Platform Options | Platform | Strengths | When to Use | |----------|-----------|-------------| | Checkly | Playwright-native, code-first, Git integration; **Rocky** AI agent (GA 2026) now does automated root-cause analysis across Playwright/API/Multistep/TCP/DNS/ICMP checks; CLI access from any AI agent; MCP server | Teams already using Playwright for E2E | | Datadog Synthetic | Deep APM integration, browser and API tests | Teams on the Datadog platform | | Grafana Synthetic Monitoring | Open source, integrates with Grafana dashboards; pairs with k6 2.x; pin a version channel (v1.x/v2.x) for reproducibility | Teams using the Grafana stack | | AWS CloudWatch Synthetics | Blueprints (heartbeat, API, broken-link, visual diff); Python or Node Puppeteer canaries | Teams already on AWS | | New Relic Synthetics | Full-stack observability integration | Teams on the New Relic platform | | Better Stack | Lightweight uptime + status pages + on-call | SMB-friendly, fast setup | | Uptime Kuma | OSS, self-hosted, lightweight | Self-hosting requirement, small surface area | | Custom (Playwright / k6 / Puppeteer + cron) | Full control, no vendor lock-in | Budget-constrained or custom requirements | > Probes can be authored in Playwright (TS/JS), Puppeteer, k6 (JS — k6 2.0 shipped May 2026 with AI-assisted test authoring and a clearer Assertions API; first-class for synthetic), or Python (CloudWatch Synthetics, Checkly). Pick what your team already maintains. Avoid: Pingdom for new setups — it is a legacy uptime tool; prefer code-first alternatives (Checkly, Grafana, custom Playwright) that version-control probes alongside your app. ### Custom and managed implementations Self-managed: schedule Playwright probes with a GitHub Actions `schedule` cron (every 5 minutes), inject prod credentials via secrets, and report results to a monitoring webhook. Managed: Checkly is Playwright-native and runs from multiple locations on a fixed frequency. Either way, make probes environment-aware so the same code runs against staging and production with different base URLs and thresholds. See `references/platforms-and-ci.md` for the GitHub Actions workflow, the Checkly config, the alert-routing rules, and the runbook template. --- ## Alerting Integration Not every probe failure is an incident. Configure rules that cut noise while catching real problems. ``` Alerting rules: - Single failure: log, do not alert (transient network issue) - 2 consecutive failures from same region: warning (possible issue) - 2 consecutive failures from 2+ regions: alert on-call (confirmed outage) - Latency >2x baseline for 10 minutes: warning (performance degradation) - Latency >3x baseline for 5 minutes: alert on-call (severe degradation) - Any probe timeout: alert if 3 consecutive (service unresponsive) ``` **Routing.** Critical failures on revenue paths (login, checkout, api-health) page on-call (PagerDuty + Slack incidents) with a short repeat interval; warnings on secondary probes go to a Slack monitoring channel; info-level events use a long repeat interval. The probe must tag each result with a `severity` and `probe` label for these routes to match — see the routing note in `references/platforms-and-ci.md` for the full `alerting-rules.yaml` and the tagging step. **Suppress synthetic alerts during planned maintenance.** A maintenance window should silence synthetic paging (the probes will fail by design) and exclude that window from error-budget math, or scheduled work burns budget and pages on-call for nothing. **Alert message format** — include enough context to start investigating immediately: ``` Alert template: Title: [SYNTHETIC] {probe_name} failing from {region} Severity: {critical|warning|info} Consecutive failures: {count} Last success: {timestamp} Error: {error_message} Duration: {last_response_time_ms}ms (threshold: {threshold}ms) Dashboard: {link_to_dashboard} Runbook: {link_to_runbook} Regions affected: {list_of_failing_regions} ``` The `{link_to_runbook}` points at a per-probe runbook (six lines: what it tests, first checks, manual repro, escalation, dashboard, owner). See the template in `references/platforms-and-ci.md`. --- ## SLA Validation ### Availability calculation ``` Availability = (total_minutes - downtime_minutes) / total_minutes × 100 Where: - total_minutes = calendar month in minutes (43,200 for a 30-day month) - downtime_minutes = minutes where synthetic probes detected failure SLA tiers (downtime/month, monthly basis on 43,200 min): 99.0% = 432 min = 7h 12min (basic web app) 99.9% = 43.2 min = 43min 12s (business application) 99.95% = 21.6 min = 21min 36s (critical SaaS) 99.99% = 4.32 min = 4min 19s (infrastructure/platform) ``` > These are *common* availability targets, not prescriptive tiers. Pick targets from a user-impact analysis, not by tier name. Modern practice (Google SRE Workbook, OpenSLO) favors explicit SLO + error-budget policies over labelled tiers — define what user-visible failure looks like, set the budget user impact tolerates, and let the SLO follow. References: https://sre.google/workbook/ ; https://openslo.com/ ### Response time percentiles Track percentiles, not averages — averages hide the worst experiences. ``` SLA response time targets (example): Homepage load: P50 50% remaining: normal operations, ship features 25-50% remaining: caution, review recent changes <25% remaining: freeze non-critical deploys, focus on reliability Budget exhausted: incident mode, every deploy needs extra scrutiny ``` Exclude synthetic-probe downtime caused by your own maintenance windows from this calculation, or planned work shows as budget burn. --- ## Multi-Region Monitoring Run probes from regions where your users are. A service that works from us-east-1 but is broken from ap-southeast-1 is broken for APAC users. ``` Region selection strategy: - Minimum 3 regions for global services - Always include: closest to primary infrastructure, largest user base, farthest from primary - Example for US-primary service: us-east-1, eu-west-1, ap-southeast-1 - Example for EU-primary service: eu-west-1, us-east-1, ap-northeast-1 ``` Track regional latency separately — a global average hides regional degradation. ``` Regional latency dashboard: Region | P50 | P95 | Status us-east-1 | 120ms | 340ms | healthy eu-west-1 | 280ms | 620ms | healthy ap-southeast | 450ms | 1200ms | warning (P95 above threshold) Alert when: - Any region's P95 exceeds its regional threshold - Latency difference between regions exceeds 5x (CDN or routing issue) - A region that was healthy becomes consistently degraded ``` **CDN validation.** Synthetic probes verify CDN caching by checking response headers (`x-cache`, `cf-cache-status`) for `HIT` and confirming the `server` header matches the expected provider. This catches CDN misconfigurations — and origin failures masked by a stale cache — before users hit slow uncached responses. --- ## Anti-Patterns ### Complex probes that break often A probe that navigates 10 pages, fills 5 forms, and asserts on 20 elements is an E2E test, not a synthetic probe. When it breaks, you cannot tell if the app is down or the probe is flaky. **Fix:** one critical path per probe, under 30 seconds, under 5 assertions. A failure should make it immediately clear what is broken. ### Alerting on every single failure Network blips, DNS hiccups, and transient cloud issues cause occasional failures. Alerting on every one produces noise that teaches the team to ignore alerts. **Fix:** require 2-3 consecutive failures and failures from 2+ regions before paging. Escalating severity: first failure logs, second warns, third pages. ### No test account isolation Probes sharing one account interfere — one probe changes a setting, another fails because it expected the default. **Fix:** one dedicated synthetic account per concurrent probe, flagged synthetic, excluded from analytics and billing. ### Monitoring only the happy path Probes that only check "page loads, returns 200" miss broken functionality behind a loading page. A login that returns 200 but never authenticates is not working. **Fix:** assert on meaningful content … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [petrkindlmann](https://github.com/petrkindlmann) - **Source:** [petrkindlmann/qa-skills](https://github.com/petrkindlmann/qa-skills) - **License:** MIT - **Homepage:** https://qa-skills.com Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-petrkindlmann-qa-skills-synthetic-monitoring - Seller: https://agentstack.voostack.com/s/petrkindlmann - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.