Hatch3r Observability Verify

Observability Verification Gate

Quick Start

This skill defines what "done" means for any feature shipping a service. Run before declaring a feature complete. The 9 gates below mix automated checks (machine-checkable on every PR) with one release-cadence gate (SLO + burn-rate alert review per release). Skipping any gate = the feature is not done. Reviewer approval and passing unit tests alone do not satisfy this bar.

Step 0 — Detect Ambiguity (P8 B1)

Before any work, scan the invocation for unresolved questions in scope, intent, acceptance criteria, target environment, or irreversibility. If any are found, ask the user via the platform-native question tool per agents/shared/user-question-protocol.md. Do not proceed under silent assumption. Default path, not an exception. Triggers for THIS skill: service scope (which routes), trace vendor (OTel collector vs vendor SDK), sample rates (head vs tail), SLO target values, and Gate 7 applicability (LLM-in-path vs pure service).

Fan-out Discipline (P8 B2)

Fan-out scales with task size; token cost never justifies serializing independent work (rules/hatch3r-fan-out-discipline.md P8 B2; agents/shared/efficiency-patterns.md). Emit sub_agents_spawned: { count, rationale } in your output.

Invoked by

This skill is the verification HARNESS — it declares HOW each observability gate is checked. The DISPATCHER that decides WHEN to run it is the CQ specialist agent:

• agents/hatch3r-reliability.md — invokes this skill as the telemetry sub-vector gate of CQ4 (OTel span coverage, structured-log + trace-id correlation, RED/USE metrics, GenAI semconv), alongside skills/hatch3r-reliability-verify for the SLO/probes/runbook sub-vector. The agent contributes the review trigger and Phase-4 dispatch; this skill contributes the 9-gate procedure.

No duplication: the agent decides WHEN, this skill defines HOW. The agent body cites this skill (agents/hatch3r-reliability.md — "cite ... skills/hatch3r-observability-verify as the closing gates"); this subsection is the symmetric upstream citation per rules/hatch3r-agent-orchestration.md (Phase-4 dispatch).

Gate 1: OTel span on request path

• Every HTTP server entry point, every RPC handler, and every queue consumer emits a root span. Every outbound DB / cache / queue / external HTTP call is wrapped in a child span.
• Discovery: enumerate route declarations via grep -E 'app\.(get|post|put|patch|delete)|router\.|@Get|@Post|fastify\.route' src/ and outbound calls via grep -E 'fetch\(|axios|prisma|redis|pg\.query'. Each match must have a tracer call on the same path: grep -E 'tracer|startSpan|@WithSpan' against the file.
• Auto-instrumentation packages (@opentelemetry/auto-instrumentations-node, opentelemetry-instrumentation Python) satisfy the spec when loaded before app imports — verify via process arg --require @opentelemetry/auto-instrumentations-node/register or equivalent loader.
• Pass criteria: >=1 root span per route + >=1 child span per outbound call. 0 routes without instrumentation. Coverage threshold: >=95% of declared routes emit at least one root span under fixture traffic.
• HTTP semconv attributes on every server span: http.request.method, http.route, http.response.status_code, url.scheme. DB spans carry db.system + db.operation.name. Span status ERROR set on every 5xx + every caught exception. Sources: rules/hatch3r-observability-tracing.md, OpenTelemetry semconv v1.41.1 (the HTTP/DB attributes named here are stable from the >=1.29 floor onward).

Gate 2: Structured logs with trace_id injection

• Every log line emitted from request scope is JSON (pino / winston / zap / loguru / slog). No console.log for application logs in production code paths.
• Every request-scoped logger carries trace_id and span_id from the active OTel context. Verify via Playwright or vitest fixture that emits a request and asserts both fields appear on the captured log line.
• Hook the logger to the active span: @opentelemetry/instrumentation-pino for Node, LoggingInstrumentor for Python — auto-injects traceid + spanid. Manual injection acceptable when auto-instrumentation is unavailable for the logger.
• W3C Trace Context (traceparent + tracestate headers) propagated on every outbound HTTP call. Test: send a request, inspect the outbound call recorded by nock / msw / a recording proxy, assert the header is present and parses as a valid traceparent string 00-{32hex}-{16hex}-{2hex}.
• Pass criteria: 0 unstructured app-log statements + 100% of request-scoped log lines carry trace_id + traceparent propagated on every outbound call. Sources: rules/hatch3r-observability-logging.md, W3C Trace Context Level 1 (W3C Recommendation 2020-02).

Gate 3: Severity and message standards

• OTel SeverityNumber mapping documented in the logger initialization. Replace ad-hoc level strings with the OTel-aligned set: TRACE / DEBUG / INFO / WARN / ERROR / FATAL mapped to SeverityNumber 1 / 5 / 9 / 13 / 17 / 21.
• Log messages follow the verb-first structure: action + object + outcome. Example: "created order" {order_id, amount}. Never embed dynamic values into the message string — pass them as fields.
• PII / secret redaction enabled via a centralized redactor — pino redact paths, winston format redactor, or a structured-log middleware. Audit: grep for password / authorization / token / email fields in log payloads; 0 unredacted hits.
• Required envelope fields on every log entry: service.name, service.version, deployment.environment, trace_id, span_id, severity_number, timestamp (RFC 3339 with millisecond precision).
• No console.log for app logs. Enforced via eslint rule no-console with error severity in production code paths; test code is exempt via override. Sources: rules/hatch3r-observability-logging.md, OpenTelemetry Logs Data Model.

Gate 4: RED + USE metrics

• Services emit RED metrics: a Rate counter, an Error counter, and a Duration histogram, each labeled route, method, status. Histogram buckets follow the rule default [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000] ms.
• Resources emit USE metrics: Utilization gauge, Saturation gauge, Errors counter on the resource pool — DB connection pool, worker pool, queue depth, file descriptor count, in-memory cache fill ratio.
• Naming follows {service}.{domain}.{metric}_{unit} in snake_case. Counter names end in _total; histogram names end in the unit (_ms, _bytes).
• Cardinality budget per metric documented in a comment next to the instrument declaration. Cap label cardinality at the value defined in rules/hatch3r-observability-metrics.md ( p95.
• Spans-per-second budget documented per service alongside expected QPS. Budget formula: target_sps = qps * head_sample * (1 + retry_factor). Re-check on every deploy.
• Log sampling for high-volume routes — health checks and static asset routes drop to 1% sample rate via a per-route override at the logger or middleware.
• Cardinality drop rules at the Collector or vendor — drop attributes that exceed the cardinality budget rather than failing ingestion. Example: drop user_id from spans before export when count > 10k unique values per 5-minute window.
• Cost-budget alert wired on monthly telemetry spend with a 80% threshold warning and 100% threshold page.
• Pass criteria: head + tail sampling declared + per-route log sample rule + cardinality drop policy + cost-budget alert. Sources: OpenTelemetry sampling docs, rules/hatch3r-observability-tracing.md.

Gate 9: Alerts-as-code with runbook URL

• Every Prometheus / Datadog / Grafana alert defined in Terraform or YAML committed to the repo. No alerts created via vendor console.
• Every alert rule carries a runbook_url annotation linking to a runbook in docs/runbooks/ or equivalent. Runbook contains: symptoms, likely causes, diagnostic steps, remediation actions, owner team, escalation policy.
• Severity tier set on every alert per the project policy: P1 page on-call within 15 min; P2 page within 1 hour; P3 Slack channel; P4 ticket only. Alerts without a severity tag fail the gate.
• CI check parses alert files and fails when runbook_url is missing or the target runbook file does not exist. Provide a validate-alerts script under scripts/ or rely on promtool check rules for Prometheus.
• Pass criteria: 100% alerts in code + 100% alerts with runbook annotation + 100% alerts with severity tier + target runbook file exists. Sources: Grafana alerting-as-code docs, Datadog Terraform provider, rules/hatch3r-observability-metrics.md.

Verdict

All 9 gates pass = the feature is "done". Anything less = not done.

The orchestrator running this skill emits a single-line verdict per gate (GATE_N: PASS|FAIL ) and aggregates them. One FAIL on a required gate blocks the merge regardless of reviewer approval status.

When this skill runs

• After hatch3r-implementer finishes service code and before hatch3r-qa-validation runs.
• On every PR that touches src/routes/, src/handlers/, src/services/, src/api/, src/middleware/, src/controllers/, src/lib/, or any file matching the four observability rule globs.
• Gate 5 (SLO + burn-rate alert review) executes at release-cut time per release; PR-level execution checks only that the SLO file exists and is non-empty.

Cross-References

• rules/hatch3r-observability-logging.md
• rules/hatch3r-observability-metrics.md
• rules/hatch3r-observability-tracing.md (includes AI agent instrumentation; was previously split as -detail)

References

• OpenTelemetry Semantic Conventions v1.41.1 — opentelemetry.io/docs/specs/semconv/
• OpenTelemetry GenAI Semantic Conventions (Development status as of v1.41.1) — opentelemetry.io/docs/specs/semconv/gen-ai/
• W3C Trace Context Level 1 — www.w3.org/TR/trace-context/
• Google SRE Workbook ch. 5 (SLO + multi-burn-rate alerts) — sre.google/workbook/alerting-on-slos/
• Grafana SLO and alerts-as-code — grafana.com/docs/grafana/latest/alerting/
• Sentry release tracking and source maps — docs.sentry.io/product/releases/
• OpenLLMetry GenAI conventions — github.com/traceloop/openllmetry

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: hatch3r
• Source: hatch3r/hatch3r
• License: MIT
• Homepage: https://docs.hatch3r.com

Install and usage instructions live in the source repository linked above.