Observation Mode

Observation Mode

Shipping is the beginning, not the end. Most of what we build here is judgment-shaped, not contract-shaped: it's a feature, a way of framing a problem, a default we picked — not an if-this-then-that invariant. That kind of work can't be validated by a red/green test (TDD presumes a fixed contract to assert against). It can only be watched. Observation mode is the eval substrate for it: ship the thing, name what would tell us it's misbehaving, watch for that signal over a window, then either graduate it to stable or catch the regression.

Strict TDD still applies where it fits — a real if-this-then-that invariant (a payload format, an idempotency guarantee, a parser edge case) deserves a test, and a test that would catch a class of breakage is cheap insurance. Don't skip a genuinely testable contract. But don't force feature/judgment work into a test shape it doesn't have — that produces brittle tests that assert an arbitrary current shape and break the moment we improve it, bringing the system down for no signal. For that work, observe.

The whole point: this gives you permission to ship more. A change that today would sit as a draft PR "because I have a little concern" can instead ship-into-observation — the concern becomes legible with data, which you only get by shipping and watching. The safety net moves from pre-merge human review (expensive: the user's attention) to post-merge watched signal (cheap: a label + a dated note). Shipping is less of a commitment when it's set up for observation, so the bar to ship drops.

The lifecycle (single-issue identity)

One issue carries the work through its whole life. The phases are GitHub state, so the register is just gh issue list --label observing — no new infrastructure.

1. Build. Implement → review → PR merges with Closes #N. The build phase is genuinely complete, so the issue closes. Good — that close is real.
2. Observe. Immediately reopen #N, add the observing label, and post the observation contract (below). The reopen is the ceremony: the GitHub timeline reads "closed → reopened," which visibly marks the phase change from built to being watched. The issue now lives in the observation register.
3. Graduate. At a later sweep, past the review-by date with no signal fired: close #N for real and label it regression-candidate. It has earned a regression test now — you'd be guarding proven-stable behavior, not speculating. (Writing the guard is optional follow-up work, not a blocker to graduating.)
4. Regress. If a signal fires — often surfaced as a new issue that turns out to be the watched behavior breaking — don't file a fresh bug in isolation. Reopen the observation issue (or keep it open), link the triggering report, and note the regression. The contract already told you what to look at, so diagnosis starts ahead.

Setup: ensure the labels exist (any repo, idempotent)

This skill is cross-project. The first time you use it in a repo — and harmlessly every time after — ensure the two labels exist. gh label create is idempotent-safe when you ignore the "already exists" error, so just run:

gh label create observing --color "0e8a16" \
  --description "Shipped and under active observation — watching a named signal before final close" 2>/dev/null || true
gh label create regression-candidate --color "5319e7" \
  --description "Graduated from observation with no adverse signal — stable enough to seed a regression guard" 2>/dev/null || true

Do this as a reflex in any project that adopts observation mode — including from inside a graveyard-shift (a worker that ships-into-observation should ensure the labels first). The register (gh issue list --label observing) only works if the labels are present, so a project that runs into this consideration should set them up the moment it does, not later. No central registry is needed — the labels live per-repo and this snippet brings any repo into compliance on first contact.

The observation contract

When you ship into observation, the reopened issue gets a comment with exactly these beats. Keep it tight — this is the thing a future sweep (or a future you) reads to decide graduate-vs-regressed.

• Shipped: PR #N, date. One sentence on what changed / what now exists that didn't.
• Watch for: the single most likely way this misbehaves — stated as an observable, not a worry. "Sync silently no-ops on a symlinked dest" not "sync might be flaky."
• Signal source (required): the concrete artifact a future sweep reads to adjudicate this — not prose about symptoms. Three parts, all required:

1. Artifact — the production thing that records the signal: a specific log line/event, a counter/metric, an error class, a dashboard/DB query, or a named non-code channel (a user-report inbox, a follow-up-issue check). For instrumentable changes, this artifact must exist — if the "watch for" needs a log line or counter that production doesn't emit yet, building it is part of this PR (see eligibility gate #2).
2. How to read it — the literal command a future shift runs: the grep, the SQL, the dashboard URL, the gh issue list filter. Copy-pasteable, not "check the logs."
3. Threshold — what reading is healthy vs. regressed: "0 occurrences," "error rate , got , vs threshold ," not a vibe check. Reading at/under threshold past the review-by → **graduate** (close, regression-candidate`); reading over threshold → regression, route into the wave as a fix with its diagnosis already started; still cooking → leave it, and record its age + last reading in the brief ("#N observing 9 days, watching for X, last reading: 0 hits as of 6/30").

• Auto-ship / prep boundary. Items that would have been draft-PR preps but pass the three eligibility gates ship-into-observation instead — merge, then reopen with the contract. This is the loosened ceiling. It does not raise the draft-PR cap; it moves items out of that cap into shipped-and-watched.
• Brief — observation register section. The shift report carries a register block: what graduated, what regressed, and what's still observing with ages. An item observing well past its window is itself a thing to surface ("#N has been observing 40 days — graduate or it's just open").

Anti-patterns

• Observation as a dumping ground. The register is not where uncertain work goes to be forgotten. Every observing issue has a review-by date and gets adjudicated at the next sweep. An issue observing far past its window is a failure, not a holding pattern — graduate it or act on the signal.
• Skipping the contract. Reopening with just the observing label and no "watch for / signal source / review-by" is theater — there's nothing to observe. The contract is the substance; the label is the index.
• Naming a signal with no production source. A "watch for X" with nowhere to read X is the new face of skipping the contract: the contract looks filled in but cannot be evaluated. If the instrumentation that would make X queryable isn't in the diff, you haven't shipped an observation — you've shipped a worry. Build the source or route to prep.
• Forcing a test where observation fits. Don't write a brittle assertion against an arbitrary current shape just to have a green check. If the thing is judgment-shaped, watch it; reserve tests for real invariants.
• Re-filing a regression as a novel bug. If it matches a watched signal, it belongs on the observation issue. A parallel bug thread loses the head start the contract gave you.
• Graduating on a timer alone when the path is barely exercised. "14 quiet days" only means something if the code actually ran. If a watched path hasn't been hit (low traffic, seasonal feature), note that and extend the window rather than graduating on silence that proves nothing.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: rhdeck
• Source: rhdeck/operating-model
• License: MIT

Install and usage instructions live in the source repository linked above.