Hindcast

hindcast

[](https://github.com/justinstimatze/hindcast/actions/workflows/ci.yml) [](https://goreportcard.com/report/github.com/justinstimatze/hindcast) [](LICENSE)

> "The unforeseen does not exist," replied Phileas Fogg, quietly. > — Jules Verne, Around the World in Eighty Days (1872)

Phileas Fogg, the gentleman who circumnavigates the world on an eighty-day budget without raising his voice, doesn't dispute that things go wrong. He keeps a log, and his log tells him how long things actually take. The unforeseen exists for everyone else.

Claude Code's wall-clock estimates come from training data — human engineers writing "refactor module = 2 days" — which was honest about how long the work used to take, and quietly miscalibrated for an agent that finishes the same work in minutes. The model has no way to know without being told.

hindcast is the log to tell it. Every turn's actual wall-clock and active-compute duration gets recorded. Before each turn runs, a local k-nearest-neighbors predictor matches the current prompt against your past turns and hands Claude a one-line calibrated prior — gated so a thin retrieval can't anchor a confident wrong number.

How the injection avoids anchoring

A naive context-injection (Lou et al. 2024) gets Claude to obey injected numbers 20–60% of the time regardless of whether the number is well-chosen. When retrieval is right, that's calibration. When retrieval is wrong, it's confident error. hindcast's injection is gated to suppress the wrong-retrieval cases:

• Tier gate. Fires only when the prediction comes from a calibrated source (regressor / kNN / task-type bucket / project median). The cross-project "global sketch" tier and the "no data" tier are silent — those are the cases where anchoring would dominate signal.
• Variance gate. If WallP75 / WallP25 > 3, the injection emits the band as the headline ("wall 1m–8m, high uncertainty") in place of a point estimate. A wide interval is honest where a precise-looking number would falsely anchor.
• Citation form. Claude is instructed to cite predictions as ~Xm wall (P25–P75: a–b, source n=N), surfacing provenance so you can spot when it's leaning on a thin sample.
• Override discipline. Claude is told to override when the prompt has a structural reason the predictor cannot see (much larger scope, blocked on long external process), not on perceived complexity — and not to pad the override out of caution.

Disable injection any time with HINDCAST_INJECT=0. Re-enable the v0.1 full-bucket-table injection (no gates) with HINDCAST_LEGACY_INJECT=1.

Install

go install github.com/justinstimatze/hindcast/cmd/hindcast@latest
hindcast install

Two commands. Pure Go, no CGO, no network, ~5 MB binary. install merges three hooks + one MCP server into ~/.claude/settings.json (timestamped backup), initializes a per-install salt, and backfills priors from your ~/.claude/projects/*/ history. Restart Claude Code and the next turn's prompt gets a calibrated prior in Claude's context.

To remove: hindcast uninstall.

Demo

> Refactor the fetcher to use the new Client type.

  [hindcast prior, injected into Claude's context]
    wall ~3m (P25–P75: 1m–6m) · active ~45s
    source: knn sim=0.42 · n=5 · task=refactor

Claude: ~3m wall (P25–P75: 1m–6m, knn n=5).
        I'll trace the fetcher's callers, update signatures, and run the test suite.

  [running…]

— actual: 4 minutes 12 seconds —

Without the prior, Claude's first-shot estimate tends to inflate — typical case is half-hour-to-hour ranges for work that finishes in single-digit minutes. With the prior, Claude cites the predicted number and lands inside the band.

The mechanism (anchoring) is documented in [Lou et al. 2024]; the gates above bound the failure case but do not eliminate it. The size of the calibration lift on actual production turns is reported in the next section.

Empirical lift

A/B against the Claude API on real-session historical prompts, claude-sonnet-4-6, control (stock) vs hindcast-v0.6 inject. Three independent n=30 samples (different seeds):

| seed | control MALR | treatment MALR | lift | 95% CI | |------|--------------|----------------|------|-----------| | 42 | 16.3× | 2.5× | 6.5× | 2.6–17.6 | | 7 | 10.6× | 2.7× | 4.0× | 1.4–8.2 | | 2026 | 17.3× | 1.7× | 9.9× | 6.9–28.0 |

Stock Claude over-estimates by 10–17× median; with hindcast, median error drops to 1.7–2.7×. Lift = 4–10× across seeds (median ≈6.5×). All three CIs are above 1.0 — treatment reduces error with statistical confidence on every sample.

Stock's worst outliers run wild — a 9-second task can get an "18000-second" answer because training-data priors are calibrated to human engineers, not agents. Hindcast clamps the central tendency to ~2× of truth with mild over-estimate bias (1.3–1.6×) — much better than stock, deliberately under-confident relative to maintainer intuition (anchoring caution).

Reproduce with hindcast eval-api -n 30 -seed 42. Requires ANTHROPIC_API_KEY. Costs Delete one project's data. hindcast rotate-salt Regenerate BM25 salt; clears indexes (records kept). hindcast calibrate Legacy online A/B (evaluates deprecated anchoring path). hindcast eval-api [--n 50] Legacy offline A/B via Claude API (anchoring path). hindcast export-seed Dump global sketch as JSON (maintainer only).

## Configuration

hindcast is zero-config on the value path. These env vars exist for edge cases:

| var | default | effect |
|---|---|---|
| `HINDCAST_SKIP` | unset | skip both recording and injection for this session entirely (UserPromptSubmit short-circuits before either) |
| `HINDCAST_CONTROL_PCT` | 10 | percentage of sessions in A/B control arm (only relevant under legacy inject) |
| `HINDCAST_INJECT` | unset (on) | set to `0` to suppress hook output to Claude (predictor still records turns) |
| `HINDCAST_LEGACY_INJECT` | unset | re-enable the v0.1 ungated full-bucket-table injection (mostly for A/B research) |
| `HINDCAST_FRESHNESS_HALFLIFE_DAYS` | `60` | half-life (days) for kNN recency weighting; `0` or negative disables and treats every record as freshness-neutral |

`.hindcast-project` in your project root overrides the default project name (useful for monorepos and symlinked trees).

## Does this work for me?

Two ways to find out without trusting the maintainer's numbers:

- `hindcast verify` — runs a leave-one-out evaluation over your own backfilled records. Tells you what the predictor's MALR is on YOUR data, broken down by source tier. Exit code 0 on PASS, 1 on FAIL.
- `hindcast bench-cross` — runs the same predictor against public agent traces (METR HCAST runs, OpenHands SWE-bench Lite). Tells you whether the architecture generalizes off the maintainer's data.

Honest cross-corpus findings (~6,000 predictions across two public corpora):

- **Architecture (per-project chronological history) generalizes.** Group-median MALR is comparable across corpora and matches what you'd see locally.
- **The BM25 prompt-similarity mechanism is use-case-dependent.** It works well when prompts are evolving multi-turn task work (typical Claude Code use). It works poorly on independent-task corpora like SWE-bench Lite (BM25 sim picks up library-name overlap, not task overlap).
- The single-threshold gate (`sim ≥ 0.5`) tuned on the maintainer's data does NOT generalize to all corpora. v0.3.1 ships `hindcast tune` (auto-runs from the Stop hook when `health.json` is stale): each install computes its own empirical cliff via prefix-LOO and persists it. View with `hindcast show --health`.

If your usage looks like Claude Code (multi-turn project work), expect kNN to earn its keep. If you use Claude Code for one-shot independent tasks, the bucket/project tier may be the ceiling.

v0.5 ships per-user adaptive tier selection. `hindcast tune` (and the Stop-hook auto-refresh) measures ladder/GBDT/linear on a chronological 50/50 held-out split of your own records and writes the winner to `~/.claude/hindcast/health.json`. If a regressor variant beats the existing ladder by ≥15% on your data, `predict.Predict` serves that regressor first; otherwise the kNN→bucket→project→global ladder runs unchanged. View the comparison with `hindcast show --health`.

The regressor is the v0.4 universal-features design (prompt length, task type, recent project velocity, BM25 signals), but with the post-turn `SizeBucket` feature removed in v0.5 (it leaked into training but was always empty at predict time, biasing v0.4 numbers optimistic). Honest cross-corpus finding stands: **linear regressor still wins on unique-instance OpenHands at 1.83× MALR (vs 2.14× group_median, 2.27× kNN)**. On the maintainer's own data, the ladder remains the winner (1.59× vs gbdt 1.72× / linear 1.70× held-out) — adaptive selection correctly stays dormant.

## Known limitations

- **Windows is build-only** — hooks use `syscall.Setsid` and `/tmp` conventions that are Linux/macOS only. CI verifies the binary compiles on Windows; `install` refuses to run there.
- **Anchoring is not eliminated, only bounded.** The variance and tier gates suppress the worst wrong-retrieval cases, but a confidently-wrong kNN match above the active sim floor with a tight band still anchors. `hindcast tune` measures a per-user empirical sim cliff and persists it to `health.json`; v0.6.6+ wires that tuned threshold into `predict.Predict` as the per-user kNN admission floor (defaulting to `knnMinSim = 0.15` when health is unset). When the verdict is "never inject," kNN is suppressed entirely and the prediction falls through to bucket / project / global tiers. The variance gate is the additional per-prediction backstop. If you find Claude consistently parroting bad numbers, set `HINDCAST_INJECT=0` per-shell or open an issue with the bad-prediction example.
- **NFS home directories** break the O_APPEND atomicity guarantee per-project JSONL writes rely on.
- **BM25 stopwords are English-only.** Non-English prompts get less filtering → slightly noisier index, slightly weaker kNN retrieval.
- **BM25 over salt-hashed tokens loses synonym signal.** "fix" and "repair" hash differently. Privacy/retrieval tradeoff is real and bounded.
- **Schema versioning is nominal** — `schema_version: 1` in records, no migration logic yet. Breaking changes in v0.3+ will require `hindcast backfill --rebuild`.
- **Cold start is thin.** Until you have ≥20 turns per project, the predictor leans on overall-project or global fallbacks, which are weak signals. Use hindcast for a week before judging it.

## Why "hindcast"

A meteorology term: running your forecast model against historical observations to calibrate it against reality. Same move here — your own past turn durations calibrate a predictor for the next one. Coolness and exactitude, on the appointed day.

## License

[MIT](LICENSE)

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [justinstimatze](https://github.com/justinstimatze)
- **Source:** [justinstimatze/hindcast](https://github.com/justinstimatze/hindcast)
- **License:** MIT

Install and usage instructions live in the source repository linked above.