Quellgeist

Quellgeist

[](https://github.com/Rajeev-Shyam/Quellgeist/actions/workflows/ci.yml) [](https://github.com/Rajeev-Shyam/Quellgeist/actions/workflows/security.yml) [](docs/case-studies/wave4-qwen-finetune.md) [](LICENSE) [](https://www.python.org/downloads/)

> First-line incident triage you can trust: ranked root-cause hypotheses where every claim cites a real evidence handle — and the agent abstains rather than guess.

Quellgeist is a model-agnostic AI agent for first-line production-incident triage. It runs a legible JSON-action ReAct loop over read-only tools (structured logs + recent deploys + metric time-series), then emits a structured Diagnosis: confidence-ranked root-cause hypotheses, each backed by a structured evidence handle (LogRef.id / CommitRef.sha / MetricRef.id) the agent actually saw — never free text. Two ideas set it apart:

• Cite-by-structured-handle. Evidence is a checkable handle, not a sentence,

so a fabricated citation is measurable and deterministically rejected by a keyless fabrication check — not a matter of fuzzy string-matching.

• Abstain-over-hallucinate. A confidently-stated wrong cause is the worst

possible answer, so "insufficient evidence" is a first-class outcome.

> Status: Wave 4 complete — the fine-tune works. The DR-0020 QLoRA fine-tune > of the local reasoner (Qwen3-4B, served via Ollama) took it from the base's > 0/16 holdout to 12/16 — zero fabrication, zero speculative-filtering, and > cheaper than the base — while beating a 31B frontier (Gemma-4-31B, 10/16) > on the same holdout at $0, fully offline. Non-memorisation is triangulated > three ways (fixtures ≈ holdout; core-fresh ≥ core-overlap; structure probe 7/10). > Two honest limits: the resource_exhaustion class didn't transfer (0/N; the > frontier passes it), and adversarial-abstention recall is 6/12 at the system > level — a ceiling the 31B frontier shares (also 6/12), not a fine-tune > regression. When this agent misses it's incomplete or too cautious, never > confidently fabricating. See > [Status & roadmap](#status--roadmap) · [fine-tune case study](docs/case-studies/wave4-qwen-finetune.md).

Why it's different

| | | |---|---| | Evidence is a handle | Each hypothesis cites a log row's source-stable id or a commit sha, copied verbatim from a tool result — the unit the deterministic fabrication check looks up. Prose lives in a display-only note. (DR-0009) | | Abstention is a feature | When signals are weak the agent returns abstained=true with a reason and an empty hypotheses list — enforced by the schema. | | Model-agnostic by construction | The loop parses JSON actions from plain chat text, so it's identical on Gemini's free tier and a local 4-bit Qwen — no dependence on any backend's native function-calling. Swap models with one config change. (DR-0008, DR-0010) | | Reliability is gated, not asserted | A keyless, deterministic CI gate (ruff + black + pytest, including the fixture-backed eval harness) runs on every push. |

What it is / what it's NOT

• It is: a first-line triage agent — ranked, evidence-cited root-cause

hypotheses (or an honest abstention) from read-only logs/deploys/metrics, over a model-agnostic loop that runs on a hosted frontier model or a local 4B.

• It is NOT: an autonomous remediator (it never mutates prod — resolution

verification is a deferred, cut-first wave); a production-hardened service (the demo is a deliberate toy); or a general-purpose agent. The holdout it's measured on is out-of-vocabulary but in-structure — not a claim about unseen incident shapes or real production data.

Quickstart (~30 seconds to a broken service + structured logs)

Requires uv and Python 3.12+.

See a real-shaped diagnosis in one keyless command (no model, no API key):

uv sync && uv run quellgeist diagnose --demo     # renders the demo incident's cited postmortem

Then run the full loop against the live toy service:

uv run uvicorn demo.app.main:app          # 1. start the toy service (leave running)

# --- in a second shell, from the repo root ---
uv run python -m demo.chaos.bad_deploy    # 2. inject a simulated bad deploy
curl -s localhost:8000/login              # 3. trip /login -> 500s + structured error logs
uv run quellgeist diagnose --show-trace   # 4. diagnose live (needs a model; see below)

uv run python -m demo.chaos.reset         # back to a green slate

The live step needs a reasoner — see [Running the model](#running-the-model). Without a key, quellgeist diagnose exits 1 with a one-line error + hint (never a traceback); --demo always works keyless and renders the same output shape deterministically from gold.

Architecture

A custom, legible loop is the orchestration layer; the three read-only tools are the evidence interface; the Diagnosis schema is the contract that the postmortem renderer and the eval judge both read.

flowchart TD
    trigger(["incident trigger - CLI"]) --> loop
    model["reasoner via LiteLLM(Gemini or local Qwen, swappable)"] -. "chat completion" .-> loop

    subgraph loopbox["model-agnostic JSON-action ReAct loop"]
      loop["run_loop()decide, call tool, observe, repeat"]
    end

    loop -- "query_logs" --> logs["logs toolstructured JSONL, stable ids"]
    loop -- "get_recent_commits" --> commits["commits tooldeploy_log.json, shas"]
    loop -- "query_metrics" --> metrics["metrics tooltime-series, named series"]
    logs -- "rows + ids" --> loop
    commits -- "commits + shas" --> loop
    metrics -- "series + names" --> loop

    loop --> diag["Diagnosis (schema.py)ranked hypotheses citingLogRef.id / CommitRef.sha / MetricRef.id, or abstains"]
    diag --> pm["postmortem rendererdeterministic Markdown"]
    diag --> judge["eval judgefixture scenarios, CI gate"]

All three tools are also exposed as MCP servers over stdio (python -m quellgeist.servers.logs_mcp, …commits_mcp, …metrics_mcp). The agent currently reuses the same tool functions in-process behind a ToolSpec registry; a stdio MCP-client path (the agent driving the servers over the wire) is on the roadmap (DR-0010).

> Deep dive: [docs/architecture.md](docs/architecture.md) walks the full > pipeline (loop → tools → verifier → postmortem), a sequence diagram, the module > map, and the cross-cutting design decisions.

The servers publish to the Official MCP Registry on each tagged release (see [docs/publishing.md](docs/publishing.md)); once published each is runnable with uvx --from quellgeist quellgeist-logs-mcp (or …-commits-mcp / …-metrics-mcp).

Example session

Inject the bad deploy — it drops a marker that flips verify_token into a NoneType regression and writes a deploy_log.json whose offending commit landed just before the errors (illustrative stdout — the timestamp reflects when you run it; paths shown relative to the repo root):

$ uv run python -m demo.chaos.bad_deploy
injected bad deploy a1b2c3d (touched demo/app/auth.py) at 2026-06-24T12:22:43Z
  marker:     demo/.bad_deploy
  deploy log: demo/deploy_log.json
next: hit /login to generate the 500s, then `quellgeist diagnose`

With a reasoner configured, quellgeist diagnose reads the logs + deploys and emits a postmortem. The CI environment has no validated model key (DR-0012), so the diagnosis below is rendered from gold — built deterministically from the fixture's labelled cause and evidence handles via render_postmortem, not live model output:

# Incident Postmortem (rendered from gold)

## Root-cause hypotheses

### 1. Bad deploy a1b2c3d (10:01:50Z) refactored auth.py and introduced a NoneType error in verify_token; /login 500s begin ~20s later at 10:02:12Z.  (confidence: 1.00)

Evidence:
- log #2
- commit a1b2c3d

Reproduce that render yourself (no model needed):

uv run python -  **Base vs tuned — important.** The `ollama pull` above is the **base** Qwen3-4B:
> the honest safe *floor* — it scores **0/16 on the holdout and abstains on
> everything**, never fabricating (DR-0019). The **12/16** headline is the
> **DR-0020 fine-tune** (`quellgeist-qwen3-dr0020`), which you build + serve via
> [`finetune/README.md`](finetune/README.md) (a free-Colab QLoRA run →
> `ollama create`). Until that tuned GGUF is published for a one-line pull, the
> base model is what a plain `ollama pull` gives you — safe, not yet useful. Use a
> hosted model (above) or the fine-tune to see live diagnoses.

Heads-up (DR-0012): a Gemini key on an unvalidated, no-billing project returns
`429 limit: 0` on current models, so the shipped CI gate is deliberately
**keyless** and model-driven evals are key-gated and run **out-of-band**
(DR-0015). At home the intended default reasoner is a local **Qwen3-4B** via
Ollama (DR-0008).

### Running the eval (reasoner + verifier + LLM-judge)

The fixture eval scores the reasoner with a deterministic keyword judge + a
zero-fabrication check (the keyless gate), and can additionally run two model
layers (DR-0016): a **verifier** that confirms cited evidence supports each
hypothesis (forcing abstention otherwise) and an advisory **LLM-judge** rubric.

```bash
export GEMINI_API_KEY="…"
export QG_MODEL="gemini/gemini-3.5-flash"
QG_VERIFY=1 QG_JUDGE_LLM=1 \
QG_MIN_CALL_INTERVAL_S=6 \      # pace calls under the free-tier RPM (avoids 429 bursts)
  uv run python -m evals.run_evals

QG_VERIFIER_MODEL / QG_JUDGE_MODEL override the model per layer (default QG_MODEL). An unreachable backend (quota/503/timeout) or a rejected credential (missing/invalid/stale key) is reported as a skip, not a failure (DR-0015/DR-0017), so the out-of-band eval never reddens on a free-tier hiccup. The LLM-judge's scores are advisory (they never gate). On a human-labelled gold subset it agreed with human verdicts at Cohen's kappa 0.81 using an independent judge (groq/llama-3.1-8b-instant ≠ the reasoner) — validated on that subset (DR-0018); still self-grading whenever QG_JUDGE_MODEL equals the reasoner.

> CI's out-of-band eval runs on Groq (groq/llama-3.3-70b-versatile, gated on > GROQ_API_KEY): Gemini's free tier proved unusable from cloud CI (429 → 503 → > timeout → invalid-key), so the reasoner was swapped with one env var — the > model-agnostic thesis in action (DR-0017). The intended home default remains a > local Qwen3-4B (DR-0008).

Using it on your real data

The demo eats three canonical files; your production signals don't look like that. quellgeist ingest is the adapter — point it at real sources and it writes the canonical files the tools read:

quellgeist ingest \
  --logs    /var/log/myapp/     # file or directory; JSONL, JSON, plain text, or mixed
  --deploys deploys.json        # JSON array, GitHub payload, or `git log` text
  --metrics prom.json           # a Prometheus response or a canonical array
  --out-dir ./signals
# prints the `export QG_*` lines; then:
quellgeist diagnose --show-trace --strict-citations   # add --model / a provider key

It tolerates messy real data (foreign field names are aliased onto the schema, timestamps normalised to UTC, a malformed line coerced rather than crashing the run), and query_logs caps how many rows one observation returns (QG_MAX_ROWS, default 200) so a large log can't blow the context window. The deterministic cite-or-abstain guarantee runs at real-use time: diagnose verifies every cited handle against your real signals and warns on a fabrication (--strict-citations exits non-zero for CI). Full guide: [docs/ingestion.md](docs/ingestion.md).

Run the live service (v2)

v2 wraps the same frozen core in a live, concurrent, observable incident-response service: a signed webhook triggers an investigation, a worker pool runs the unchanged loop over an isolated per-incident snapshot, every run is persisted to SQLite with its trace and cost, an operator approves / steers / rejects before it posts to Slack + a self-contained HTML page, and after a sandbox fix the agent re-reads signals to confirm recovery. Everything additive; the frozen measurement surface is untouched. The whole stack runs from one file:

cp .env.example .env            # set QG_WEBHOOK_SECRET, QG_OPERATOR_TOKEN, QG_VERIFIER_MODEL…
docker compose up --build       # demo service + agent service + Ollama
docker compose exec ollama ollama pull qwen3:4b-instruct-2507-q4_K_M   # first run only

# break it, trigger an investigation, review, then confirm the fix in the sandbox:
docker compose exec demo python -m demo.chaos.bad_deploy
#   … POST a signed incident to :8000/incidents, approve it on the HTML page …
docker compose exec demo python -m demo.chaos.fix_deploy   # heal without wiping the log
#   … POST /incidents/{id}/verify-resolution → recovered | not_recovered | inconclusive

Secrets stay env-only (public repo); the service is fail-closed — no webhook secret rejects every request, no operator token closes the operator surface, and it never posts a fabricated or unverified diagnosis. Design: [DR-0023](docs/quellgeist-adr-log.md) + [spec](docs/quellgeist-v2-spec.md).

Status & roadmap

Built in rolling waves — only the current wave is implemented in detail (see [docs/quellgeist-plan-rolling-wave.md](docs/quellgeist-plan-rolling-wave.md)). The full decision history lives in the [ADR log](docs/quellgeist-adr-log.md).

| Wave | Scope | Status | |---|---|---| | 0 | De-risk the model bet (4B can orchestrate the loop) | ✅ done — default = Qwen3-4B (DR-0008) | | 1 | Bad-deploy slice: demo → break → diagnose → postmortem; eval harness + CI | ✅ done — spine built & unit-tested | | 2 | Reliability core: verifier pass, deterministic fabrication check, abstention, LLM-as-judge | ✅ built — keyless deterministic gate + opt-in verifier/judge; first real run passed with zero fabrication (DR-0016/DR-0017). Judge validation + a reliability rate carry into Wave 3 | | 3 | Breadth: config/env + resource-exhaustion classes, metrics, ~50 scenarios | ✅ done — 3 classes across a 65-scenario suite; first full run 61/65, 0 fabricated; judge validated (kappa 0.81). See the [reliability](docs/case-studies/wave3-reliability-rate.md) + [judge](docs/case-studies/wave3-judge-validation.md) case studies | | 4 | Cost / fine-tune: QLoRA Qwen3-4B vs base vs frontier, with/without verifier | ✅ done — base 0/16 → tuned 12/16 holdout (0 fabricated, 0 speculative-filter, cheaper than base); frontier-competitive vs Gemma-4-31B (beats it 10/16 on capability, ties 6/12 on abstention); resource_exhaustion unlearned + adversarial abstention a shared 6/12 ceiling ([case study](docs/case-studies/wave4-qwen-finetune.md), DR-0019/DR-0020) | | 5 | Polish & ship: HTML render, security pass, MCP registry, launch | 🚧 engineering complete — release-gated (HTML render + security scanners + threat model + registry/OIDC scaffolding done; the release tag + launch are the remaining steps) | | 6 | Resolution-verification loop | ⤳ folded into v2 (Wave 9) | | v2 (7–9) | Live incident-response service (webhook → concurrent workers → persisted runs → HITL review → Slack/HTML → sandbox resolution re-check) + Dockerfile/compose | ✅ built — Waves 7–9 shipped: signed webhook → concurrent workers → persisted cited runs → fail-closed HITL review gate → Slack/HTML → deterministic sandbox resolution re-check; non-root Docker + compose; 339 keyless tests, frozen diff empty ([DR-0023](docs/quellgeist-adr-log.md)/[0027](docs/quellgeist-adr-log.md)/[0028](docs/quellgeist-adr-log.md), [spec](docs/quellgeist-v2-spec.md)) | | v2 Track B (10) | Reliability track: timing-aware verifier + structure-varied / out-of-structure generalisation eval | 🚧 scoped — DR-0024–0026 |

The wave boundary is deliberate, not unfinished: only the current wave is built i

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Rajeev-Shyam
• Source: Rajeev-Shyam/Quellgeist
• License: MIT
• Homepage: https://pypi.org/project/quellgeist/

Install and usage instructions live in the source repository linked above.