# Quellgeist

> Incident-triage agent that cites real evidence or abstains — model-agnostic, read-only MCP servers, frontier-competitive on a local 4B at $0.

- **Type:** MCP server
- **Install:** `agentstack add mcp-rajeev-shyam-quellgeist`
- **Verified:** Pending review
- **Seller:** [Rajeev-Shyam](https://agentstack.voostack.com/s/rajeev-shyam)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Rajeev-Shyam](https://github.com/Rajeev-Shyam)
- **Source:** https://github.com/Rajeev-Shyam/Quellgeist
- **Website:** https://pypi.org/project/quellgeist/

## Install

```sh
agentstack add mcp-rajeev-shyam-quellgeist
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Quellgeist

[](https://github.com/Rajeev-Shyam/Quellgeist/actions/workflows/ci.yml)
[](https://github.com/Rajeev-Shyam/Quellgeist/actions/workflows/security.yml)
[](docs/case-studies/wave4-qwen-finetune.md)
[](LICENSE)
[](https://www.python.org/downloads/)

> First-line incident triage you can trust: ranked root-cause hypotheses where **every claim cites a real evidence handle** — and the agent **abstains rather than guess**.

Quellgeist is a model-agnostic AI agent for first-line production-incident triage.
It runs a legible JSON-action ReAct loop over read-only tools (structured logs +
recent deploys + metric time-series), then emits a structured **Diagnosis**:
confidence-ranked root-cause hypotheses, each backed by a structured evidence
handle (`LogRef.id` / `CommitRef.sha` / `MetricRef.id`) the agent actually saw —
never free text. Two ideas set it apart:

- **Cite-by-structured-handle.** Evidence is a checkable handle, not a sentence,
  so a fabricated citation is *measurable* and **deterministically rejected** by a
  keyless fabrication check — not a matter of fuzzy string-matching.
- **Abstain-over-hallucinate.** A confidently-stated wrong cause is the worst
  possible answer, so *"insufficient evidence"* is a first-class outcome.

> **Status: Wave 4 complete — the fine-tune works.** The DR-0020 QLoRA fine-tune
> of the local reasoner (Qwen3-4B, served via Ollama) took it from the base's
> **0/16 holdout to 12/16** — zero fabrication, zero speculative-filtering, and
> *cheaper* than the base — while **beating a 31B frontier** (Gemma-4-31B, 10/16)
> on the same holdout at **$0, fully offline**. Non-memorisation is triangulated
> three ways (fixtures ≈ holdout; core-fresh ≥ core-overlap; structure probe 7/10).
> Two honest limits: the `resource_exhaustion` class didn't transfer (0/N; the
> frontier passes it), and adversarial-abstention recall is **6/12** at the system
> level — a ceiling the 31B frontier *shares* (also 6/12), not a fine-tune
> regression. When this agent misses it's *incomplete* or *too cautious*, never
> confidently fabricating. See
> [Status & roadmap](#status--roadmap) · [fine-tune case study](docs/case-studies/wave4-qwen-finetune.md).

## Why it's different

| | |
|---|---|
| **Evidence is a handle** | Each hypothesis cites a log row's source-stable `id` or a commit `sha`, copied verbatim from a tool result — the unit the deterministic fabrication check looks up. Prose lives in a display-only `note`. (DR-0009) |
| **Abstention is a feature** | When signals are weak the agent returns `abstained=true` with a reason and an empty hypotheses list — enforced by the schema. |
| **Model-agnostic by construction** | The loop parses JSON actions from plain chat text, so it's identical on Gemini's free tier and a local 4-bit Qwen — no dependence on any backend's native function-calling. Swap models with one config change. (DR-0008, DR-0010) |
| **Reliability is gated, not asserted** | A keyless, deterministic CI gate (ruff + black + `pytest`, including the fixture-backed eval harness) runs on every push. |

### What it is / what it's NOT

- **It is:** a first-line *triage* agent — ranked, evidence-cited root-cause
  hypotheses (or an honest abstention) from read-only logs/deploys/metrics, over a
  model-agnostic loop that runs on a hosted frontier model **or** a local 4B.
- **It is NOT:** an autonomous remediator (it never mutates prod — resolution
  verification is a deferred, cut-first wave); a production-hardened service (the
  demo is a deliberate toy); or a general-purpose agent. The holdout it's measured
  on is *out-of-vocabulary but in-structure* — not a claim about unseen incident
  shapes or real production data.

## Quickstart (~30 seconds to a broken service + structured logs)

Requires [uv](https://docs.astral.sh/uv/) and Python 3.12+.

**See a real-shaped diagnosis in one keyless command** (no model, no API key):

```bash
uv sync && uv run quellgeist diagnose --demo     # renders the demo incident's cited postmortem
```

Then run the full loop against the live toy service:

```bash
uv run uvicorn demo.app.main:app          # 1. start the toy service (leave running)

# --- in a second shell, from the repo root ---
uv run python -m demo.chaos.bad_deploy    # 2. inject a simulated bad deploy
curl -s localhost:8000/login              # 3. trip /login -> 500s + structured error logs
uv run quellgeist diagnose --show-trace   # 4. diagnose live (needs a model; see below)

uv run python -m demo.chaos.reset         # back to a green slate
```

The live step needs a reasoner — see [Running the model](#running-the-model).
Without a key, `quellgeist diagnose` exits 1 with a one-line error + hint (never a
traceback); `--demo` always works keyless and renders the same output shape
deterministically from gold.

## Architecture

A custom, legible loop is the orchestration layer; the three read-only tools are
the evidence interface; the `Diagnosis` schema is the contract that the
postmortem renderer and the eval judge both read.

```mermaid
flowchart TD
    trigger(["incident trigger - CLI"]) --> loop
    model["reasoner via LiteLLM(Gemini or local Qwen, swappable)"] -. "chat completion" .-> loop

    subgraph loopbox["model-agnostic JSON-action ReAct loop"]
      loop["run_loop()decide, call tool, observe, repeat"]
    end

    loop -- "query_logs" --> logs["logs toolstructured JSONL, stable ids"]
    loop -- "get_recent_commits" --> commits["commits tooldeploy_log.json, shas"]
    loop -- "query_metrics" --> metrics["metrics tooltime-series, named series"]
    logs -- "rows + ids" --> loop
    commits -- "commits + shas" --> loop
    metrics -- "series + names" --> loop

    loop --> diag["Diagnosis (schema.py)ranked hypotheses citingLogRef.id / CommitRef.sha / MetricRef.id, or abstains"]
    diag --> pm["postmortem rendererdeterministic Markdown"]
    diag --> judge["eval judgefixture scenarios, CI gate"]
```

All three tools are also exposed as **MCP servers** over stdio
(`python -m quellgeist.servers.logs_mcp`, `…commits_mcp`, `…metrics_mcp`). The
agent currently reuses the same tool *functions* in-process behind a `ToolSpec`
registry; a stdio MCP-*client* path (the agent driving the servers over the
wire) is on the roadmap (DR-0010).

> **Deep dive:** [`docs/architecture.md`](docs/architecture.md) walks the full
> pipeline (loop → tools → verifier → postmortem), a sequence diagram, the module
> map, and the cross-cutting design decisions.

The servers publish to the **Official MCP Registry** on each tagged release (see
[`docs/publishing.md`](docs/publishing.md)); once published each is runnable with
`uvx --from quellgeist quellgeist-logs-mcp` (or `…-commits-mcp` / `…-metrics-mcp`).

## Example session

Inject the bad deploy — it drops a marker that flips `verify_token` into a
NoneType regression and writes a `deploy_log.json` whose offending commit landed
just before the errors (illustrative `stdout` — the timestamp reflects when you
run it; paths shown relative to the repo root):

```text
$ uv run python -m demo.chaos.bad_deploy
injected bad deploy a1b2c3d (touched demo/app/auth.py) at 2026-06-24T12:22:43Z
  marker:     demo/.bad_deploy
  deploy log: demo/deploy_log.json
next: hit /login to generate the 500s, then `quellgeist diagnose`
```

With a reasoner configured, `quellgeist diagnose` reads the logs + deploys and
emits a postmortem. The CI environment has no validated model key (DR-0012), so
the diagnosis below is **rendered from gold** — built deterministically from the
fixture's labelled cause and evidence handles via `render_postmortem`, *not*
live model output:

```text
# Incident Postmortem (rendered from gold)

## Root-cause hypotheses

### 1. Bad deploy a1b2c3d (10:01:50Z) refactored auth.py and introduced a NoneType error in verify_token; /login 500s begin ~20s later at 10:02:12Z.  (confidence: 1.00)

Evidence:
- log #2
- commit a1b2c3d
```

Reproduce that render yourself (no model needed):

```bash
uv run python -  **Base vs tuned — important.** The `ollama pull` above is the **base** Qwen3-4B:
> the honest safe *floor* — it scores **0/16 on the holdout and abstains on
> everything**, never fabricating (DR-0019). The **12/16** headline is the
> **DR-0020 fine-tune** (`quellgeist-qwen3-dr0020`), which you build + serve via
> [`finetune/README.md`](finetune/README.md) (a free-Colab QLoRA run →
> `ollama create`). Until that tuned GGUF is published for a one-line pull, the
> base model is what a plain `ollama pull` gives you — safe, not yet useful. Use a
> hosted model (above) or the fine-tune to see live diagnoses.

Heads-up (DR-0012): a Gemini key on an unvalidated, no-billing project returns
`429 limit: 0` on current models, so the shipped CI gate is deliberately
**keyless** and model-driven evals are key-gated and run **out-of-band**
(DR-0015). At home the intended default reasoner is a local **Qwen3-4B** via
Ollama (DR-0008).

### Running the eval (reasoner + verifier + LLM-judge)

The fixture eval scores the reasoner with a deterministic keyword judge + a
zero-fabrication check (the keyless gate), and can additionally run two model
layers (DR-0016): a **verifier** that confirms cited evidence supports each
hypothesis (forcing abstention otherwise) and an advisory **LLM-judge** rubric.

```bash
export GEMINI_API_KEY="…"
export QG_MODEL="gemini/gemini-3.5-flash"
QG_VERIFY=1 QG_JUDGE_LLM=1 \
QG_MIN_CALL_INTERVAL_S=6 \      # pace calls under the free-tier RPM (avoids 429 bursts)
  uv run python -m evals.run_evals
```

`QG_VERIFIER_MODEL` / `QG_JUDGE_MODEL` override the model per layer (default
`QG_MODEL`). An unreachable backend (quota/503/timeout) **or** a rejected
credential (missing/invalid/stale key) is reported as a **skip**, not a failure
(DR-0015/DR-0017), so the out-of-band eval never reddens on a free-tier hiccup.
The LLM-judge's scores are **advisory** (they never gate). On a human-labelled
gold subset it agreed with human verdicts at **Cohen's kappa 0.81** using an
independent judge (`groq/llama-3.1-8b-instant` ≠ the reasoner) — validated on that
subset (DR-0018); still self-grading whenever `QG_JUDGE_MODEL` equals the reasoner.

> **CI's out-of-band eval runs on Groq** (`groq/llama-3.3-70b-versatile`, gated on
> `GROQ_API_KEY`): Gemini's free tier proved unusable from cloud CI (429 → 503 →
> timeout → invalid-key), so the reasoner was swapped with one env var — the
> model-agnostic thesis in action (DR-0017). The intended *home* default remains a
> local Qwen3-4B (DR-0008).

## Using it on your real data

The demo eats three canonical files; your production signals don't look like that.
`quellgeist ingest` is the adapter — point it at real sources and it writes the
canonical files the tools read:

```bash
quellgeist ingest \
  --logs    /var/log/myapp/     # file or directory; JSONL, JSON, plain text, or mixed
  --deploys deploys.json        # JSON array, GitHub payload, or `git log` text
  --metrics prom.json           # a Prometheus response or a canonical array
  --out-dir ./signals
# prints the `export QG_*` lines; then:
quellgeist diagnose --show-trace --strict-citations   # add --model / a provider key
```

It tolerates messy real data (foreign field names are aliased onto the schema,
timestamps normalised to UTC, a malformed line coerced rather than crashing the
run), and `query_logs` caps how many rows one observation returns (`QG_MAX_ROWS`,
default 200) so a large log can't blow the context window. The deterministic
cite-or-abstain guarantee runs **at real-use time**: `diagnose` verifies every
cited handle against your real signals and warns on a fabrication (`--strict-citations`
exits non-zero for CI). Full guide: [`docs/ingestion.md`](docs/ingestion.md).

## Run the live service (v2)

v2 wraps the same frozen core in a live, concurrent, observable incident-response
service: a **signed webhook** triggers an investigation, a worker pool runs the
*unchanged* loop over an isolated per-incident snapshot, every run is persisted to
SQLite with its trace and cost, an operator **approves / steers / rejects** before it
posts to Slack + a self-contained HTML page, and after a sandbox fix the agent re-reads
signals to confirm recovery. Everything additive; the frozen measurement surface is
untouched. The whole stack runs from one file:

```bash
cp .env.example .env            # set QG_WEBHOOK_SECRET, QG_OPERATOR_TOKEN, QG_VERIFIER_MODEL…
docker compose up --build       # demo service + agent service + Ollama
docker compose exec ollama ollama pull qwen3:4b-instruct-2507-q4_K_M   # first run only

# break it, trigger an investigation, review, then confirm the fix in the sandbox:
docker compose exec demo python -m demo.chaos.bad_deploy
#   … POST a signed incident to :8000/incidents, approve it on the HTML page …
docker compose exec demo python -m demo.chaos.fix_deploy   # heal without wiping the log
#   … POST /incidents/{id}/verify-resolution → recovered | not_recovered | inconclusive
```

Secrets stay env-only (public repo); the service is **fail-closed** — no webhook secret
rejects every request, no operator token closes the operator surface, and it never posts a
fabricated *or* unverified diagnosis. Design: [DR-0023](docs/quellgeist-adr-log.md) +
[spec](docs/quellgeist-v2-spec.md).

## Status & roadmap

Built in **rolling waves** — only the current wave is implemented in detail
(see [`docs/quellgeist-plan-rolling-wave.md`](docs/quellgeist-plan-rolling-wave.md)).
The full decision history lives in the
[**ADR log**](docs/quellgeist-adr-log.md).

| Wave | Scope | Status |
|---|---|---|
| 0 | De-risk the model bet (4B can orchestrate the loop) | ✅ done — default = Qwen3-4B (DR-0008) |
| 1 | Bad-deploy slice: demo → break → diagnose → postmortem; eval harness + CI | ✅ done — spine built & unit-tested |
| 2 | Reliability core: verifier pass, deterministic fabrication check, abstention, LLM-as-judge | ✅ built — keyless deterministic gate + opt-in verifier/judge; first real run passed with zero fabrication (DR-0016/DR-0017). Judge validation + a reliability *rate* carry into Wave 3 |
| 3 | Breadth: config/env + resource-exhaustion classes, metrics, ~50 scenarios | ✅ done — 3 classes across a 65-scenario suite; first full run **61/65, 0 fabricated**; judge validated (kappa 0.81). See the [reliability](docs/case-studies/wave3-reliability-rate.md) + [judge](docs/case-studies/wave3-judge-validation.md) case studies |
| **4** | **Cost / fine-tune: QLoRA Qwen3-4B vs base vs frontier, with/without verifier** | ✅ **done** — base **0/16 → tuned 12/16** holdout (0 fabricated, 0 speculative-filter, cheaper than base); frontier-competitive vs Gemma-4-31B (beats it 10/16 on capability, ties 6/12 on abstention); `resource_exhaustion` unlearned + adversarial abstention a shared 6/12 ceiling ([case study](docs/case-studies/wave4-qwen-finetune.md), DR-0019/DR-0020) |
| **5** | Polish & ship: HTML render, security pass, MCP registry, launch | 🚧 **engineering complete — release-gated** (HTML render + security scanners + threat model + registry/OIDC scaffolding done; the release tag + launch are the remaining steps) |
| 6 | Resolution-verification loop | ⤳ folded into v2 (Wave 9) |
| **v2 (7–9)** | Live incident-response service (webhook → concurrent workers → persisted runs → HITL review → Slack/HTML → sandbox resolution re-check) + Dockerfile/compose | ✅ **built** — Waves 7–9 shipped: signed webhook → concurrent workers → persisted cited runs → fail-closed HITL review gate → Slack/HTML → deterministic sandbox resolution re-check; non-root Docker + compose; **339 keyless tests**, frozen diff empty ([DR-0023](docs/quellgeist-adr-log.md)/[0027](docs/quellgeist-adr-log.md)/[0028](docs/quellgeist-adr-log.md), [spec](docs/quellgeist-v2-spec.md)) |
| v2 Track B (10) | Reliability track: timing-aware verifier + structure-varied / out-of-structure generalisation eval | 🚧 scoped — DR-0024–0026 |

The wave boundary is deliberate, not unfinished: only the current wave is built i

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Rajeev-Shyam](https://github.com/Rajeev-Shyam)
- **Source:** [Rajeev-Shyam/Quellgeist](https://github.com/Rajeev-Shyam/Quellgeist)
- **License:** MIT
- **Homepage:** https://pypi.org/project/quellgeist/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** yes

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: flagged — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-rajeev-shyam-quellgeist
- Seller: https://agentstack.voostack.com/s/rajeev-shyam
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.