# Wiki Ingest

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-matrixfounder-universal-skills-wiki-ingest`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [MatrixFounder](https://agentstack.voostack.com/s/matrixfounder)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [MatrixFounder](https://github.com/MatrixFounder)
- **Source:** https://github.com/MatrixFounder/Universal-skills/tree/main/skills/wiki-ingest

## Install

```sh
agentstack add skill-matrixfounder-universal-skills-wiki-ingest
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Wiki-Ingest — LLM-Wiki Maintenance Layer

**Purpose**: Turn raw sources into nodes of a *compounding* knowledge base, not isolated summaries. Karpathy's llm-wiki pattern says: each new source must (a) get its own summary page AND (b) touch 10–15 other pages — concept pages, entity pages, the index, the log. `summarizing-meetings` already produces excellent single-file summaries; this skill is the *maintenance layer* that wires those summaries into a living wiki.

The skill enforces three invariants:
1. **Raw layer is immutable** — sources are read, never modified.
2. **Wiki layer is LLM-owned and additive** — pages get richer over time, prior content is never silently overwritten.
3. **Every claim on a concept/entity page traces back to a source** via footnote citation. An auditable wiki at 50 ingests, not a noise pile.

## 1. Red Flags (Anti-Rationalization)

**STOP and READ THIS if you are thinking:**
- "I'll just create the summary, the user can update concepts themselves" → **WRONG**. That's exactly the maintenance burden llm-wiki is designed to remove. Without page upserts the `[[wiki-links]]` stay dangling and the vault never compounds. Always do the full ingest cycle.
- "The existing concept page is short, I'll rewrite it cleaner" → **WRONG**. Concept pages are *additive*. You append new facts under existing ones with footnote citations. Rewriting destroys the audit trail and silently drops prior sources' contributions.
- "These two facts disagree, I'll pick the newer one" → **WRONG**. Mark with a `⚠️ Contradiction:` block citing both sources. The operator decides — your job is to surface, not silently resolve.
- "I'll skip the log entry, the user can see what changed from git" → **WRONG**. log.md is grep-friendly chronological memory for *the LLM in future sessions*. Git diffs are for humans; the log is for you, next time.
- "WIKI_SCHEMA.md isn't there, I'll improvise conventions" → **WRONG**. Run `wiki_ops.py init` to scaffold it from the bundled template, then read it. Improvised conventions diverge across ingests and break the wiki.
- "The source has no obvious concepts, I'll create a summary and skip upserts" → **WRONG**. Every source has at least 2–3 concepts/entities worth a page. Re-read; if you genuinely find none, ask the operator before proceeding.
- "I'll call summarizing-meetings without telling it what concepts already exist" → **WRONG**. Pass the known-concepts list. Otherwise the summary's `[[wiki-links]]` use new variant names ("Hermes" vs "Hermes Agent") and dangle forever.

## 2. Capabilities

The skill supports four modes. Most uses invoke `ingest` (the default); `query`, `lint`, and `reindex` are maintenance modes.

- **`ingest`** (default) — process one source into the wiki:
  - Scaffold a fresh vault (`WIKI_SCHEMA.md`, `index.md`, `log.md`, `_sources/`, `_concepts/`, `_entities/`) on first run
  - Discover existing concepts/entities by scanning frontmatter
  - Delegate summary generation to `summarizing-meetings` with vault context injected
  - Upsert concept/entity pages additively, with footnote citations
  - Detect and flag (not resolve) contradictions
  - Update `index.md` and append to `log.md`
  - Idempotent: re-running an ingest of the same source does not duplicate rows or footnotes
- **`query`** — answer a question against the wiki:
  - Search pages by keyword to assemble a shortlist
  - Read shortlisted pages, synthesise an answer with `[[wiki-link]]` citations
  - Optionally file the answer back as a new wiki page (compounding pattern)
  - Append a `query` entry to `log.md`
- **`lint`** — health check the vault:
  - Report orphan pages (no inbound `[[wiki-links]]`)
  - Report dangling link targets (referenced but no page)
  - Report open `## Contradictions` blocks pending operator resolution
  - Report concepts mentioned in N+ sources without a dedicated page
- **`reindex`** — rebuild `index.md` from disk, recovering from drift; preserves the `## Notes` section
- **`promote` / `demote`** (cross-course; vault-root mode) — lazy operator-triggered merging of same-named concept/entity pages across courses into the vault's shared `_concepts/`/`_entities/`. See **Phase P** below and [`references/cross_course_promotion.md`](references/cross_course_promotion.md) for the full operator playbook.

## Two-Tier Vault Model (cross-course)

When the operator's vault holds **multiple parallel courses** each
with their own `WIKI_SCHEMA.md` (schema_version `1.x`), an OUTER
`WIKI_SCHEMA.md` at the vault root (`schema_version: 2.0`, `kind: vault-root`)
turns on the shared layer. `_concepts/` and `_entities/` at the vault
root hold concepts/entities the operator has explicitly promoted out
of individual courses. **Sources never live at root** (spec §2.5); the
shared layer is **lazy** — populated only via `promote` (R8.5 / R13).

**Invariant (load-bearing)**: a given canonical filename lives in
**exactly one place** — either some course's `_concepts/`/`_entities/`
OR the root's. `lint` detects violations and exits non-zero.

## Install on PATH (TASK 017 R1)

The skill ships a thin POSIX shell wrapper at
[`scripts/wiki-ingest`](scripts/wiki-ingest) (no `.sh` suffix) so the
binary is invokable as `wiki-ingest ` instead of
`python3 scripts/wiki_ops.py `. The wrapper resolves its
own path via `readlink -f` (GNU + BusyBox) with a `python3
os.path.realpath` fallback for stock macOS BSD `readlink`.

```sh
# Install (any one of these patterns works):
ln -s "$PWD/skills/wiki-ingest/scripts/wiki-ingest" ~/.local/bin/wiki-ingest

# Or invoke directly:
skills/wiki-ingest/scripts/wiki-ingest --version    # → "wiki-ingest 1.1.0"
```

Both forms — `wiki-ingest ` (wrapper) and `python3 wiki_ops.py
` (direct) — produce identical stdout / stderr / exit codes
(locked by `tests/test_cli_wrapper.py` equivalence assertions). The
direct form is the supported pattern for tooling that cannot rely on
PATH; the wrapper is the supported pattern for human operators and
external bridges (e.g., the `obsidian-llm-wiki` `/wiki-enrich` skill
shells out as `wiki-ingest …`).

## Top-level orchestrator: `wiki-ingest ingest` (TASK 017)

The v1.1 contract adds `ingest` — a single subcommand that composes
the atomic ops (`register-summary` → N× `upsert-page` → `update-index`
→ `append-log` → `log-event`) into one call and emits a stable JSON
manifest on stdout. Designed for external bridges (e.g.,
`obsidian-llm-wiki` `/wiki-enrich`) that need an end-to-end ingest in
one subprocess invocation.

```sh
wiki-ingest ingest --source  \
                   --vault   \
                   --output-format json \
                   [--vault-id ] \
                   [--source-hash ]
```

**Scope (v1.1)**: `--source` MUST be a pre-made summary (frontmatter
`type:` ∈ {`summary`, `lesson-summary`, `meeting-summary`}). Raw
transcripts fail-fast with `phase:"needs-pre-summarization"` + exit
26 — the operator / bridge pre-summarises via the
[`summarizing-meetings`](../summarizing-meetings/) skill first. (The
synthesizer-subagent integration is a documented future enhancement;
see [`references/manifest_schema.md`](references/manifest_schema.md) §8.)

**Manifest shape**: see
[`references/manifest_schema.md`](references/manifest_schema.md) (in-repo
mirror of the external CONTRACT). Top-level fields include
`manifest_version: "1.1"`, `vault_id` (string or null), `vault_root`,
`course` (string or null per Q-5), `source: {path, slug, hash}`,
`written[]` (each entry `{path, action, kind, scope}`), `created[]`,
`touched[]`, `contradictions`, `summary_path`, `log_event`,
`llm_tokens_used`.

**Exit codes**: see
[`references/exit_codes.md`](references/exit_codes.md). The v1.1
contract band is 20..26 — `EXIT_PARTIAL=20`, `EXIT_SUBPROCESS=21`,
`EXIT_LLM=22`, `EXIT_MISSING_VAULT_ID=23`, `EXIT_INVALID_VAULT_ID=24`,
`EXIT_VAULT_ID_MISMATCH=25`, `EXIT_TIMEOUT=26`.

**Idempotency (UC-4)**: the orchestrator records `source_hash:` in
the registered `_sources/.md` frontmatter on first ingest.
Re-running with `--source-hash ` short-circuits — emits
`action: "unchanged"` + empty `written[]`, exits 0.

## vault_id migration (TASK 017 R3)

Existing two-tier vaults that want to integrate with the index layer
(e.g., `obsidian-llm-wiki`) MUST add a `vault_id: ` field to
the root `WIKI_SCHEMA.md` frontmatter:

```yaml
schema_version: "2.0"
kind: vault-root
vault_id: my-vault              # ← one-line add
```

The slug pattern is `^[a-z][a-z0-9-]{1,30}[a-z0-9]$` (3..32 chars,
lowercase ASCII kebab-case, no `--`). Standalone wiki-ingest users
DO NOT need this field — it stays absent and the orchestrator emits
`vault_id: null` in the manifest. Strict-mode `--vault-id `
validation only fires when callers (e.g., the bridge) demand it.

To scaffold a new vault root WITH a `vault_id`:

```sh
wiki-ingest init  --root --vault-id my-vault
```

## 3. Execution Mode

- **Mode**: `hybrid`
- **Why this mode**: File-format operations (upsert without duplicates, append to log, dedupe footnotes, rewrite index sections) are deterministic and MUST be scripted — text-based logic at this density fails 30% of the time. Judgement steps (which facts from the new summary belong on which concept page, whether two claims contradict, whether two concept names refer to the same thing) MUST stay with the LLM. The split keeps both halves reliable.

## 4. Script Contract

- **Command(s)**:
  - `python3 scripts/wiki_ops.py scan ` → JSON dump of vault state (existing concepts, entities, sources, schema presence)
  - `python3 scripts/wiki_ops.py init ` → scaffold missing `WIKI_SCHEMA.md`, `index.md`, `log.md`, and standard subdirs (idempotent)
  - `python3 scripts/wiki_ops.py upsert-page  --kind {concept|entity} --name  --source-slug  --source-title  --source-date  [--definition ] [--fact ] [--contradicts ]`
  - `python3 scripts/wiki_ops.py register-summary  --summary-path  [--slug ] [--title ] [--force]` → ingest a pre-made summary file directly into `_sources/` (skip delegating to `summarizing-meetings`). Returns JSON with slug/title/date/concepts/related for Phase 3+.
  - `python3 scripts/wiki_ops.py update-index  --source-slug  --source-title  --source-date  --summary  [--new-concepts ] [--new-entities ]`
  - `python3 scripts/wiki_ops.py append-log  --title  --slug  --source-path  --touched  --created  --contradictions ` — ingest-specific shortcut
  - `python3 scripts/wiki_ops.py log-event  --event  --title  [--detail key=value ...]` — generic log append (query, lint, reindex events)
  - `python3 scripts/wiki_ops.py find  --terms "" [--limit N] [--kinds source,concept,entity]` → ranked JSON hits for the keyword search backing `query` mode
  - `python3 scripts/wiki_ops.py lint  [--threshold N]` → JSON health report (orphans, dangling links, open contradictions, missing concept pages)
  - `python3 scripts/wiki_ops.py reindex ` → rebuild `index.md` from on-disk pages (preserves `## Notes`)
  - `python3 scripts/wiki_ops.py classify-folder  [--group-by ]` → Phase 0 of folder-ingest: detect grouping pattern + classify each file into primary/metadata/merge/link/derived-output; emit a plan JSON. Pure read-only; no vault required.
  - `python3 scripts/wiki_ops.py init  --root` → scaffold a vault-ROOT layer (`WIKI_SCHEMA.md` with `schema_version: 2.0`, `_concepts/`, `_entities/`). NO `_sources/` or `log.md` at the root. Idempotent; never overwrites. See [`references/cross_course_promotion.md`](references/cross_course_promotion.md).
  - `python3 scripts/wiki_ops.py promote  --vault  [--kind concept|entity] [--apply]` → merge ≥2 course-local copies of the same concept/entity into a single root-level page. **Dry-run by default**; `--apply` commits. Unions frontmatter (earliest `created`, longer `description`, `promoted_from:` list), additive body merge, rewrites footnotes to vault-relative form, deletes course copies, updates course `index.md`s + root `index.md`, appends `log.md` entries. Detects literal-line-diff contradictions and surfaces them via `## Contradictions` blocks (operator resolves).
  - `python3 scripts/wiki_ops.py demote  --to  --vault  [--dry-run]` → move a root-level page back to a named course. Refuses if any course OTHER than `--to` cites the page via its `_sources/`. Rewrites footnotes to short form, strips `promoted_from:`, updates indexes + log. `--dry-run` NOT default (demote is reversible; cheap to undo).
  - `wiki-ingest ingest --source  --vault  [--output-format json] [--vault-id ] [--source-hash ]` → **v1.1 top-level orchestrator (TASK 017)**. Composes register-summary → upsert-page × N → update-index → append-log → log-event in one call. Emits a stable JSON manifest. Scope: summary-passthrough only (raw transcripts pre-summarised by the operator / bridge). Full contract in [`references/manifest_schema.md`](references/manifest_schema.md). Exit codes in [`references/exit_codes.md`](references/exit_codes.md) (v1.1 band: 20..26).
  - `wiki-ingest --version` → emits `wiki-ingest 1.1.0\n` and exits 0 (CONTRACT §7 minimum-version check).
- **Inputs**: vault root path; for upserts, the source's slug/title/date and the concept name and either a definition (stub-creation) or a fact (additive update).
- **Outputs**: stdout JSON for `scan`; mutated markdown files for the rest; non-zero exit on missing required args or invalid vault.
- **Failure semantics**: exits 1 with stderr message on invalid args / missing vault; exits 2 if `WIKI_SCHEMA.md` is absent and `init` was not run first (prevents improvised conventions).
- **Idempotency**: `init` is fully idempotent — never overwrites existing files. `upsert-page` deduplicates source rows and footnotes by slug, and refuses case-colliding names (`--force` to override). `update-index` deduplicates rows by slug. `append-log` deduplicates by (date, event, source-slug) — re-running an identical ingest is a no-op; pass `--force-log` to append anyway.
- **Dry-run support**: `--dry-run` on all mutating subcommands prints the diff to stdout without writing.

## 5. Safety Boundaries

- **Allowed scope**: a single vault directory provided as ``. All writes stay inside it.
- **Default exclusions**: raw source files (operator's transcripts, articles); the script reads them only when invoked by the agent for inspection, never mutates them.
- **Destructive actions**: the only operations that can remove operator content are `reindex` (rewrites `index.md`) and `--force` overrides. `reindex` preserves every non-default section (anything outside `_sources/_concepts/Entities` — including `Notes` and any custom sections the operator added) — but its output is reported as a list under `preserved_sections`; review the diff before accepting. The agent never passes `--force` or `--force-log` without explicit operator approval.
- **Contradiction handling is non-destructive**: contradictions are *marked* with a `⚠️ Contradiction:` block linking both sources. The skill never auto-resolves. The `--contradicts` text MUST match a substring on the existing page (script verifies); operator must pass `--force` to record an unverified citation.
- **Filesystem safety**: `--name`/`--slug`/`--source-slug` reject path traversal (`..`, `/`, `\`, leading `.`), control characters, markdown link metacharacters (`[`, `]`, `|`, `^`), template placeholders (`{{`, `}}`), and overlong (>200 char) values. Case-insensitive collisions (e.g., `sharpe score.md` vs `Sharpe Score.md` on macOS APFS) AND slug-equivalent collisions (e.g., `_Foo_.md` vs `foo.md`) are detected and refused without `--force`. Names are NFKC-normalised so visually-identical Unicode variants (composed vs decomposed, fullwidth vs ASCII) collapse onto the same on-disk filename.
- **No network calls**: the script is local-only. Transcript fetching is a separate skill (`transcript-fetcher`); plumb that upstream if needed.

## 6. Validation Evidence

- **Local verification**:
  - `python3 scripts/wiki_ops.py scan ` after ingest → confirm `last_ingest` slug and counts match expectations
  - `grep "^## \[" /log.md | tail -

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [MatrixFounder](https://github.com/MatrixFounder)
- **Source:** [MatrixFounder/Universal-skills](https://github.com/MatrixFounder/Universal-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** yes
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-matrixfounder-universal-skills-wiki-ingest
- Seller: https://agentstack.voostack.com/s/matrixfounder
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.