Wiki Ingest

Wiki-Ingest — LLM-Wiki Maintenance Layer

Purpose: Turn raw sources into nodes of a compounding knowledge base, not isolated summaries. Karpathy's llm-wiki pattern says: each new source must (a) get its own summary page AND (b) touch 10–15 other pages — concept pages, entity pages, the index, the log. summarizing-meetings already produces excellent single-file summaries; this skill is the maintenance layer that wires those summaries into a living wiki.

The skill enforces three invariants:

1. Raw layer is immutable — sources are read, never modified.
2. Wiki layer is LLM-owned and additive — pages get richer over time, prior content is never silently overwritten.
3. Every claim on a concept/entity page traces back to a source via footnote citation. An auditable wiki at 50 ingests, not a noise pile.

1. Red Flags (Anti-Rationalization)

STOP and READ THIS if you are thinking:

• "I'll just create the summary, the user can update concepts themselves" → WRONG. That's exactly the maintenance burden llm-wiki is designed to remove. Without page upserts the [[wiki-links]] stay dangling and the vault never compounds. Always do the full ingest cycle.
• "The existing concept page is short, I'll rewrite it cleaner" → WRONG. Concept pages are additive. You append new facts under existing ones with footnote citations. Rewriting destroys the audit trail and silently drops prior sources' contributions.
• "These two facts disagree, I'll pick the newer one" → WRONG. Mark with a ⚠️ Contradiction: block citing both sources. The operator decides — your job is to surface, not silently resolve.
• "I'll skip the log entry, the user can see what changed from git" → WRONG. log.md is grep-friendly chronological memory for the LLM in future sessions. Git diffs are for humans; the log is for you, next time.
• "WIKI_SCHEMA.md isn't there, I'll improvise conventions" → WRONG. Run wiki_ops.py init to scaffold it from the bundled template, then read it. Improvised conventions diverge across ingests and break the wiki.
• "The source has no obvious concepts, I'll create a summary and skip upserts" → WRONG. Every source has at least 2–3 concepts/entities worth a page. Re-read; if you genuinely find none, ask the operator before proceeding.
• "I'll call summarizing-meetings without telling it what concepts already exist" → WRONG. Pass the known-concepts list. Otherwise the summary's [[wiki-links]] use new variant names ("Hermes" vs "Hermes Agent") and dangle forever.

2. Capabilities

The skill supports four modes. Most uses invoke ingest (the default); query, lint, and reindex are maintenance modes.

• ingest (default) — process one source into the wiki:
• Scaffold a fresh vault (WIKI_SCHEMA.md, index.md, log.md, _sources/, _concepts/, _entities/) on first run
• Discover existing concepts/entities by scanning frontmatter
• Delegate summary generation to summarizing-meetings with vault context injected
• Upsert concept/entity pages additively, with footnote citations
• Detect and flag (not resolve) contradictions
• Update index.md and append to log.md
• Idempotent: re-running an ingest of the same source does not duplicate rows or footnotes
• query — answer a question against the wiki:
• Search pages by keyword to assemble a shortlist
• Read shortlisted pages, synthesise an answer with [[wiki-link]] citations
• Optionally file the answer back as a new wiki page (compounding pattern)
• Append a query entry to log.md
• lint — health check the vault:
• Report orphan pages (no inbound [[wiki-links]])
• Report dangling link targets (referenced but no page)
• Report open ## Contradictions blocks pending operator resolution
• Report concepts mentioned in N+ sources without a dedicated page
• reindex — rebuild index.md from disk, recovering from drift; preserves the ## Notes section
• promote / demote (cross-course; vault-root mode) — lazy operator-triggered merging of same-named concept/entity pages across courses into the vault's shared _concepts//_entities/. See Phase P below and [references/cross_course_promotion.md](references/crosscoursepromotion.md) for the full operator playbook.

Two-Tier Vault Model (cross-course)

When the operator's vault holds multiple parallel courses each with their own WIKI_SCHEMA.md (schema_version 1.x), an OUTER WIKI_SCHEMA.md at the vault root (schema_version: 2.0, kind: vault-root) turns on the shared layer. _concepts/ and _entities/ at the vault root hold concepts/entities the operator has explicitly promoted out of individual courses. Sources never live at root (spec §2.5); the shared layer is lazy — populated only via promote (R8.5 / R13).

Invariant (load-bearing): a given canonical filename lives in exactly one place — either some course's _concepts//_entities/ OR the root's. lint detects violations and exits non-zero.

Install on PATH (TASK 017 R1)

The skill ships a thin POSIX shell wrapper at [scripts/wiki-ingest](scripts/wiki-ingest) (no .sh suffix) so the binary is invokable as wiki-ingest instead of python3 scripts/wiki_ops.py . The wrapper resolves its own path via readlink -f (GNU + BusyBox) with a python3 os.path.realpath fallback for stock macOS BSD readlink.

# Install (any one of these patterns works):
ln -s "$PWD/skills/wiki-ingest/scripts/wiki-ingest" ~/.local/bin/wiki-ingest

# Or invoke directly:
skills/wiki-ingest/scripts/wiki-ingest --version    # → "wiki-ingest 1.1.0"

Both forms — wiki-ingest (wrapper) and python3 wiki_ops.py (direct) — produce identical stdout / stderr / exit codes (locked by tests/test_cli_wrapper.py equivalence assertions). The direct form is the supported pattern for tooling that cannot rely on PATH; the wrapper is the supported pattern for human operators and external bridges (e.g., the obsidian-llm-wiki /wiki-enrich skill shells out as wiki-ingest …).

Top-level orchestrator: wiki-ingest ingest (TASK 017)

The v1.1 contract adds ingest — a single subcommand that composes the atomic ops (register-summary → N× upsert-page → update-index → append-log → log-event) into one call and emits a stable JSON manifest on stdout. Designed for external bridges (e.g., obsidian-llm-wiki /wiki-enrich) that need an end-to-end ingest in one subprocess invocation.

wiki-ingest ingest --source  \
                   --vault   \
                   --output-format json \
                   [--vault-id ] \
                   [--source-hash ]

Scope (v1.1): --source MUST be a pre-made summary (frontmatter type: ∈ {summary, lesson-summary, meeting-summary}). Raw transcripts fail-fast with phase:"needs-pre-summarization" + exit 26 — the operator / bridge pre-summarises via the [summarizing-meetings](../summarizing-meetings/) skill first. (The synthesizer-subagent integration is a documented future enhancement; see [references/manifest_schema.md](references/manifest_schema.md) §8.)

Manifest shape: see [references/manifest_schema.md](references/manifest_schema.md) (in-repo mirror of the external CONTRACT). Top-level fields include manifest_version: "1.1", vault_id (string or null), vault_root, course (string or null per Q-5), source: {path, slug, hash}, written[] (each entry {path, action, kind, scope}), created[], touched[], contradictions, summary_path, log_event, llm_tokens_used.

Exit codes: see [references/exit_codes.md](references/exit_codes.md). The v1.1 contract band is 20..26 — EXIT_PARTIAL=20, EXIT_SUBPROCESS=21, EXIT_LLM=22, EXIT_MISSING_VAULT_ID=23, EXIT_INVALID_VAULT_ID=24, EXIT_VAULT_ID_MISMATCH=25, EXIT_TIMEOUT=26.

Idempotency (UC-4): the orchestrator records source_hash: in the registered _sources/.md frontmatter on first ingest. Re-running with --source-hash short-circuits — emits action: "unchanged" + empty written[], exits 0.

vault_id migration (TASK 017 R3)

Existing two-tier vaults that want to integrate with the index layer (e.g., obsidian-llm-wiki) MUST add a vault_id: field to the root WIKI_SCHEMA.md frontmatter:

schema_version: "2.0"
kind: vault-root
vault_id: my-vault              # ← one-line add

The slug pattern is ^[a-z][a-z0-9-]{1,30}[a-z0-9]$ (3..32 chars, lowercase ASCII kebab-case, no --). Standalone wiki-ingest users DO NOT need this field — it stays absent and the orchestrator emits vault_id: null in the manifest. Strict-mode --vault-id validation only fires when callers (e.g., the bridge) demand it.

To scaffold a new vault root WITH a vault_id:

wiki-ingest init  --root --vault-id my-vault

3. Execution Mode

• Mode: hybrid
• Why this mode: File-format operations (upsert without duplicates, append to log, dedupe footnotes, rewrite index sections) are deterministic and MUST be scripted — text-based logic at this density fails 30% of the time. Judgement steps (which facts from the new summary belong on which concept page, whether two claims contradict, whether two concept names refer to the same thing) MUST stay with the LLM. The split keeps both halves reliable.

4. Script Contract

• Command(s):
• python3 scripts/wiki_ops.py scan → JSON dump of vault state (existing concepts, entities, sources, schema presence)
• python3 scripts/wiki_ops.py init → scaffold missing WIKI_SCHEMA.md, index.md, log.md, and standard subdirs (idempotent)
• python3 scripts/wiki_ops.py upsert-page --kind {concept|entity} --name --source-slug --source-title --source-date [--definition ] [--fact ] [--contradicts ]
• python3 scripts/wiki_ops.py register-summary --summary-path [--slug ] [--title ] [--force] → ingest a pre-made summary file directly into _sources/ (skip delegating to summarizing-meetings). Returns JSON with slug/title/date/concepts/related for Phase 3+.
• python3 scripts/wiki_ops.py update-index --source-slug --source-title --source-date --summary [--new-concepts ] [--new-entities ]
• python3 scripts/wiki_ops.py append-log --title --slug --source-path --touched --created --contradictions — ingest-specific shortcut
• python3 scripts/wiki_ops.py log-event --event --title [--detail key=value ...] — generic log append (query, lint, reindex events)
• python3 scripts/wiki_ops.py find --terms "" [--limit N] [--kinds source,concept,entity] → ranked JSON hits for the keyword search backing query mode
• python3 scripts/wiki_ops.py lint [--threshold N] → JSON health report (orphans, dangling links, open contradictions, missing concept pages)
• python3 scripts/wiki_ops.py reindex → rebuild index.md from on-disk pages (preserves ## Notes)
• python3 scripts/wiki_ops.py classify-folder [--group-by ] → Phase 0 of folder-ingest: detect grouping pattern + classify each file into primary/metadata/merge/link/derived-output; emit a plan JSON. Pure read-only; no vault required.
• python3 scripts/wiki_ops.py init --root → scaffold a vault-ROOT layer (WIKI_SCHEMA.md with schema_version: 2.0, _concepts/, _entities/). NO _sources/ or log.md at the root. Idempotent; never overwrites. See [references/cross_course_promotion.md](references/crosscoursepromotion.md).
• python3 scripts/wiki_ops.py promote --vault [--kind concept|entity] [--apply] → merge ≥2 course-local copies of the same concept/entity into a single root-level page. Dry-run by default; --apply commits. Unions frontmatter (earliest created, longer description, promoted_from: list), additive body merge, rewrites footnotes to vault-relative form, deletes course copies, updates course index.mds + root index.md, appends log.md entries. Detects literal-line-diff contradictions and surfaces them via ## Contradictions blocks (operator resolves).
• python3 scripts/wiki_ops.py demote --to --vault [--dry-run] → move a root-level page back to a named course. Refuses if any course OTHER than --to cites the page via its _sources/. Rewrites footnotes to short form, strips promoted_from:, updates indexes + log. --dry-run NOT default (demote is reversible; cheap to undo).
• wiki-ingest ingest --source --vault [--output-format json] [--vault-id ] [--source-hash ] → v1.1 top-level orchestrator (TASK 017). Composes register-summary → upsert-page × N → update-index → append-log → log-event in one call. Emits a stable JSON manifest. Scope: summary-passthrough only (raw transcripts pre-summarised by the operator / bridge). Full contract in [references/manifest_schema.md](references/manifestschema.md). Exit codes in [references/exit_codes.md](references/exitcodes.md) (v1.1 band: 20..26).
• wiki-ingest --version → emits wiki-ingest 1.1.0\n and exits 0 (CONTRACT §7 minimum-version check).
• Inputs: vault root path; for upserts, the source's slug/title/date and the concept name and either a definition (stub-creation) or a fact (additive update).
• Outputs: stdout JSON for scan; mutated markdown files for the rest; non-zero exit on missing required args or invalid vault.
• Failure semantics: exits 1 with stderr message on invalid args / missing vault; exits 2 if WIKI_SCHEMA.md is absent and init was not run first (prevents improvised conventions).
• Idempotency: init is fully idempotent — never overwrites existing files. upsert-page deduplicates source rows and footnotes by slug, and refuses case-colliding names (--force to override). update-index deduplicates rows by slug. append-log deduplicates by (date, event, source-slug) — re-running an identical ingest is a no-op; pass --force-log to append anyway.
• Dry-run support: --dry-run on all mutating subcommands prints the diff to stdout without writing.

5. Safety Boundaries

• Allowed scope: a single vault directory provided as ``. All writes stay inside it.
• Default exclusions: raw source files (operator's transcripts, articles); the script reads them only when invoked by the agent for inspection, never mutates them.
• Destructive actions: the only operations that can remove operator content are reindex (rewrites index.md) and --force overrides. reindex preserves every non-default section (anything outside _sources/_concepts/Entities — including Notes and any custom sections the operator added) — but its output is reported as a list under preserved_sections; review the diff before accepting. The agent never passes --force or --force-log without explicit operator approval.
• Contradiction handling is non-destructive: contradictions are marked with a ⚠️ Contradiction: block linking both sources. The skill never auto-resolves. The --contradicts text MUST match a substring on the existing page (script verifies); operator must pass --force to record an unverified citation.
• Filesystem safety: --name/--slug/--source-slug reject path traversal (.., /, \, leading .), control characters, markdown link metacharacters ([, ], |, ^), template placeholders ({{, }}), and overlong (>200 char) values. Case-insensitive collisions (e.g., sharpe score.md vs Sharpe Score.md on macOS APFS) AND slug-equivalent collisions (e.g., _Foo_.md vs foo.md) are detected and refused without --force. Names are NFKC-normalised so visually-identical Unicode variants (composed vs decomposed, fullwidth vs ASCII) collapse onto the same on-disk filename.
• No network calls: the script is local-only. Transcript fetching is a separate skill (transcript-fetcher); plumb that upstream if needed.

6. Validation Evidence

• Local verification:
• python3 scripts/wiki_ops.py scan after ingest → confirm last_ingest slug and counts match expectations
• `grep "^## \[" /log.md | tail -

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: MatrixFounder
• Source: MatrixFounder/Universal-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.