Knowledge Layer

Knowledge layer for AI-native teams

[](LICENSE) [](https://github.com/mihaelamaciuca/knowledge-layer/actions/workflows/audit-docs.yml) [](https://www.python.org/downloads/)

[Architecture](ARCHITECTURE.md) · [Quickstart](QUICKSTART.md) · [Adoption](ADOPTION.md) · [Contributing](CONTRIBUTING.md) · [Method](methodology.md) · [Writing guide](docs/00-fwk-writing-guide.md)

> Your team's memory, made queryable by both humans and agents. A write-first method, a reference implementation, and a forkable starting point.

A knowledge layer is a single, agent-readable substrate that underlies product, engineering, design, and operations work. It serves two populations through the same interface: the humans on the team (who write the documents and read them back), and the agents that execute work on the team's behalf (Claude Code, Claude.ai, or any other MCP-aware client). Both ask the same questions of the same store; both get the same answers, grounded in the team's settled truth rather than in training data, the public web, or the last few messages in a chat window.

The method that produces the substrate is write-first: every spec, decision, and policy is written down before it's acted on. The layer is the artefact that makes those writings useful to machines as well as to people.

This repository is the reference implementation. It ships the indexer, the MCP server, the per-PR audits, and the document standards. Everything except the corpus. The corpus is yours to write.

flowchart LR
    humans(["Humanson the team"]):::pop
    layer["Knowledge layer(corpus + indexer + MCP server)"]:::core
    agents(["AI agentsClaude Code, Claude.ai, ..."]):::pop

    humans |write and read| layer
    layer |7 MCP tools| agents

    classDef pop fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e;
    classDef core fill:#fef3c7,stroke:#92400e,stroke-width:2px,color:#7c2d12;

---

Agents break on the ambiguity humans tolerated

For a decade the default in software delivery was code-first: talk through the change, write the code, document selectively, move on. The docs that existed were for onboarding, compliance, or sales; they lagged the code and contradicted it, so nobody trusted them and nobody updated them. That was survivable because the readers were humans, and humans tolerate ambiguity. A new engineer read what existed and then asked whoever had been there longest. The real transfer happened in conversation, and the documents were scaffolding.

Three things ended that. Agents now do the work the ambiguity-tolerant humans used to do, and they tolerate ambiguity badly: a vague spec produces confident, vague code, and an agent that can't tell which decision is current will act on one that was overturned six months ago. The cost of writing has collapsed, because an agent that knows your standards and your prior decisions drafts a usable spec in minutes. And the cost of not writing has risen, because vague input now returns vague output at high speed and high token cost.

The documents stop being scaffolding and become the substrate.

---

RAG was built for chatbots, not agents

From 2023 to 2025, the dominant pattern for connecting an LLM to a team's own knowledge was naive retrieval-augmented generation: embed documents into a vector store, fetch the top-k chunks at query time, stuff them into the prompt. That design was built for chatbots answering questions, not for agents acting on a team's behalf. The shift from chatbot to agent broke its central assumption, that the retrieval system could stay silent about authority.

A cluster of recent work, from different vantage points, is converging on the same point.

Pinecone's announcement of Pinecone Nexus [[1]](#citations) (May 2026) calls it a knowledge engine for agents and reframes the work as compilation rather than retrieval: documents have status, history, and relationships to each other; artefacts are versioned and traceable to their sources; conflicts between documents resolve deterministically rather than by whichever chunk happened to rank higher. That this comes from one of the infrastructure companies that built the previous era of RAG is itself the signal.

The research arrives at the same place from a different angle. Microsoft's GraphRAG [[2]](#citations) (February 2024) built an entity-and-relationship graph over the corpus because vector similarity over disconnected chunks can't traverse the shared attributes that connect information across documents, nor hold a concept that spans a large collection. VectifyAI's PageIndex [[3]](#citations) (September 2025) preserved the intra-document section structure that uniform chunking destroys, organising each document into its natural table-of-contents tree rather than artificial fixed-size pieces. Both are admissions that retrieval has to respect the structure the writing put there, not just the text the chunker can split.

Barnett et al.'s Seven Failure Points When Engineering a RAG System [[4]](#citations) (CAIN 2024) catalogued how these systems break in production: Missing Content, Missed the Top Ranked Documents, Not in Context, Not Extracted, Wrong Format, Incorrect Specificity, and Incomplete. Two of them are directly about whether the right material exists in the corpus and ranks well; several others are downstream consequences of poorly written or poorly structured source documents. RAGChecker [[5]](#citations) (Ru et al., NeurIPS 2024) put numbers behind the picture: across eight RAG systems on ten domains, stronger retrievers drove consistent gains in overall F1 and claim recall, but better retrieval also raised the generator's sensitivity to noise, because the chunks then carried more irrelevant content alongside the relevant facts. Retrieval quality and source quality are tightly coupled. You can't paper over poorly written documents by tuning the retriever harder, and you can't tune the retriever harder without caring about what the chunks contain.

Harrison Chase at LangChain [[6]](#citations) (June 2025) names the corresponding skill: context engineering, building dynamic systems that supply the right information, tools, and format so the model can plausibly accomplish the task. The work of building agentic systems is increasingly curation rather than model selection. The model is a commodity; the context is the differentiator.

The thesis they converge on: a team that wants AI agents to do useful work on its behalf needs a substrate, and the substrate is a written corpus with structure, status, and dependency information. The corpus has to be maintained, the maintenance has to be cheap enough to actually happen, and the writing itself, ahead of the retrieval, is where the leverage is. The retrieval substrate is still RAG-shaped; what's changed is the way of working around it.

This repository is one concrete embodiment of that direction.

---

Corpus, indexer, and server in one tree

Everything the system needs lives in one git repository:

1. The corpus. A docs/ directory of markdown documents (specs, decisions, policies, frameworks, strategies, research notes), each with YAML frontmatter declaring its identity, lifecycle state, and dependency edges. The corpus is the source of truth. If a fact isn't in the corpus, the system doesn't know it. This repository ships document standards and templates, not your corpus content. That part is yours.

1. The indexer. A scripts/ directory plus the shared rag_core/ package: a Python pipeline that parses each document, chunks it while preserving section structure, scrubs fields that must never leave the team's boundary, embeds the chunks with an off-the-shelf model, and upserts into Postgres + pgvector. The indexer runs in CI on every push.

1. The MCP server. A src/ FastAPI application that exposes seven tools over the MCP Streamable HTTP transport. Any MCP-aware client (Claude Code, Claude.ai, or other agents) connects and calls the tools. Humans access the same corpus directly through the doc files on disk and through any docs site built on top of them.

1. Worked examples. At least one end-to-end example per doc type under docs/: 01-str-knowledge-strategy.md (strategy), 02-spec-onboarding-flow.md and 03-spec-search-api.md (spec), 03-dec-tech-stack.md (decision), 03-res-vector-stores.md (research), 05-pol-data-retention.md (policy), and the framework docs at docs/00-fwk-*.md. Plus two CLAUDE.md samples (examples/CLAUDE.example.md for the app a team builds, examples/CLAUDE.corpus.example.md for authoring the corpus) and examples/sample-docs/ (a before/after pair showing the writing rules applied).

The corpus, the indexer, and the server live in the same repository on purpose. Documents, the code that indexes them, and the code that serves them evolve together. One tree, one history, one PR.

flowchart LR
    docs["docs/markdown corpus"]:::store
    ci["GitHub Actionsync-to-rag.yml"]:::ci
    db[("Postgres+ pgvector")]:::store
    mcp["FastAPI MCP serversrc/main.py"]:::svc
    agents(["AgentsClaude Code, Claude.ai"]):::pop
    humans(["Humanson the team"]):::pop

    docs -->|git push| ci
    ci -->|chunk + scrub + embed| db
    mcp -->|SQL queries| db
    mcp -->|7 MCP tools| agents
    humans -->|POST /search + bearer| mcp
    docs -->|read on disk| humans

    classDef store fill:#fef3c7,stroke:#92400e,color:#7c2d12;
    classDef ci fill:#ede9fe,stroke:#5b21b6,color:#4c1d95;
    classDef svc fill:#dcfce7,stroke:#15803d,color:#14532d;
    classDef pop fill:#e0f2fe,stroke:#0369a1,color:#0c4a6e;

---

Backend services and static frontends

Agents only ever call one of these; the rest serve humans or feed the index.

Backend services

• The MCP server. A FastAPI application exposing seven tools over the MCP Streamable HTTP transport. Authenticated via bearer tokens. Deployable to any platform that runs Python.
• A Postgres + pgvector database. Holds the indexed chunks, the dependency graph, the decisions registry, and a query log. Read path for the MCP server, write path for the indexer.
• An embedding provider. The reference template uses OpenAI text-embedding-3-small at 1536 dimensions; the indexer is provider-agnostic and any embedding API works.

Static frontends (auth-gated)

• A docs site. The corpus rendered as a wiki-style human-readable site. Use any static-site generator (Hugo, MkDocs, MdBook, Docusaurus, Nextra) and any static host (Cloudflare Pages, Netlify, Vercel, GitHub Pages). Built on every push from the same corpus the MCP server reads.
• A graph view. An interactive visualization of the dependency graph (nodes coloured by doc type, edges by relation). Built at deploy time from the same database the MCP server queries. Useful for spotting clusters, orphans, and unexpected dependencies.

The MCP server is the only piece agents talk to directly. The two static frontends exist for humans.

---

The standards the corpus enforces

The corpus is the foundation. Everything else exists to serve it.

Each document is a single markdown file with YAML frontmatter at the top. The frontmatter is parsed by the indexer, validated by an audit script on every PR, and used to build the dependency graph. The standards are defined in docs/00-pol-document-standards.md:

• Filename convention. --.md. Area is a two-digit capability-area number; type is one of spec, dec, pol, fwk, str, res.
• Status vocabulary. draft, in-progress, complete, superseded, needs-review. Superseded documents stay searchable but return with a banner pointing to their replacement.
• Relationship fields. depends-on (this document assumes the target is current), feeds-into (this document's output is input to the target), also-touches (areas affected without a hard dependency), supersedes (this document replaces a prior one).
• Chunking contract. Sections under 4000 characters, one concept per section. Oversized sections are flagged by the audit and split.
• Excluded-fields list. A project-defined roster of strings that must never appear in any chunk's content (PII, internal tokens, anything the project marks excluded). Scrubbing is enforced at index time and verified by a fixture-based test on every PR, plus a post-reindex database assertion that fails the workflow if anything leaks.

The standards are machine-enforced on every PR, not aspirational. A PR that breaks them can't merge.

The corpus also supports document renames. When a document is renamed, the old slug is recorded in a rename map; the indexer treats both names as the same document while references inside other docs are migrated. The audit script's --fix mode performs the migration in bulk.

---

Write the way people search

Structure and status make a document findable; the prose inside it decides whether retrieval returns meaning or noise. A document can pass every frontmatter check and still embed as mush. Five rules keep the prose retrievable. Only the size limit is enforced by the audit; the other four are checked while you author and again in review. The full guide with worked examples is [docs/00-fwk-writing-guide.md](docs/00-fwk-writing-guide.md).

• Section summaries after every heading. One line under each ## carrying the terms someone would actually type. A bare "Error states" embeds too vaguely to match anything; the summary makes the section findable by the question it answers.
• Inline cross-references, never bare ones. Don't write "see section 9"; inline what section 9 says. Each chunk is embedded on its own, so a bare pointer carries no searchable meaning.
• One concept per section. A chunk becomes a single vector, an average of the meanings in the text. The more concepts packed in, the blurrier the vector; focused sections embed sharply.
• Concrete searchable terms, not abstract jargon. "16-day grace period where the user keeps full access" beats "configurable access retention window." Same thing, but only one is findable.
• Sections under 4000 characters. Longer sections get split mechanically and lose coherence. The right size is the one that holds a complete thought.

The rules are the floor, not the ceiling. A document can follow all five and still be incoherent, and the research behind them says nothing about whether the decision is well-reasoned or the spec captures the design. That's a separate problem. The harder work, as always, is thinking clearly enough to write down something worth retrieving.

---

The seven MCP tools

| Tool | What it answers | |---|---| | search_docs | "Find me everything in our knowledge about X." Hybrid vector + lexical search across the corpus with filters on status, area, and doc_type. Superseded documents are excluded by default. When explicitly included they return with a [SUPERSEDED, see ] banner so the caller doesn't silently act on a retired claim. | | get_decision | "What is our settled position on X?" Looks up a decision by id, key, or fuzzy text and returns it with a [CURRENT], [SUPERSEDED], or [DRAFT] banner. The banner is load-bearing: it tells the caller whether the answer it's about to act on is still in force. | | get_impact_targets | "If I change X, what else has to change?" Given a decision or a document, returns the affected neighborhood from the dependency graph: every document that declares a dependency on it, its own declared edges, and the supersession chain. | | get_doc_neighborhood | "What does this document touch, and what touches it?" Declared outgoing edges (depends-on, feeds-into, also-touches), inverse incoming edges, the supersession chain, and warnings for dangling references. | | get_doc_outline | "Give me the section tree of this long document so I can navigate it without grepping the body." Headers at levels 1 through 3 with anchors

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mihaelamaciuca
• Source: mihaelamaciuca/knowledge-layer
• License: MIT

Install and usage instructions live in the source repository linked above.