# Grounding Ai

> Local-first document corpus pipeline for grounded AI agents

- **Type:** MCP server
- **Install:** `agentstack add mcp-andyliszewski-grounding-ai`
- **Verified:** Pending review
- **Seller:** [andyliszewski](https://agentstack.voostack.com/s/andyliszewski)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [andyliszewski](https://github.com/andyliszewski)
- **Source:** https://github.com/andyliszewski/grounding-ai
- **Website:** https://pypi.org/project/grounding-ai/

## Install

```sh
agentstack add mcp-andyliszewski-grounding-ai
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Grounding

[](https://github.com/andyliszewski/grounding-ai/actions/workflows/ci.yml)
[](https://pypi.org/project/grounding-ai/)
[](https://pypi.org/project/grounding-ai/)
[](LICENSE)

**Local-first document corpus pipeline for grounded AI agents.**

Grounding converts PDF, EPUB, DOCX, and Markdown documents into a structured, searchable corpus with per-agent embedding indexes. Drop documents into staging, get chunked Markdown with provenance hashing, FAISS vector indexes, and agent-filtered search -- all running locally, no cloud APIs required.

## What It Does

```
Documents (PDF/EPUB/DOCX/MD)
    |
    v
[ Parse ] ── Unstructured / Marker
    |
    v
[ Chunk ] ── LangChain text splitters + YAML front matter
    |
    v
[ Hash ]  ── SHA-1 + SHA-256 + BLAKE3 provenance
    |
    v
[ Index ] ── FAISS embeddings, filtered per agent
    |
    v
[ Query ] ── Local RAG via Ollama with agentic tool calling
```

## Key Features

- **Deterministic pipeline** -- same inputs produce byte-identical outputs
- **Content provenance** -- SHA-1, SHA-256, and BLAKE3 hashing on every document and chunk
- **Agent-based corpus partitioning** -- YAML-defined agents filter the corpus by collection tags, each with their own FAISS embedding index
- **Persona system** -- agents have configurable communication styles, expertise areas, and greeting messages
- **Staging watcher** -- drop files into a folder, auto-ingest with embedding updates
- **Multi-machine ready** -- optional Syncthing-based architecture for dedicated ingestion servers
- **Fully local** -- no cloud APIs, no telemetry, your documents stay on your machine
- **Agentic RAG** -- local LLMs autonomously decide when to search the corpus via tool calling

## Why Grounding? (vs. LlamaIndex / Haystack / txtai)

Grounding is an opinionated pipeline, not a framework:

- **Per-agent FAISS partitioning as a first-class primitive.** Each agent YAML defines a corpus slice and gets its own index -- no single monolithic index where unrelated domains compete for top-k. Other frameworks treat this as a filter you bolt on at query time; here it's the core data model.
- **Determinism and provenance for citable output.** Every chunk carries SHA-1 / SHA-256 / BLAKE3 hashes, page ranges, and section headings; same inputs produce byte-identical chunks. Built for agents that must cite sources.
- **Local-first by construction.** No cloud APIs, no telemetry. Runs against Ollama or any OpenAI-compatible local server. Cloud isn't a default path -- it's not a path at all.
- **Opinionated fixed pipeline** (parse → chunk → hash → index → query). Less surface area than a framework, less to configure, less to break.

Use LlamaIndex / Haystack if you want a framework to assemble custom retrieval flows. Use grounding-ai if you want a citation-grade local RAG pipeline with per-agent separation, working today.

## Quick Start

### Install from PyPI

```bash
python3 -m venv venv   # Python 3.10-3.13 supported
source venv/bin/activate
pip install grounding-ai
```

Then grab the example configs and agents from the repo:

```bash
curl -O https://raw.githubusercontent.com/andyliszewski/grounding-ai/main/config.example.yaml
curl -O https://raw.githubusercontent.com/andyliszewski/grounding-ai/main/.mcp.example.json
mkdir -p agents && cd agents && \
  curl -O https://raw.githubusercontent.com/andyliszewski/grounding-ai/main/agents/examples/scientist.yaml && \
  cd ..
cp config.example.yaml config.yaml
cp .mcp.example.json .mcp.json
```

### Install from source (for development)

```bash
git clone https://github.com/andyliszewski/grounding-ai.git
cd grounding-ai
python3 -m venv venv
source venv/bin/activate
pip install -e .
cp config.example.yaml config.yaml
cp .mcp.example.json .mcp.json
cp agents/examples/*.yaml agents/
```

### First run (end-to-end)

```bash
# 1. Ingest some documents
grounding ./my-documents ./corpus --collections science

# 2. Generate embeddings for the scientist agent
grounding embeddings --agent scientist --corpus ./corpus

# 3. Query with a local LLM (requires Ollama running)
python scripts/local_rag.py --agent scientist -A
```

A typical session looks like this:

```
$ python scripts/local_rag.py --agent scientist -A
🔬 Scientist agent ready (3,142 chunks indexed across 8 collections)

> What does Kuhn mean by a paradigm shift?

[searching corpus... 5 chunks retrieved]

A paradigm shift, in Kuhn's framing, is a discontinuous change in the
fundamental assumptions of a scientific community [Source: Kuhn, The
Structure of Scientific Revolutions, corpus]. It happens when accumulated
anomalies can no longer be explained within the existing paradigm and a
new framework displaces the old one — not through gradual refinement but
through a gestalt-like reorientation.

[Derived] The process is social as much as epistemic: Kuhn emphasizes that
competing paradigms are often incommensurable, meaning proponents of each
literally see the world differently.
```

## Agent System

Agents are YAML files that define a persona and a corpus filter:

```yaml
name: scientist
description: Scientific research agent

persona:
  icon: "🔬"
  style: |
    You communicate like a rigorous scientist: analytical,
    evidence-based, and methodical.
  expertise:
    - Scientific method and experimental design
    - Biology and biochemistry
    - Physics fundamentals
  greeting: |
    I'm your scientific advisor. What would you like to investigate?

corpus_filter:
  collections:
    - science
    - biology
    - chemistry
    - physics
```

Each agent gets its own FAISS embedding index containing only documents matching its collections. See `agents/examples/` for starter templates.

### Creating Your Own Agents

1. **Define the agent.** Create a YAML file in `agents/`:

   ```yaml
   # agents/my-agent.yaml
   name: my-agent
   description: What this agent knows about

   persona:
     icon: "🎯"
     style: |
       How you want the agent to communicate.
     expertise:
       - Domain area 1
       - Domain area 2
     greeting: |
       Message shown when the agent activates.

   corpus_filter:
     collections:
       - collection-tag-1
       - collection-tag-2
   ```

2. **Ingest documents with matching collection tags.** Collections are kebab-case labels you assign when ingesting:

   ```bash
   grounding ./physics-textbooks ./corpus --collections physics
   grounding ./biology-papers ./corpus --collections biology,science
   ```

   A document can belong to multiple collections (comma-separated). An agent sees all documents whose collection tags overlap with its `corpus_filter.collections` list.

3. **Generate embeddings** for the agent:

   ```bash
   grounding embeddings --agent my-agent --corpus ./corpus
   ```

   This builds a FAISS index at `embeddings/my-agent/` containing only chunks from documents matching the agent's collection filter.

4. **Query** the agent:

   ```bash
   python scripts/local_rag.py --agent my-agent -A
   ```

### Where Do My Agent Files Live?

By default, `agents/*.yaml` is **gitignored** (only `agents/examples/` is tracked). This means YAMLs you create in `agents/` won't show up in `git status` and won't be committed to your fork. There are three common workflows depending on how you want to manage your agents:

#### Workflow A: Local-only (simplest, no version control)

Just create YAMLs in `agents/` and use them. Nothing extra to manage.

```bash
cp agents/examples/scientist.yaml agents/my-physicist.yaml
# Edit, then use immediately
grounding embeddings --agent my-physicist --corpus ./corpus
```

**Good for:** Trying things out, single machine, agents you don't need to back up.

#### Workflow B: Separate private agents repo (recommended for serious use)

Create your own private repo for agent definitions and point `AGENTS_DIR` at it. This is how the maintainer runs grounding -- agents are version-controlled and sync between machines via git.

```bash
# Create a private repo with this structure:
#   my-agents/
#   ├── agents/
#   │   ├── physicist.yaml
#   │   └── biologist.yaml
#   └── commands/         # Optional: Claude Code slash commands

# Clone it alongside grounding-ai
git clone git@github.com:youruser/my-agents.git ~/my-agents

# Point grounding at it
grounding embeddings --agent physicist --corpus ./corpus --agents-dir ~/my-agents/agents

# Or set the environment variable for the staging watcher
export AGENTS_DIR=~/my-agents/agents
```

**Good for:** Multi-machine setups, version-controlled agent definitions, keeping personal agents private while contributing back to grounding-ai.

#### Workflow C: Fork grounding-ai

Fork the repo and remove `agents/*.yaml` from `.gitignore`. Your agents become part of your fork.

```bash
# After forking
sed -i '' '/agents\/\*\.yaml/d' .gitignore   # remove the gitignore line
git add agents/ .gitignore
git commit -m "track personal agent definitions"
```

**Good for:** Single-repo workflow, public agent libraries, contributing agent templates back upstream.

### Organizing Collections

Collections are free-form tags -- there's no predefined list. Choose whatever makes sense for your domain:

```
staging/
├── physics/           # Collection: physics
├── biology/           # Collection: biology
├── game-theory/       # Collection: game-theory
└── machine-learning/  # Collection: machine-learning
```

One agent can span many collections (a "scientist" agent might include physics, biology, and chemistry). Multiple agents can share the same collections. The agent YAML is the only thing that defines which slices of the corpus each agent can search.

## Project Structure

```
grounding-ai/
├── grounding/              # Python package (the pipeline)
├── scripts/                # Watcher, local RAG, utilities
├── mcp_servers/            # MCP corpus search server
├── agents/
│   └── examples/           # Starter agent definitions
├── tests/                  # Test suite
├── config.example.yaml     # Configuration template
├── .mcp.example.json       # MCP server config template
└── staging/                # Drop documents here for ingestion
```

## Configuration

Copy `config.example.yaml` to `config.yaml` and adjust paths:

```yaml
paths:
  corpus: ./corpus
  embeddings: ./embeddings
  staging: ./staging
  agents: ./agents
  originals: ./originals
```

**Single machine (default):** All paths are relative, everything runs locally.

**Multi-machine:** Point paths at Syncthing-shared directories. A dedicated server runs the staging watcher and generates embeddings; workstations sync the corpus and query it. See `docs/multi-machine.md`.

## CLI Reference

```bash
# Ingest documents
grounding ./input-dir ./output-dir [options]
  --chunk-size 1200        # Characters per chunk (default: 1200)
  --chunk-overlap 150      # Overlap between chunks (default: 150)
  --parser marker          # Parser: unstructured or marker
  --ocr auto               # OCR: auto, on, or off
  --collections sci,math   # Collection tags (comma-separated)
  --dry-run                # Preview without writing
  --verbose                # Debug logging

# Agent management
grounding agents list --agents-dir ./agents
grounding agents show scientist --agents-dir ./agents

# Embedding generation
grounding embeddings --agent scientist --corpus ./corpus
grounding embeddings --agent scientist --corpus ./corpus --incremental
grounding embeddings --agent scientist --corpus ./corpus --check
```

## Staging Watcher (Auto-Ingest)

For continuous ingestion, run the staging watcher: drop a document into your staging folder and it's automatically parsed, chunked, hashed, moved to `originals/`, and (optionally) added to affected agents' embedding indexes.

### Single-machine setup

**Requirements:**
- Linux: `inotify-tools` (`sudo apt install inotify-tools`)
- macOS: `fswatch` (`brew install fswatch`) — the shipped script uses `inotifywait`; macOS users typically wrap it with `fswatch` or run the watcher inside a Linux VM/container

**Run manually:**

```bash
export STAGING_DIR=./staging
export CORPUS_DIR=./corpus
export ORIGINALS_DIR=./originals
export SKIPPED_DIR=./skipped
export AGENTS_DIR=./agents
export EMBEDDINGS_DIR=./embeddings
export AUTO_EMBEDDINGS=true
export LOG_FILE=./watcher.log

./scripts/staging-watcher.sh
```

Then drop documents into a collection subfolder:

```bash
mkdir -p staging/science
cp ~/Downloads/paper.pdf staging/science/
# Watcher logs show: parsing → chunking → hashing → embeddings update
```

**Processing rules:**

| Source location | Collection tag | Destination after processing |
|---|---|---|
| `staging/science/paper.pdf` | `science` | `corpus//`, original → `originals/science/` |
| `staging/biology/book.epub` | `biology` | `corpus//`, original → `originals/biology/` |
| Scanned PDF (no text yield) | — | moved to `skipped//` |

When `AUTO_EMBEDDINGS=true`, every ingested document triggers incremental embedding updates for each agent whose `corpus_filter.collections` matches the document's collection.

### Run as a systemd service (Linux)

```bash
# Copy the sample unit file and edit the paths
cp scripts/grounding-watcher.service.example ~/.config/systemd/user/grounding-watcher.service
# Edit Environment= lines to point at your directories

systemctl --user daemon-reload
systemctl --user enable --now grounding-watcher
journalctl --user -u grounding-watcher -f    # follow logs
```

### Multi-machine setup

Point the watcher's paths at Syncthing-shared directories and run it on a dedicated ingestion server. Workstations sync corpus and embeddings and never run the watcher themselves. See [`docs/multi-machine.md`](docs/multi-machine.md).

## Querying Your Corpus

Once you have an agent with embeddings, there are two ways to query its corpus. Both run entirely locally -- no cloud APIs, no telemetry.

### Option 1: Local LLM via Ollama (recommended)

Ground a local LLM in your corpus using `scripts/local_rag.py`. The script loads your agent's persona, retrieves relevant chunks from its FAISS index, and feeds them to the LLM as context.

```bash
# Install Ollama and pull a model
brew install ollama          # macOS (or curl ... | sh on Linux)
ollama pull qwen2.5:14b      # Recommended for 32GB+ RAM
ollama serve

# Query your agent
python scripts/local_rag.py --agent scientist -A
```

The `-A` flag enables **agentic mode**, where the LLM autonomously decides when to search the corpus via tool calling. It can also perform multi-step reasoning, web searches, and file operations.

```bash
# Interactive REPL session
python scripts/local_rag.py --agent scientist -A

# Single query
python scripts/local_rag.py --agent scientist -A \
  --query "What are the thermodynamic limits of a Carnot engine?"

# With verbose output (see tool calls)
python scripts/local_rag.py --agent scientist -A -v
```

See [`how-to-local-agent.md`](how-to-local-agent.md) for the full guide -- model recommendations, REPL commands, troubleshooting, and tool reference.

### Option 2: MCP Server (for Claude Code, Cursor, etc.)

Grounding ships with an MCP (Model Context Protocol) server that exposes corpus search to any MCP-compatible client.

```bash
# 1. Install the MCP runtime into your venv (not a default dependency)
./venv/bin/pip install mcp

# 2. Copy the example and edit paths to match your setup
cp .mcp.example.json .mcp.json
```

Edit the three env vars in `.mcp.json`:

| Variable | Points at |
|----------|-----------|
| `CORPUS_DIR` | Root containing `_index.json` and `/chunks/` (e.g. `./corpus` or `~/Documents/Corpora/corpus`) |
| `EMBEDDINGS_DIR` | Root containing `/_embeddings.faiss` and `_chunk_map.json` |
| `AGENTS_DIR` | Directory of agent YAML files. Can point outside the repo if you keep agents in a separate repo. |

`CORPUS_DIR` must be the same root used when embeddings were generated -- chunk paths in the FAISS chunk map are stored relative to it.

`.mcp.json` is gitignored by default so machine-specific paths don't leak into commits. Restart your MCP client (e.g., C

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [andyliszewski](https://github.com/andyliszewski)
- **Source:** [andyliszewski/grounding-ai](https://github.com/andyliszewski/grounding-ai)
- **License:** MIT
- **Homepage:** https://pypi.org/project/grounding-ai/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: flagged — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-andyliszewski-grounding-ai
- Seller: https://agentstack.voostack.com/s/andyliszewski
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.