# Vecgrep > grep using meaning and similarity instead of keywords - local/self-hosted, multimodal, and runnable on any user-defined corpus - **Type:** MCP server - **Install:** `agentstack add mcp-jeffbai996-vecgrep` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [jeffbai996](https://agentstack.voostack.com/s/jeffbai996) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [jeffbai996](https://github.com/jeffbai996) - **Source:** https://github.com/jeffbai996/vecgrep ## Install ```sh agentstack add mcp-jeffbai996-vecgrep ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # vecgrep > grep for meaning, not keywords **v1.0.0** · stable CLI + HTTP API surface · see [CHANGELOG.md](CHANGELOG.md) for release history `vecgrep` is a local-first semantic search engine for any corpus you throw at it. Drop in documents — text, markdown, PDFs, URLs — and search by concept instead of exact words. Runs on your machine, no cloud roundtrip required. *The web UI: index a corpus from the sidebar, pick a mode (hybrid / vector / bm25), search. Each result shows confidence (high / soft / weak), which retriever found it (V semantic, K keyword, VK both), and the matched chunk in context. Tucked into the bottom of the page is a primer explaining how vecgrep finds things — vector vs BM25 vs hybrid, what the % can and can't tell you.* ``` $ vecgrep search "what did we decide about rate hikes" --corpus notes [1] 92.4% notes papers/2026-q1-review.md ... the committee discussed the path forward and the committee opted to hold rates steady through Q2 citing softening labor data ... [2] 88.1% notes notes/meeting-mar-12.md ... after some back and forth the rate path discussion landed on no change for the rest of the year ... ``` ## Why Keyword `grep` is brittle. You search for `rate hikes`, you miss the doc that says `monetary tightening`. Embeddings fix that — semantically equivalent text scores high regardless of wording. `vecgrep` wraps that into a tool that feels like a CLI utility, not a research notebook. No cloud account, no Docker, no Postgres extension, no ceremony. Index a folder, run a query, get ranked chunks back with surrounding context. The closest equivalents — `txtai`, `chroma`, `LlamaIndex` — are libraries you import. `vecgrep` is a binary you run. ## Contents - [Status](#status) - [Install](#install) - [Quickstart](#quickstart) - [What it indexes](#what-it-indexes) - [How it works](#how-it-works) - [Storage layout](#storage-layout) - [Configuration](#configuration) - [Web UI](#web-ui) - [MCP server](#mcp-server) - [Roadmap](#roadmap) - [Why not just use X?](#why-not-just-use-x) - [Contributing](#contributing) - [Tests](#tests) - [License](#license) ## Status Alpha (`v0.7.x`). What's shipped: - **Retrieval** — hybrid (BM25 + vector + RRF, BM25-weighted by default), cross-encoder rerank, optional per-corpus recency decay. - **Serving** — MCP server (stdio + streamable HTTP), named corpora, optional bearer-token auth. Corpora embedded with *different* models can be served side by side — each queries with its own model. - **Ops** — incremental indexing, file watcher, embedding cache, model/backend migration, `vecgrep status` diagnostic, hermetic pytest suite. - **UI** — web UI with confidence-tier coloring + V/K/VK match badges. - **Adapters** — plaintext, markdown, PDF, URLs, Discord JSONL, Claude export, ChatGPT export. The public API (HTTP + CLI flags) is unstable until v1.0 — expect breaking changes within v0.x. ## Install ```bash pip install vecgrep # base — Ollama embedding, hybrid search pip install "vecgrep[openai]" # also: OpenAI embedding fallback pip install "vecgrep[rerank]" # also: cross-encoder reranking (~hundreds of MB, torch) pip install "vecgrep[watch]" # also: file watcher for `vecgrep watch` pip install "vecgrep[mcp]" # also: MCP server for Claude Desktop / Cursor ``` You also need [Ollama](https://ollama.com) running locally for the default embedding backend: ```bash ollama pull bge-m3 ollama serve ``` If you'd rather use OpenAI, install with `[openai]`, set `OPENAI_API_KEY`, and `vecgrep` will fall back to `text-embedding-3-small` automatically when Ollama isn't reachable. ## Quickstart ```bash # Index a folder (mixed file types are fine; incremental — unchanged files skipped) vecgrep index ./my-docs --corpus papers # Re-embed everything regardless of content hash vecgrep index ./my-docs --corpus papers --force # Index a URL — vecgrep fetches and strips boilerplate vecgrep index https://example.com/article --corpus web # Watch a folder and re-index on change vecgrep watch ./my-docs --corpus papers # Hybrid search (default — BM25 + vector fused via RRF), top 10 vecgrep search "missile guidance systems" --corpus papers --top 10 # Pure-vector or pure-BM25 if you want to A/B vecgrep search "rate hikes" --mode vector vecgrep search "FOMC" --mode bm25 # Watch mode — re-run the same query at a fixed interval, print a diff # of source IDs. Pair with `vecgrep watch` while ingesting to confirm # new chunks are showing up where you expect. vecgrep search "rate hikes" --watch --interval 5 # Cross-encoder reranking on the candidate pool — slower, more accurate vecgrep search "what did we decide about rates" --rerank # Filter results — repeatable, all ANDed vecgrep search "rate hikes" --filter "source:*2026*.md" --filter "corpus:papers" # Show the score decomposition for each hit (cosine, BM25, RRF, rerank) vecgrep search "rate hikes" --explain # Inspect / clear the embedding cache vecgrep cache stats vecgrep cache clear --identity ollama:bge-m3 # Re-embed a corpus to a different backend / model vecgrep corpora migrate papers --to-backend openai --to-model text-embedding-3-small # Recency decay: down-rank stale chunks. Half-life in days; a hit one half-life # old ranks as if half as relevant. Good for chat logs / journals; leave off for # durable reference. No re-index needed — applied at search time from each # chunk's parsed date. vecgrep corpora decay chatlogs --half-life 14 vecgrep corpora decay chatlogs --off # Search across every corpus you have vecgrep search "rate hikes" # Don't persist — index, query, throw away vecgrep index ./scratch --corpus tmp --ephemeral # Move a corpus between machines vecgrep corpora export papers --out papers.tar.gz vecgrep corpora import papers.tar.gz --rename papers-from-laptop # Manage corpora vecgrep corpora list vecgrep corpora delete papers # One-shot diagnostic: daemon, auth, per-corpus chunk counts, last-update age vecgrep status vecgrep status --json # for scripting / monitoring # Launch the web UI on http://127.0.0.1:8765 vecgrep serve ``` ## What it indexes | Adapter | Inputs | |---|---| | `plaintext` | `.txt`, `.log`, `.csv`, `.tsv`, `.rst`, `.org`, `.tex`, `.srt`, extensionless | | `markdown` | `.md`, `.markdown`, `.mdx` | | `pdf` | `.pdf` (text layer only — scanned PDFs need OCR first) | | `url` | `http://`, `https://` — strips ``, ``, etc., keeps prose | | `discord_jsonl` | `.jsonl` files where each line is a Discord message (DiscordChatExporter format or any one-message-per-line JSONL with `author` + `content`) | | `claude_export` | Anthropic data export `conversations.json` — one Document per conversation | | `chatgpt_export` | OpenAI data export `conversations.json` — one Document per conversation, follows main thread | Pointing `vecgrep index` at a directory walks it recursively and dispatches each file to the matching adapter. Unrecognized files are skipped silently — no errors. ## How it works ``` ┌──▶ vector (qdrant) ──┐ docs ──▶ adapters ──▶ chunkers ──┤ ├──▶ RRF ──▶ [decay] ──▶ dedup ──▶ [rerank] ──▶ top-k └──▶ bm25 (inverted) ──┘ ``` - **Adapters** convert source formats to text. They run once per source; chunkers handle slicing. - **Chunkers** slice text into overlapping windows. `SentenceWindowChunker` is the default — 3 sentences with 1-sentence overlap. `FixedTokenChunker` (tiktoken-backed) is the alternate for code, logs, anything where sentence boundaries are noisy. - **Embed backends** are pluggable. Each corpus pins the model it was built with; corpora on different models can be served at once, each querying with its own. - Default: Ollama `bge-m3` (1024-dim) — strong on paraphrase-heavy and multilingual queries. - Alternates: any other Ollama model via `VECGREP_EMBED_MODEL` (e.g. `nomic-embed-text`, 768-dim, lighter/faster; `mxbai-embed-large`, 1024-dim). - Fallback: OpenAI `text-embedding-3-small` (1536-dim) takes over when Ollama is unreachable and `OPENAI_API_KEY` is set. - **Qdrant** runs in embedded mode (no server, no Docker) at `~/.vecgrep/qdrant/`. Each named corpus is its own collection. - **BM25** index runs alongside Qdrant, persisted as a pickle per corpus. Tokenizer splits identifiers (`sharpe_ratio` → `sharpe`, `ratio`) so code search isn't blind to underscore- or camelCase-style naming. - **Hybrid retrieval** is the default. Each retriever returns its top 50 candidates; their ranks are fused via Reciprocal Rank Fusion (`score = Σ w / (60+rank)`). BM25's weight is `1.5` by default — high enough to float exact-keyword hits over the vector noise floor on short queries, low enough to leave long conceptual queries vector-dominated (override with `VECGREP_BM25_WEIGHT`). Pure-vector or pure-BM25 are available with `--mode vector` / `--mode bm25`. - **Recency decay** (optional, per corpus). Set a half-life in days with `vecgrep corpora decay --half-life N` and a hit's fused score is multiplied by `0.5 ** (age_days / half_life)` — a chunk one half-life old ranks as if half as relevant. Applied *before* the top-k cut, so a fresh chunk can rescue itself above a stale one, and lexical closeness alone can't float a stale chunk to the top. Off by default; undated chunks are never penalized. The date comes from a `doc_timestamp` parsed at index time (frontmatter `date:`/`Saved:` lines, a `YYYY-MM-DD` filename, then file mtime). Tune fast for chat/journal corpora, off for durable reference. - **Dedup.** The sentence-window chunker emits overlapping windows, so one passage can surface as several near-identical hits. vecgrep collapses same-source hits whose character ranges overlap before truncating to top-k, keeping the strongest, so the result list isn't padded with the same text at three ranks. - **Confidence display.** The displayed `%` is a calibrated relevance estimate, not a raw score; ranking always uses the underlying fused score. - With reranking on: comes straight from the cross-encoder (the cleanest `P(relevant)` proxy). - Otherwise: a sigmoid over cosine, **calibrated per embedding model** — different models put their noise floor and signal range in different places, so the same `%` means the same thing across corpora. - BM25-only hits (which have no cosine) are rescaled rank-relative, so a real keyword match doesn't read as the `~1.6%` raw-RRF noise floor. - The web UI surfaces V/K/VK badges and tier colors plus a "how search works" panel. - **Cross-encoder reranker** (`--rerank`, off by default) rescores the candidate pool with `BAAI/bge-reranker-base`. Lazy-loaded — the heavy `torch` import only happens when you ask for it. It's a real quality win on hard, paraphrase-heavy queries where plain hybrid whiffs, but it adds meaningful latency (order ~100ms+ for a small candidate pool) and on easy literal queries it can be a wash or worse — so it's opt-in, not the default. Reach for it when a hybrid search returns near-misses for something you know is in the corpus. Each corpus pins the embedding backend, model, and dimension at index time and refuses to mix models *within* a single corpus — but the engine resolves each corpus's query embedding from its own pinned model, so multiple corpora on different models coexist fine. To change the model of an existing corpus, `vecgrep corpora migrate` it (or recreate it). ## Storage layout ``` ~/.vecgrep/ ├── qdrant/ # vector store, one collection per corpus ├── bm25/ # BM25 inverted index, one pickle per corpus ├── corpora.json # named-corpus metadata └── config.json # optional, env vars override ``` `--ephemeral` mode (CLI flag or UI toggle) keeps everything in memory and skips both files and the vector store on disk. Useful for a one-shot grep over a directory you don't want polluting your indexed corpora. ## Configuration `vecgrep` reads from `~/.vecgrep/config.json` and environment variables, in that order. Env vars win. | Variable | Default | Notes | |---|---|---| | `VECGREP_HOME` | `~/.vecgrep` | Storage root | | `VECGREP_OLLAMA_URL` | `http://localhost:11434` | Ollama endpoint | | `VECGREP_EMBED_MODEL` | `bge-m3` | Ollama model | | `VECGREP_OPENAI_EMBED_MODEL` | `text-embedding-3-small` | OpenAI model | | `OPENAI_API_KEY` | unset | If set, used as fallback when Ollama is down | | `VECGREP_API_HOST` | `127.0.0.1` | API bind host | | `VECGREP_API_PORT` | `8765` | API port | | `VECGREP_API_TOKEN` | unset | If set, `/api/*` requires `Authorization: Bearer ` (health stays public) | | `VECGREP_TOP_K` | `5` | Default `--top` value | | `VECGREP_ALIASES_FILE` | `$VECGREP_HOME/aliases.json` | Entity alias map (personal data — keep it out of any repo). See `docs/aliases.example.json`. Missing = no expansion. | | `VECGREP_OAUTH_ENABLED` | unset | `1` enables OAuth 2.1 on `/mcp` (embedded auth server: /authorize, /token, /.well-known). | | `VECGREP_OAUTH_ISSUER_URL` | unset | Public base URL the MCP endpoint is reachable at. Required when OAuth is on. | | `VECGREP_BM25_WEIGHT` | `1.5` | Weight on BM25 contribution to RRF fusion. >1 boosts literal-keyword matches over semantic noise on short queries. Set to `1.0` for pure RRF, higher for keyword-leaning ranking. | | `VECGREP_BM25_COVERAGE_MODE` | `penalty` | How BM25 treats docs that match only some query tokens. `penalty` keeps them but demotes the score by `(matched/total)²`; `filter` drops anything below a coverage threshold (higher precision, but can zero out the BM25 half of a multi-token query). | | `VECGREP_EMBED_CACHE_MAX_ROWS` | `50000` | Row cap on the sqlite embedding cache (~1 GB at a 1024-dim model). Oldest entries are evicted first once over the cap. Set ` ` / `GET /api/chunk/...` / MCP `get_chunk`. - **Hard filters** — the caller passes explicit constraints; vecgrep never guesses intent: `--filter date:2026-01-15` (or `date:today`), `after:` (or relative: `after:7d`, `after:24h`, `after:2w`), `before:`, `channel:`, `source_path:`, `speaker:` (who said it — chunk-level, ` [bot]` suffix optional), `bot:true|false`, `has:code|table|link` (content shape). A leading `-` inverts any filter (`-corpus:scratch` excludes). Recognized filters fail closed (a typo'd date reads as zero results, not silently ignored) — in either polarity. - **Insight tools (v1.2)** — `related ` (query-by-example: more evidence like this chunk, no re-embedding), `compare` (one query, two time windows, source-level delta — "how did we talk about X then vs now"), `stats ` (counts, date coverage, gap days — a broken archiver shows up here), `summarize ` (speaker tally + span + sampled chunks for rollups; sampling always explicit). All four: CLI + `/api/*` + MCP. - **Timeline mode** — "what happened?" gets an ordered event sequence, not ranked chunks: `vecgrep timeline "query"` / `POST /api/timeline` / MCP `timeline`. Contiguous chronological slices grouped by source file, speakers + timestamps preserved. - **Incident object** — `service.incident()` / MCP `incident`: one structured answer (title, sources, participants, time range, primary timeline, related context separated, confidence). - **Alias expansion** — one entity, many surface forms (nickname ⇄ handle ⇄ another language), config-driven via an out-of-repo map (`VECGREP_ALIASES_FILE`); a query naming one form finds evidence written under the others. - **Clear scores** — every result carries `relevance_pct`, a qualitative `relevance_label` (exact / strong / related / weak), raw component scores, and a precise `anchor` citation (`path#L12-L24`). ## Web UI `vecgrep serve` boots the FastAPI server and serves a single-page React UI from the same port. Every action the UI supports has a CLI equivalent. Controls: - Index forms with a built-in dropdown explainer for source types. - Corpus list with delete. - Search bar with top-k slider, mode toggle (hybrid/vector/bm25), and reranker checkbox. … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [jeffbai996](https://github.com/jeffbai996) - **Source:** [jeffbai996/vecgrep](https://github.com/jeffbai996/vecgrep) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** yes - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-jeffbai996-vecgrep - Seller: https://agentstack.voostack.com/s/jeffbai996 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.