Codebase Skill

Codebase Skill — Semantic Code Indexing & Search

Overview

Turn any code repository into a searchable knowledge base using Tree-sitter parsing, embedding vectors (ModernBERT, zero-config), and pgvector. Instead of loading entire files into context, search surgically for the chunks you need.

When to Use

• Agent needs to understand a large codebase without reading every file
• User asks "where is X implemented?" or "how does Y work?"
• Need to find all functions/classes related to a concept
• Preparing context for code review or modification
• Don't use for: tiny repos (

Creates: PostgreSQL DB + user, pgvector extension, tables, Python venv + deps.

### MCP Configuration

**Hermes Agent** — add to `~/.hermes/config.yaml`:

```yaml
mcp_servers:
  codebase-skill:
    command: /data/codebase-skill/.venv/bin/python3
    args: ["/data/codebase-skill/mcp_server.py"]
    env:
      CODEINDEX_DB_PASSWORD: "${CODEINDEX_DB_PASSWORD}"

Claude Code / Cursor / other MCP clients — add to MCP settings JSON:

{
  "mcpServers": {
    "codebase-skill": {
      "command": "/path/to/codebase-skill/.venv/bin/python3",
      "args": ["/path/to/codebase-skill/mcp_server.py"],
      "env": { "CODEINDEX_DB_PASSWORD": "your_password" }
    }
  }
}

Pi Agent / Codex — same MCP stdio protocol. Adjust command path to the venv python on that host.

Environment Variables

Required: CODEINDEX_DB_PASSWORD. All others have defaults (see .env.example).

The MCP server auto-loads ~/.hermes/.env on startup. When using Hermes, you STILL need the env block in the MCP config because Hermes filters subprocess env vars.

Auto-Reindex (optional cron)

# Manual run
.venv/bin/python3 auto_reindex.py

# Force full reindex of all repos
.venv/bin/python3 auto_reindex.py --force

# Hermes cron (every 4h — already configured)
# See: cronjob action='list' → codebase-auto-reindex

Index First Repo

.venv/bin/python3 cli.py index /path/to/repo

Keeping the Index Fresh

Incremental Reindex

When you re-run index or call reindex on an already-indexed repo:

1. Modified files: compared by mtime vs last_indexed — only changed files are re-parsed and re-embedded
2. Deleted files: chunks for files no longer on disk are automatically purged from the DB
3. Stats include orphan_chunks_purged count

# Via MCP
# mcp_codebase_reindex(repo_path="/data/AIssistant")
# mcp_codebase_reindex(repo_path="/data/AIssistant", force_reindex=true)

# Via CLI
.venv/bin/python3 cli.py index /data/AIssistant   # incremental (fast)
.venv/bin/python3 cli.py index /data/AIssistant --force  # full re-index

Auto-Reindex Cron

A cron job (codebase-auto-reindex) runs auto_reindex.py every 4h, which:

• Iterates all registered projects
• Runs incremental reindex on each
• Purges orphans
• Reports only if changes were made

MCP Tools

| Tool | Parameters | Description | |------|-----------|-------------| | search | query, topk, language, filepattern, repopath, minscore | Semantic search across indexed codebases | | file_context | filepath, focus, topk | File's chunks + related chunks | | stats | repopath? | Indexing statistics | | reindex | repopath, force_reindex | Refresh repo: detect changes + purge deleted files | | list_projects | (none) | List all indexed repositories |

Search Filters

• language — Filter by programming language (python, javascript, etc.)
• file_pattern — SQL LIKE pattern (e.g. %/auth%)
• repo_path — Restrict to one repository
• min_score — Minimum cosine similarity (0-1, default 0.3)

Configuration

Environment variables (or defaults in config.py):

| Variable | Default | Purpose | |----------|---------|---------| | CODEINDEX_DB_HOST | localhost | PostgreSQL host | | CODEINDEX_DB_PORT | 5432 | PostgreSQL port | | CODEINDEX_DB_NAME | codeindex | Database name | | CODEINDEX_DB_USER | codeindex | DB user | | CODEINDEX_DB_PASSWORD | (required) | DB password | | CODEINDEX_EMBED_MODEL | modernbert-embed-base | Embedding model (run python scripts/detect_model.py --write-env to auto-detect) | | CODEINDEX_EMBED_BACKEND | auto (sentence_transformers) | sentence_transformers or ollama (auto-detected from model) | | CODEINDEX_EMBED_API_BASE | http://localhost:11434 | Embedding API base URL (only for ollama backend) | | CODEINDEX_API_HOST | 127.0.0.1 | API server host | | CODEINDEX_API_PORT | 8900 | API server port |

Supported Languages (25)

Python, JavaScript, TypeScript, TSX, JSX, Go, Rust, Java, C, C++, C#, Ruby, PHP, Swift, Kotlin, Scala, Lua, R, Bash, SQL, HTML, CSS, JSON, YAML, TOML, Markdown.

Common Pitfalls

1. pgvector extension missing. Must be created by a superuser: CREATE EXTENSION IF NOT EXISTS vector;. The deploy.sh script does this but requires sudo.

1. tree-sitter version mismatch. Use tree-sitter=1.10. Newer tree-sitter has incompatible API.

1. Embedding model not available. ModernBERT models auto-download from HuggingFace (sentence-transformers backend, zero config). If using the optional Ollama backend: ollama pull nomic-embed-text.

1. Large repos take time to embed. First index of a 10k-file repo may take 10-30 min. Incremental reindex is fast (only changed files).

1. Zero vectors on embedding failure. If the embedding service is down, embeddings become zero vectors. Search still works but returns random results. Check logs.

1. Module-level chunks may capture decorator lines. For Python, @dataclass decorators before classes appear in both module and definition chunks if overlap detection fails.

1. File path in chunks is absolute. Searching with relative paths won't match. Always use absolute paths or LIKE patterns.

1. Hermes MCP env filtering. Hermes filters subprocess env vars. You MUST include the env block in the mcp_servers config to pass CODEINDEX_DB_PASSWORD through, even though mcp_server.py auto-loads ~/.hermes/.env.

1. Embedding service 500 intermittents (Ollama backend only). With the Ollama backend, the /api/embeddings endpoint can return 500 sporadically. The embedder retries 3x with exponential backoff. This does not affect the default ModernBERT backend. Never run two index operations simultaneously.

1. tree-sitter FutureWarning. tree-sitter 0.21.x emits FutureWarning (no impact, compatibility with tree-sitter-languages).

1. Deleted file chunks are stale. Orphan chunks for files removed from disk are purged during reindex. If you never reindex, they persist. The auto-reindex cron handles this automatically.

Auto-Setup for All Agents (cbsetup)

After indexing a repository, run cbsetup to automatically generate instruction files and MCP configuration for every supported coding agent. This ensures any agent working in the repo knows it must use semantic search before reading files.

What It Generates

| File | Agent | Purpose | |------|-------|---------| | AGENTS.md | Pi, Codex, generic agents | Project instructions (search-first protocol) | | CLAUDE.md | Claude Code | Project instructions | | .cursorrules | Cursor (legacy) | Project rules | | .cursor/rules/codebase-search.mdc | Cursor (newer) | Always-apply project rule | | .windsurfrules | Windsurf / Codeium | Project rules | | .clinerules | Cline | Project rules | | .github/copilot-instructions.md | GitHub Copilot | Repo instructions | | .claude/settings.json | Claude Code | MCP server config | | .cursor/mcp.json | Cursor | MCP server config | | .cline/mcp.json | Cline | MCP server config | | .windsurf/mcp.json | Windsurf | MCP server config | | .pi-indexed | All agents | Marker with indexing metadata |

Each instruction file tells the agent:

1. This repo is indexed — use semantic search BEFORE reading files
2. How to call the MCP tools (or CLI if MCP is unavailable)
3. Never blindly grep/cat multiple files when search suffices

Usage

# Full setup — all agents, all MCP configs
cbsetup /path/to/repo

# Or via Python directly
.venv/bin/python3 cbsetup.py /path/to/repo

# Preview without writing
.venv/bin/python3 cbsetup.py /path/to/repo --dry-run

# Only instruction files, no MCP configs
.venv/bin/python3 cbsetup.py /path/to/repo --instructions-only

# Only MCP configs, no instruction files
.venv/bin/python3 cbsetup.py /path/to/repo --mcp-only

# Specific agents only
.venv/bin/python3 cbsetup.py /path/to/repo --agents claude_md cursorrules --mcp claude cursor

Idempotent & Safe

• Existing files are appended to, never overwritten
• A `` marker allows automatic updates without losing existing content
• Re-running cbsetup updates the section in place
• The .pi-indexed marker file tracks indexing metadata

Typical Workflow

# 1. Index your repo
.venv/bin/python3 cli.py index /data/myproject

# 2. Generate agent files
.venv/bin/python3 cbsetup.py /data/myproject

# 3. Commit (optional — share the instructions with your team)
cd /data/myproject && git add AGENTS.md CLAUDE.md .cursorrules .pi-indexed && git commit -m "Add codebase-skill search protocol"

Verification Checklist

• [ ] pgvector extension installed: SELECT * FROM pg_extension WHERE extname = 'vector';
• [ ] Embedding service running: curl $CODEINDEX_EMBED_API_BASE/api/tags
• [ ] Embedding model configured: python scripts/detect_model.py (ModernBERT auto-downloads, no external service needed)
• [ ] Index works: .venv/bin/python3 cli.py index /some/repo
• [ ] Search returns results: .venv/bin/python3 cli.py search "test"
• [ ] MCP tools appear in agent: check tool list
• [ ] Auto-reindex cron active: cronjob action='list'
• [ ] Orphan purge works: delete a file, reindex, check stats for orphan_chunks_purged > 0

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: dr34dl10n
• Source: dr34dl10n/Codebase-Agent-Skill
• License: MIT

Install and usage instructions live in the source repository linked above.