Semantic Search

Semantic Search Skill

Orchestrator for Semantic Code Intelligence via Agent Delegation

This skill orchestrates two specialized agents for semantic search operations. It provides bash scripts that import Python modules from the claude-context-local library (NOT an MCP server - no server process runs, just Python imports via PYTHONPATH). Unlike traditional text-based search (Grep) or pattern matching (Glob), semantic search understands the meaning of content, finding functionally similar text even when using different wording, variable names, or patterns.

The skill uses the library's venv Python interpreter to import merkle, chunking, and embedding modules, enabling semantic search, indexing, and similarity finding across any text content (code, docs, markdown, configs).

🎬 Orchestration Instructions

When this skill is active, you MUST spawn the appropriate agent via Task tool.

This skill uses a 2-agent architecture for token optimization:

• semantic-search-reader: Handles READ operations (search, find-similar, list-projects)
• semantic-search-indexer: Handles WRITE operations (index, incremental-reindex, status)

Decision Logic: Which Agent to Spawn?

| User Request Contains | Operation Type | Agent to Spawn | |----------------------|----------------|----------------| | "find X", "search for Y", "where is Z" | search | semantic-search-reader | | "find similar to...", "similar chunks" | find-similar | semantic-search-reader | | "what projects", "list indexed", "show projects" | list-projects | semantic-search-reader | | "index this", "create index", "full reindex" | index | semantic-search-indexer | | "incremental reindex", "auto reindex", "update index" | incremental-reindex | semantic-search-indexer | | "check index", "index status", "is it indexed" | status | semantic-search-indexer |

Agent Spawn Examples

Example 1: Search Operation (semantic-search-reader)

Task(
    subagent_type="semantic-search-reader",
    description="Search project semantically",
    prompt="""You are the semantic-search-reader agent.

Operation: search
Query: "user authentication logic"
K: 10
Project: /path/to/project

Execute the search operation using scripts/search and return interpreted results with explanations."""
)

Example 2: Index Operation (semantic-search-indexer)

Task(
    subagent_type="semantic-search-indexer",
    description="Index project for semantic search",
    prompt="""You are the semantic-search-indexer agent.

Operation: index
Directory: /path/to/project
Full: true

Execute the indexing operation using scripts/incremental-reindex and return interpreted results with statistics."""
)

Example 3: Incremental Reindex Operation (semantic-search-indexer)

Task(
    subagent_type="semantic-search-indexer",
    description="Incremental reindex with change detection",
    prompt="""You are the semantic-search-indexer agent.

Operation: incremental-reindex
Directory: /path/to/project
Max Age: 360  # minutes (6 hours)

Execute smart auto-reindexing using scripts/incremental-reindex.
This will detect changed files using Merkle tree, then auto-fallback to full reindex.
Return statistics showing total files indexed and total chunks."""
)

Example 4: Find Similar (semantic-search-reader)

Task(
    subagent_type="semantic-search-reader",
    description="Find similar content chunks",
    prompt="""You are the semantic-search-reader agent.

Operation: find-similar
Chunk ID: "src/auth.py:45-67:function:authenticate"
K: 5
Project: /path/to/project

Execute the find-similar operation using scripts/find-similar and return interpreted results."""
)

Example 5: Status Check (semantic-search-indexer)

Task(
    subagent_type="semantic-search-indexer",
    description="Check semantic index status",
    prompt="""You are the semantic-search-indexer agent.

Operation: status
Project: /path/to/project

Execute the status operation using scripts/status and return interpreted results with statistics."""
)

Important Notes

• NEVER run bash scripts directly - always spawn the appropriate agent
• Agents handle error interpretation - they convert JSON errors to natural language
• Token optimization: Agent execution happens in separate context (saves YOUR tokens)
• Wait for agent completion - agents return summarized results, not raw JSON

🎯 When to Use This Skill

✅ Use Semantic Search When:

1. Exploring Unfamiliar Projects

• "How does this codebase handle user authentication?"
• "Where is database connection pooling implemented?"
• "Show me error handling patterns in this project"
• "Find documentation about the architecture"

2. Finding Functionality Without Keywords

• Looking for implementations but don't know the exact function names
• Need to find code that "does X" without knowing how it's named
• Searching across multiple languages/frameworks with different conventions

3. Discovering Similar Code

• "Find code similar to this payment processing logic"
• "Are there other implementations of rate limiting?"
• "What other modules use this pattern?"

4. Cross-Reference Discovery

• Finding all authentication methods in a polyglot codebase
• Locating retry logic across different services
• Identifying validation patterns in various modules

5. Searching Documentation & Configuration

• "Find documentation explaining the deployment process"
• "Locate configuration examples for database connections"
• "Search for troubleshooting guides or setup instructions"
• "Find ADRs (Architecture Decision Records) about API design"
• "Locate markdown files about testing strategies"

6. Cross-Format Content Discovery

• "Find all references to environment variables (across code, docs, configs)"
• "Search for rate limiting mentions in any format"
• "Locate authentication documentation and implementation together"
• "Find deployment guides and deployment scripts"

❌ Do NOT Use Semantic Search When:

Use Grep instead for:

• Exact string matching: "import React"
• Known variable/function names: "getUserById"
• Regex patterns: "function.*export"
• File content search with known keywords

Use Glob instead for:

• Finding files by name pattern: "**/*.test.js"
• Locating configuration files: "**/config.yml"
• File system navigation: "src/components/**/*.tsx"

Use Read instead for:

• Reading specific known files
• Examining file contents after Grep/Glob narrowed results
• Sequential file analysis

📋 Prerequisites

Required: Python Library Dependency

> IMPORTANT: This is NOT an MCP server - it's a Python library dependency. No server process runs. Our scripts import Python modules via PYTHONPATH.

This skill requires the claude-context-local Python library for semantic indexing:

# Clone Python library to standard location (5 minutes)
git clone https://github.com/FarhanAliRaza/claude-context-local.git ~/.local/share/claude-context-local

# Set up Python virtual environment and install dependencies
cd ~/.local/share/claude-context-local
python3 -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e .

What this installs:

• Merkle tree change detection (80KB)
• Multi-language code chunking (192KB) - supports 15+ languages
• Embedding generation (76KB) - wraps sentence-transformers
• Dependencies: faiss-cpu, sentence-transformers, tree-sitter

Installation location:

• macOS/Linux: ~/.local/share/claude-context-local
• Windows: %LOCALAPPDATA%\claude-context-local

License: claude-context-local is GPL-3.0. We import via PYTHONPATH (dynamic linking), which preserves our Apache 2.0 license. See docs/architecture/MCP-DEPENDENCY-STRATEGY.md for details.

Index Creation

This skill provides an index script that creates and updates the semantic content index. The index is stored in ~/.claude_code_search/projects/{project_name}_{hash}/ and contains:

• code.index - FAISS vector index
• metadata.db - SQLite database with chunk metadata
• chunk_ids.pkl - Chunk ID mappings
• stats.json - Index statistics

You can verify an index exists using the status script or the list-projects script.

🔄 Auto-Reindex System

Automatic Index Management (Updated v3.0.x - First-Prompt Architecture)

The semantic-search skill now automatically maintains index freshness via the First-Prompt hook, eliminating the need for manual reindexing after code changes. The reindex runs in the background after your first prompt, allowing instant session startup.

How It Works

Background Trigger Logic: The first user prompt after session start spawns a detached background process that checks for changes and updates the index:

| Trigger | Index State | Action | Duration | |---------|-------------|--------|----------| | First prompt | Never indexed | Full index (background) | 3-10 min | | First prompt | Indexed before | Smart reindex (background) | 3-10 min (Merkle check: 3.5s) | | Post-write hook | File modified | Incremental update (synchronous) | ~2.7 sec | | Session start | Any | State initialization only | 30min old) scripts/incremental-reindex /path/to/project --max-age 30

Check if reindex needed without executing

scripts/incremental-reindex /path/to/project --check-only

### Performance Characteristics

**Session Start**: ~0.5 seconds (no reindex blocking)
- Setup & initialization: 360min old / 6 hours (default)
scripts/incremental-reindex /path/to/project

# Custom age threshold (reindex if >30min old)
scripts/incremental-reindex /path/to/project --max-age 30

# Force full reindex regardless of age
scripts/incremental-reindex /path/to/project --full

# Check if reindex needed without executing
scripts/incremental-reindex /path/to/project --check-only

Output: JSON with detailed statistics:

{
  "success": true,
  "full_index": true,
  "files_indexed": 205,
  "chunks_added": 6152,
  "total_chunks": 6152,
  "time_taken": 195.46
}

Key Benefits:

• ✅ Simple: Uses IndexFlatIP (same as MCP - proven, reliable)
• ✅ Compatible: Works on all platforms including Apple Silicon (mps:0)
• ✅ Smart: Merkle tree detects when files changed (triggers full reindex)
• ✅ Safe: Full reindex guarantees no stale data or desynchronization
• ✅ Automatic: Can be triggered by hooks based on age threshold

Operation 3: List Indexed Projects

When to use: See all projects that have been indexed

scripts/list-projects

Output: JSON with array of projects including paths, hashes, creation dates, and index statistics.

Operation 4: Check Index Status

When to use: Verify index exists and inspect statistics for a project

scripts/status --project /path/to/project

Output: JSON with index statistics (chunk count, embedding dimension, files indexed, top folders, chunk types).

Operation 5: Search by Natural Language Query

When to use: Find content by describing what it does or contains

# Basic search (returns top 5 results)
scripts/search --query "user authentication logic" --project /path/to/project

# More results
scripts/search --query "error handling patterns" --k 10 --project /path/to/project

# Search across all indexed projects (omit --project)
scripts/search --query "database queries" --k 5

Output: JSON with ranked results including file paths, line numbers, kind, similarity scores, chunk IDs, and snippets.

Operation 6: Find Similar Content Chunks

When to use: Discover content semantically similar to a reference chunk

# Find similar implementations (use chunk_id from search results)
scripts/find-similar --chunk-id "src/auth.py:45-67:function:authenticate" --project /path/to/project

# More results
scripts/find-similar --chunk-id "lib/utils.py:120-145:method:retry" --k 10 --project /path/to/project

Output: JSON with reference chunk and array of similar chunks ranked by semantic similarity.

📊 JSON Output Format

All scripts output standardized JSON:

Success:

{
  "success": true,
  "data": {
    "results": [...],
    "query": "user authentication",
    "total_results": 5
  }
}

Error:

{
  "success": false,
  "error": "Index not found",
  "suggestion": "Run indexing first or check storage-dir path",
  "path": ".code-search-index"
}

🔄 Typical Workflow

Step 1: Index the Project (One-Time Setup)

scripts/incremental-reindex /path/to/project --full

Step 2: Verify Index Status

scripts/status --project /path/to/project
# or
scripts/list-projects

Step 3: Broad Semantic Search

scripts/search --query "authentication methods" --k 10 --project /path/to/project

Step 4: Find Similar Implementations

# Using chunk_id from search results
scripts/find-similar --chunk-id "src/auth/oauth.py:34-56:function:oauth_login" --project /path/to/project

Step 5: Reindex After Changes

# Auto-reindex (detects changes via Merkle tree, then full reindex)
scripts/incremental-reindex /path/to/project

# Force full reindex (explicit request)
scripts/incremental-reindex /path/to/project --full

Step 6: Narrow with Traditional Tools

# After identifying relevant files, use Read/Grep for details

📚 Reference Documentation

For detailed guidance, see the references/ directory:

• [effective-queries.md](references/effective-queries.md): Query patterns, good/bad examples, domain-specific tips
• [troubleshooting.md](references/troubleshooting.md): Common errors, corner cases, compatibility notes
• [performance-tuning.md](references/performance-tuning.md): Optimizing k values, large codebase strategies

⚙️ Arguments Reference

index

• DIRECTORY (required): Directory to index (positional argument)
• --project-name NAME (optional): Custom project name (default: directory basename)
• --full (optional): Do full reindex (default: incremental)
• -h, --help: Show usage information

list-projects

• No arguments required
• Lists all indexed projects with statistics

status

• --project PATH (optional): Project path to check status for (default: current project or error)

search

• --query "QUERY" (required): Natural language search query
• --k NUM (optional, default: 5): Number of results (5-50 recommended)
• --project PATH (optional): Project path to search in (default: all projects)

find-similar

• --chunk-id "CHUNK_ID" (required): Reference chunk identifier from search results
• --k NUM (optional, default: 5): Number of similar chunks to return
• --project PATH (optional): Project path to search in (default: current project)

🎓 Learning Path

Beginners: Start with effective-queries.md to learn query patterns Troubleshooting: Consult troubleshooting.md for common issues Performance: Read performance-tuning.md for large codebases (>10k files)

🔒 Design Rationale

Why Bash Orchestrators for Python Library Imports?

> Clarification: We use bash scripts to import Python modules, NOT an MCP server. No MCP protocol is used.

1. Simplicity: Bash scripts import existing Python modules directly - no reimplementation needed
2. Reusability: Imports merkle, chunking, embeddings modules (same IndexFlatIP as MCP)
3. Auto-venv: Scripts automatically use claude-context-local's venv Python interpreter
4. Token Efficiency: Scripts are compact (~50 lines each) vs bundling 352KB of Python code
5. Composability: Scripts output JSON, enabling shell pipelines and automation
6. License-safe: Dynamic linking via PYTHONPATH preserves Apache 2.0 license (GPL-compliant)

Orchestrator Pattern

Each bash script:

1. Sets VENV_PYTHON to ~/.local/share/claude-context-local/.venv/bin/python
2. Sets `PYTHONPATH

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ahmedibrahim085
• Source: ahmedibrahim085/Claude-Multi-Agent-Research-System-Skill
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.