Subcog

Subcog

[](https://github.com/zircote/subcog/actions/workflows/ci.yml) [](https://github.com/zircote/subcog/releases) [](https://crates.io/crates/subcog) [](https://www.rust-lang.org/) [](LICENSE) [](https://github.com/rust-lang/rust-clippy) [](https://github.com/EmbarkStudios/cargo-deny)

A persistent memory system for AI coding assistants. Subcog captures decisions, learnings, and context from coding sessions and surfaces them when relevant.

Overview

Subcog delivers:

• Single-binary distribution (=0.7**: Strong semantic match
• >=0.5: Moderate relevance
• Note**: This configuration requires the subcog binary to be installed and in your PATH. Install via cargo install subcog, Homebrew, or download from GitHub Releases. If you cannot install the binary, use npx as a fallback:

> ``json > { "command": "npx", "args": ["-y", "@zircote/subcog", "serve"] } > ``

Available MCP Tools

Subcog exposes ~22 consolidated MCP tools (see [ADR-0061](docs/adrs/adr_0061.md) for the consolidation design):

| Category | Tools | Description | |----------|-------|-------------| | Core | subcog_capture, subcog_recall, subcog_status | Memory CRUD and search | | Lifecycle | subcog_consolidate, subcog_reindex | Maintenance operations | | CRUD | subcog_get, subcog_update, subcog_delete, subcog_list | Individual memory operations | | Bulk | subcog_delete_all, subcog_restore, subcog_history | Bulk and recovery operations | | Graph | subcog_graph, subcog_entities, subcog_relationships | Knowledge graph queries | | Prompts | subcog_prompts, prompt_understanding | Prompt template management | | Templates | subcog_templates | Context template management | | Session | subcog_init, subcog_get_summary, subcog_namespaces | Session and namespace info | | Compliance | subcog_gdpr_export | Data export for compliance |

When working with an agent, treat inputs that match MCP tool names as tool invocations (not shell commands) unless you explicitly say "shell" or "run in terminal".

Transport Security

Subcog supports two transport modes with different security models:

Stdio Transport (Default)

The stdio transport is the default and recommended mode for local development:

| Property | Description | |----------|-------------| | Trust Model | Process isolation via OS - parent spawns subcog as child process | | Network Exposure | None - communication only via stdin/stdout pipes | | Authentication | Implicit - same-user execution, no credentials required | | Confidentiality | Data stays on local machine (SQLite is authoritative) | | Integrity | OS guarantees pipe integrity, no MITM attacks possible |

When to use: Local development with Claude Desktop or other MCP clients that spawn subcog directly.

HTTP Transport (Optional)

Enable the HTTP transport for network-accessible deployments:

# Enable HTTP transport
subcog serve --http --port 8080

# With JWT authentication (recommended for production)
subcog serve --http --port 8080 --jwt-secret "your-secret-key"

| Property | Description | |----------|-------------| | Trust Model | JWT token authentication required | | Network Exposure | Binds to specified port (localhost by default) | | Authentication | JWT tokens with configurable expiry | | CORS | Configurable allowed origins | | TLS | Use a reverse proxy (nginx, Caddy) for HTTPS |

When to use: Team environments, remote access, or integration with web-based clients.

Data Protection

Both transports include built-in security features:

• Secrets Detection: API keys, tokens, and passwords are detected and optionally redacted
• PII Filtering: Personal information can be masked before storage
• Encryption at Rest: Enable with encryption_enabled = true (default: true)
• Audit Logging: All operations are logged for compliance (SOC2, GDPR)

See [environment-variables.md](docs/environment-variables.md) for security configuration options.

Claude Code Hooks

Subcog integrates with all 5 Claude Code hooks:

| Hook | Purpose | |------|---------| | SessionStart | Inject relevant context at session start | | UserPromptSubmit | Detect capture signals in prompts | | PostToolUse | Surface related memories after file operations | | PreCompact | Analyze conversation for auto-capture | | Stop | Finalize session, capture pending memories |

Configure in ~/.claude/settings.json:

{
  "hooks": {
    "SessionStart": [{ "command": "subcog hook session-start" }],
    "UserPromptSubmit": [{ "command": "subcog hook user-prompt-submit" }],
    "PostToolUse": [{ "command": "subcog hook post-tool-use" }],
    "PreCompact": [{ "command": "subcog hook pre-compact" }],
    "Stop": [{ "command": "subcog hook stop" }]
  }
}

Migration

Upgrade existing memories to use real embeddings:

# Dry-run (see what would be migrated)
subcog migrate embeddings --dry-run

# Migrate all memories without embeddings
subcog migrate embeddings

# Force re-generation of all embeddings
subcog migrate embeddings --force

# Migrate from a specific repository
subcog migrate embeddings --repo /path/to/repo

The migration command:

• Scans all memories in the index
• Generates embeddings using fastembed (all-MiniLM-L6-v2)
• Stores embeddings in the vector backend (usearch HNSW)
• Skips memories that already have embeddings (unless --force)
• Shows progress with migrated/skipped/error counts

Architecture

Subcog uses a three-layer storage architecture to separate concerns:

System Architecture Diagram

flowchart TB
    subgraph Access["Access Layer"]
        CLI["CLIsubcog capture/recall/status"]
        MCP["MCP ServerJSON-RPC over stdio"]
        Hooks["Claude Code HooksSessionStart, UserPrompt, Stop"]
    end

    subgraph Services["Service Layer"]
        Capture["CaptureServiceMemory ingestion"]
        Recall["RecallServiceHybrid search"]
        GC["GCServiceBranch cleanup"]
        Dedup["DeduplicationService3-tier duplicate detection"]
        Context["ContextBuilderAdaptive injection"]
    end

    subgraph Storage["Three-Layer Storage"]
        subgraph Persistence["Persistence Layer(Authoritative)"]
            SQLiteP["SQLite(default)"]
            PostgresP["PostgreSQL"]
            FS["Filesystem"]
        end

        subgraph Index["Index Layer(Searchable)"]
            SQLiteI["SQLite + FTS5(default)"]
            PostgresI["PostgreSQL FTS"]
            Redis["RediSearch"]
        end

        subgraph Vector["Vector Layer(Embeddings)"]
            usearch["usearch HNSW(default)"]
            pgvector["pgvector"]
            RedisV["Redis Vector"]
        end
    end

    subgraph External["External Systems"]
        FastEmbed["FastEmbedall-MiniLM-L6-v2"]
        LLM["LLM ProviderAnthropic/OpenAI/Ollama"]
    end

    CLI --> Capture
    CLI --> Recall
    MCP --> Capture
    MCP --> Recall
    Hooks --> Context
    Hooks --> Capture

    Capture --> Persistence
    Capture --> Index
    Capture --> Vector
    Capture --> FastEmbed
    Capture --> Dedup

    Recall --> Index
    Recall --> Vector
    Recall --> FastEmbed

    Context --> Recall
    Context --> LLM

    Dedup --> Recall
    Dedup --> FastEmbed

    GC --> Persistence
    GC --> Index

    style Access fill:#e1f5fe
    style Services fill:#fff3e0
    style Storage fill:#e8f5e9
    style External fill:#fce4ec

Data Flow Diagram

sequenceDiagram
    participant User
    participant CLI/MCP
    participant CaptureService
    participant Dedup
    participant FastEmbed
    participant Persistence
    participant Index
    participant Vector

    User->>CLI/MCP: subcog capture "decision..."
    CLI/MCP->>CaptureService: CaptureRequest
    CaptureService->>Dedup: Check duplicate
    Dedup->>Index: Hash tag lookup (exact)
    Dedup->>FastEmbed: Generate embedding
    Dedup->>Vector: Similarity search (semantic)
    Dedup-->>CaptureService: Not duplicate

    CaptureService->>FastEmbed: Generate embedding
    FastEmbed-->>CaptureService: [384-dim vector]

    par Store in all layers
        CaptureService->>Persistence: Store memory
        CaptureService->>Index: Index for FTS
        CaptureService->>Vector: Store embedding
    end

    CaptureService-->>CLI/MCP: CaptureResult{id, urn}
    CLI/MCP-->>User: Memory captured

Hybrid Search Flow

flowchart LR
    Query["Query: 'database storage decision'"]

    subgraph Search["Parallel Search"]
        BM25["BM25 Search(Index Layer)"]
        VectorSearch["Vector Search(Vector Layer)"]
    end

    subgraph Results["Raw Results"]
        BM25Results["id1: 2.3id2: 1.8id3: 1.2"]
        VectorResults["id2: 0.92id1: 0.85id4: 0.78"]
    end

    RRF["RRF Fusionscore = sum(1/(k+rank))"]

    Final["Final Results(normalized 0.0-1.0)id2: 1.00id1: 0.87id3: 0.45id4: 0.38"]

    Query --> BM25
    Query --> VectorSearch
    BM25 --> BM25Results
    VectorSearch --> VectorResults
    BM25Results --> RRF
    VectorResults --> RRF
    RRF --> Final

ASCII Architecture Reference

                              +-----------------+
                              |   Access Layer  |
                              +-----------------+
                              |  CLI | MCP | Hooks
                              +--------+--------+
                                       |
                              +--------v--------+
                              |  Service Layer  |
                              +-----------------+
                              | Capture | Recall | GC
                              +--------+--------+
                                       |
        +------------------------------+------------------------------+
        |                              |                              |
+-------v-------+             +--------v-------+             +--------v-------+
|  Persistence  |             |     Index      |             |     Vector     |
|    Layer      |             |     Layer      |             |     Layer      |
+---------------+             +----------------+             +----------------+
|               |             |                |             |                |
| - Authoritative             | - Full-text    |             | - Embeddings   |
|   source of truth           |   search (BM25)|             |   (384-dim)    |
| - ACID storage              | - Faceted      |             | - Similarity   |
| - Durable                   |   filtering    |             |   search (ANN) |
|               |             |                |             |                |
+-------+-------+             +--------+-------+             +--------+-------+
        |                              |                              |
+-------v-------+             +--------v-------+             +--------v-------+
|    SQLite     |             | SQLite + FTS5  |             |    usearch     |
|  (default)    |             |   (default)    |             |   (HNSW)       |
+---------------+             +----------------+             +----------------+
|  PostgreSQL   |             |  PostgreSQL    |             |   pgvector     |
+---------------+             +----------------+             +----------------+
|  Filesystem   |             |  RediSearch    |             | Redis Vector   |
+---------------+             +----------------+             +----------------+

Layer Responsibilities

| Layer | Purpose | Default Backend | Alternatives | |-------|---------|-----------------|--------------| | Persistence | Authoritative storage, ACID guarantees | SQLite | PostgreSQL, Filesystem | | Index | Full-text search, BM25 ranking | SQLite + FTS5 | PostgreSQL, RediSearch | | Vector | Embedding storage, ANN search | usearch (HNSW) | pgvector, Redis Vector |

For detailed architecture documentation, see [src/storage/traits/mod.rs](src/storage/traits/mod.rs).

Development

Prerequisites

• Rust 1.88+ (Edition 2024)
• Git 2.30+
• cargo-deny for supply chain security

Setup

git clone https://github.com/zircote/subcog.git
cd subcog

# Build
cargo build

# Run tests
cargo test

# Run all checks
cargo fmt -- --check && \
cargo clippy --all-targets --all-features -- -D warnings && \
cargo test && \
cargo doc --no-deps && \
cargo deny check

Project Structure

src/
├── lib.rs              # Library entry point
├── main.rs             # CLI entry point
├── models/             # Data structures (Memory, Domain, Namespace)
├── storage/            # Storage backends (SQLite, PostgreSQL, usearch)
│   └── traits/         # Backend trait definitions (see mod.rs for docs)
├── services/           # Business logic (Capture, Recall, GC)
├── mcp/                # MCP server implementation
├── hooks/              # Claude Code hook handlers
├── embedding/          # Vector embedding generation
└── observability/      # Tracing, metrics, logging

docs/
├── QUICKSTART.md       # Getting started guide
├── TROUBLESHOOTING.md  # Common issues and solutions
├── PERFORMANCE.md      # Performance tuning guide
├── research/           # Research documents
└── spec/               # Specification documents

Configuration

Configuration file at ~/.config/subcog/config.toml:

[storage]
backend = "sqlite"  # "sqlite", "postgres", "filesystem"
data_dir = "~/.local/share/subcog"

[embedding]
model = "all-MiniLM-L6-v2"
dimensions = 384

[hooks]
enabled = true
session_start_timeout_ms = 2000
user_prompt_timeout_ms = 50

[llm]
provider = "anthropic"  # Optional: for Tier 3 features

Faceted Capture

Memories can be tagged with project, branch, and file path:

# Capture with facets (auto-detected from git context)
subcog capture --namespace decisions "Use PostgreSQL"

# Capture with explicit facets
subcog capture --namespace decisions --project my-project --branch feature/auth "Added JWT support"

# Search within a project
subcog recall "authentication" --project my-project

# Search within a branch
subcog recall "bug fix" --branch feature/auth

# Include tombstoned memories
subcog recall "old decision" --include-tombstoned

Branch Garbage Collection

Clean up memories from deleted branches:

# GC current project (dry-run)
subcog gc --dry-run

# GC specific branch
subcog gc --branch feature/old-branch

# Purge tombstoned memories older than 30 days
subcog gc --purge --older-than 30d

Performance Targets

| Metric | Target | Actual | |--------|--------|--------| | Cold start | <10ms | ~5ms | | Capture latency | <30ms | ~25ms | | Search latency (100 memories) | <20ms | ~82us | | Search latency (1,000 memories) | <50ms | ~413us | | Search latency (10,000 memories) | <100ms | ~3.7ms | | Binary size | <100MB | ~50MB | | Memory (idle) | <50MB | ~30MB |

All performance targets are exceeded by 10-100x. Benchmarks run via cargo bench.

For performance tuning, see [docs/PERFORMANCE.md](docs/PERFORMANCE.md).

Documentation

| Document | Description | |----------|-------------| | [INSTALLATION.md](docs/INSTALLATION.md) | Complete installation guide (npm, Docker, Homebrew, Cargo) | | [QUICKSTART.md](docs/QUICKSTART.md) | Getting started guide | | [TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) | Common issues and solutions | | [PERFORMANCE.md](docs/PERFORMANCE.md) | Performance tuning guide | | [environment-variables.md](docs/environment-variables.md) | Environment variable reference | | [URN-GUIDE.md](docs/URN-GUIDE.md) | Memory URN scheme documentation |

Specification

Active Work

See docs/spec/active/ for current work in progress.

Completed Specifications

Full specification documents for the storage architecture are in [docs/spec/completed/2026-01-03-storage-simplification/](docs/spec/completed/2026-01-03-stora

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: zircote
• Source: zircote/subcog
• License: MIT

Install and usage instructions live in the source repository linked above.