Textrawl

Textrawl

[](LICENSE) [](https://nodejs.org) [](https://www.typescriptlang.org) [](https://www.postgresql.org) [](https://neon.tech) [](https://openai.com) [](https://ollama.com) [](https://modelcontextprotocol.io)

Your second brain, wired for AI.

Textrawl is a personal knowledge server with persistent memory, searchable documents, and proactive insights. Import your emails, PDFs, and notes — then ask questions, recall past conversations, and discover connections you missed. Access it all through MCP, the web dashboard, or the REST API.

How It Works

┌──────────────┐         ┌───────────────────────────────────────────┐
│              │         │           Your Second Brain               │
│  MCP Client  │◄───────►│                                           │
│  (Claude,    │   MCP   │   Documents    Memory     Conversations   │
│   ChatGPT)   │         │   ┌────────┐  ┌───────┐  ┌────────────┐  │
└──────────────┘         │   │ Emails │  │ Facts │  │Past sessions│  │
                         │   │ PDFs   │  │People │  │ Summaries  │  │
┌──────────────┐         │   │ Notes  │  │ Links │  │ Context    │  │
│   Dashboard  │◄───────►│   └───┬────┘  └──┬────┘  └─────┬──────┘  │
│  (Web UI)    │  REST   │       └──────────┼──────────────┘         │
└──────────────┘         │                  ▼                        │
                         │     ┌───────────────────────────┐         │
                         │     │  Hybrid Search + Fusion   │         │
                         │     └───────────────────────────┘         │
                         │                  │                        │
                         │                  ▼                        │
                         │      Insights · Daily Briefing            │
                         └───────────────────────────────────────────┘
                                            ▲
                                            │
                                 ┌──────────┴──────────┐
                                 │                     │
                            Desktop App            CLI Tools
                           (drag & drop)        (batch import)

Why Textrawl?

Beyond keyword search. Most search tools only match exact words. Textrawl combines semantic understanding (finds "automobile" when you search "car") with traditional keyword matching — so you get relevant results without missing exact phrases.

Your data, your choice. Use OpenAI's embeddings for best accuracy, Google AI for multimodal support, or run locally with Ollama and local Postgres to keep document text and embeddings on your machine.

Import everything. Emails from Gmail exports, PDFs from your research, saved web pages, images, audio files, Google Takeout archives — Textrawl converts them into searchable knowledge where the relevant converter/provider is configured.

Features

| Feature | Description | |---------|-------------| | Hybrid Search | Vector similarity + full-text search with Reciprocal Rank Fusion | | Persistent Memory | Remember facts about people, projects, and concepts across sessions | | Conversation Recall | Save and query past conversation context across sessions | | Proactive Insights | Automatically discover connections, patterns, and outliers in your knowledge | | Daily Briefing | Summary of recent additions, new insights, and resurfaced knowledge | | Unified RAG | ask tool searches documents, memory, and conversations in one query | | Web Dashboard | Command center with knowledge explorer, timeline, agent orchestration, and applets | | Multimodal | Process images (Claude vision) and audio (Whisper transcription) alongside documents | | Desktop App | Drag-and-drop file conversion and upload (macOS, Windows, Linux) | | Multi-Format | PDF, DOCX, XLSX, PPTX, HTML, MBOX/EML emails, Google Takeout | | MCP + REST + WebSocket | MCP tools, REST API, and real-time WebSocket events | | Agent Discovery | A2A protocol at /.well-known/agent.json for agent-to-agent interaction | | Flexible Embeddings | OpenAI, Google AI, or Ollama (free, local) | | Smart Chunking | Paragraph-aware splitting with overlap for context | | CLI Tools | Batch processing for large archives | | Cloud Ready | Deploy to Docker, Cloud Run, or any container platform; large uploads require GCS/Cloud Tasks configuration |

Privacy Model

Textrawl is self-hosted, but data leaves your machine when you configure cloud services. Document text, chunks, embeddings, extracted memories, conversation summaries, images, or audio may be sent to providers such as OpenAI/Google embeddings, Anthropic/OpenAI/Google extraction, Neon/Supabase/RDS, Cloud Run, or GCS. For sensitive data, prefer Ollama/local Postgres and disable cloud LLM extraction/insights.

Quick Start

1. Set Up the Server

git clone https://github.com/jeffgreendesign/textrawl.git
cd textrawl
pnpm install
pnpm setup    # Interactive setup for credentials
pnpm dev      # Start the server

2. Set Up Your Database

1. Create a free project at neon.tech
2. Copy the pooled connection string from the Neon dashboard into DATABASE_URL in .env
3. Run the base schema against your database:

``bash psql $DATABASE_URL -f scripts/setup-db.sql ``

(Use setup-db-ollama.sql for Ollama / setup-db-ollama-v2.sql for nomic-embed-text-v2-moe / setup-db-google.sql for Google AI)

1. (Optional) For memory tools: psql $DATABASE_URL -f scripts/setup-db-memory.sql
2. (Optional) For conversation tools: psql $DATABASE_URL -f scripts/setup-db-conversation.sql
3. (Optional) For Row Level Security hardening: psql $DATABASE_URL -f scripts/security-rls.sql

3. Connect Claude Desktop

Add to your Claude config (~/Library/Application Support/Claude/claude_desktop_config.json). Create this file if it doesn't exist:

{
  "mcpServers": {
    "textrawl": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://localhost:3000/mcp",
        "--header",
        "Accept: application/json, text/event-stream"
      ]
    }
  }
}

Note: Requires Node.js 22+. If using nvm, ensure your default is set: nvm alias default 22

If you've set API_BEARER_TOKEN in .env, add the auth header:

"--header",
"Authorization: Bearer "

Restart Claude Desktop - you'll now see Textrawl's tools available.

3b. Connect ChatGPT Desktop (Alternative)

ChatGPT Desktop supports MCP servers natively (Pro/Plus required):

1. Open Settings → Connectors → Advanced → Developer mode
2. Add a new connector with your server URL: http://localhost:3000/mcp
3. If using auth, add the Authorization: Bearer header

See OpenAI MCP documentation for details.

4. Add Your Documents

Imported documents, extracted memories, and conversation summaries are stored in your configured database/storage until deleted. Treat a Textrawl server as single-tenant unless you have added your own user isolation. Set API_BEARER_TOKEN, restrict CORS with ALLOWED_ORIGINS, and avoid importing third-party or private data without consent. Use forget_entity and delete_conversation to remove memory/conversation data, and list_documents/update_document to audit imported documents.

Option A: Desktop App (easiest)

pnpm desktop:dev

Drag files onto the window to convert and upload.

Option B: CLI (for batch imports)

pnpm convert -- mbox ~/Mail/archive.mbox
pnpm upload -- ./converted/

Documentation

| Guide | Description | |-------|-------------| | [Database Sizing](docs/guides/database-requirements.mdx) | Vector dimensions, index counts, and storage estimates by embedding provider | | [CLI Tools](docs/cli/) | Batch conversion and upload from command line | | [Security](docs/guides/security-hardening.mdx) | Row Level Security and access controls |

---

Configuration

| Variable | Required | Description | |----------|----------|-------------| | DATABASE_URL | Yes | Neon (or any PostgreSQL) pooled connection string | | DATABASE_URL_UNPOOLED | No | Direct connection for schema migrations (optional) | | EMBEDDING_PROVIDER | No | openai (default), ollama, or google | | OPENAI_API_KEY | If OpenAI | For text-embedding-3-small (1536d) | | OLLAMA_BASE_URL | If Ollama | Default: http://localhost:11434 | | OLLAMA_MODEL | If Ollama | Default: nomic-embed-text | | GOOGLE_AI_API_KEY | If Google | For gemini-embedding-2-preview (3072d) | | GOOGLE_EMBEDDING_MODEL | If Google | Default: gemini-embedding-2-preview | | API_BEARER_TOKEN | Prod only | Min 32 chars (openssl rand -hex 32) | | PORT | No | Default: 3000 | | LOG_LEVEL | No | debug, info, warn, error | | ALLOWED_ORIGINS | No | Comma-separated CORS origins | | ENABLE_MEMORY | No | Enable memory tools (default: true); requires setup-db-memory.sql | | ENABLE_CONVERSATIONS | No | Enable conversation memory tools (default: true); requires setup-db-conversation.sql | | ENABLE_INSIGHTS | No | Enable proactive insight tools (default: true) | | ENABLE_MEMORY_EXTRACTION | No | Enable LLM-based memory extraction (default: false) | | ANTHROPIC_API_KEY | If extraction | Required for extract_memories tool | | EXTRACTION_MODEL | No | Model for extraction (default: claude-haiku-4-5-20251001) | | INSIGHT_MODEL | No | Model for insight synthesis (default: claude-sonnet-4-6) | | COMPACT_RESPONSES | No | Token-efficient responses (default: true) | | CHUNKING_MODE | No | fixed (default) or semantic (embedding-based splits) | | SEMANTIC_SIMILARITY_THRESHOLD | No | Semantic split sensitivity 0–1 (default: 0.5) | | REDIS_URL | No | Redis URL for shared rate limiting across instances | | GOOGLE_CLIENT_ID | OAuth | OAuth 2.0 client ID (all four OAuth vars required together) | | GOOGLE_CLIENT_SECRET | OAuth | OAuth 2.0 client secret | | OAUTH_JWT_SECRET | OAuth | Min 32-char secret for JWT signing | | OAUTH_ALLOWED_EMAILS | OAuth | Comma-separated email allowlist (optional) | | OAUTH_SERVER_URL | OAuth | Public server URL for OAuth redirect | | PG_REPORT_DIR | No | pg_analyze report directory (default: ./reports/pg-analysis) |

Tool surfaces (compact vs full)

Textrawl advertises one of three tool surfaces via the MCP_TOOLSET environment variable. The default normal surface is a small set of workflow tools — the recommended interface for personal/family assistants. This follows the current MCP maintainer guidance (Anthropic, Writing effective tools for AI agents): fewer tools, consolidated by workflow into distinct, typed, well-named tools — not a single "intent" dispatcher (which would discard per-tool schemas, annotations, and the name signals models use to pick tools). The MCP spec defines no standard tool-filtering primitive, so MCP_TOOLSET is a server-local convention; host/harness lazy tool loading remains the canonical way to surface fewer tools at a time.

| MCP_TOOLSET | Tools advertised | |---|---| | normal (default) | Workflow tools: ask, search, get_document, capture, daily_briefing, timeline (+ remember when ENABLE_MEMORY=true). Set EXPOSE_ADMIN_TOOLS=true to also expose read-only diagnostics (health_check, get_stats, insight + Postgres tools). | | full | Workflow tools + diagnostics + all original granular tools (backward compatible). | | legacy | Exactly the original tool set (no workflow tools). |

Recommended exposure: personal/family bots → MCP_TOOLSET=normal and EXPOSE_ADMIN_TOOLS=false (so destructive forget_entity/delete_conversation are never reachable); admin/dev contexts → MCP_TOOLSET=full.

Model guidance (GPT-5.5, Claude Sonnet, Claude Haiku): express intent through the workflow tool that matches the job — ask to answer a question across sources, search for raw matches, capture to save a note or URL, remember to store facts, timeline/daily_briefing for temporal/briefing views. Use audience to scope who an answer is for; family_shared/public_safe automatically exclude private memory, conversations, and insights.

MCP Tools

Read-only tools (search, get_document, list_documents, query_memory, query_conversations, get_stats, health_check) include outputSchema and return structuredContent for programmatic consumption alongside the text content response.

Workflow Tools (compact normal surface)

| Tool | Description | |------|-------------| | capture | Save content to the knowledge base — kind="note" (title+content) or kind="url" (fetch + clip). Consolidates add_note + save_url. | | remember | Write structured knowledge to the memory graph — facts and/or relations. Consolidates remember_fact + build_knowledge + relate_entities. |

ask, search, get_document, daily_briefing, and timeline (documented below) complete the workflow surface. remember is only advertised when ENABLE_MEMORY=true (omitted otherwise). ask/search accept audience + allowCrossProfile for privacy scoping.

Document Tools

| Tool | Description | |------|-------------| | search | Hybrid semantic + full-text search. Set includeMemories/includeConversations for cross-source fusion. audience scopes private sources. | | get_document | Retrieve document by ID | | list_documents | List with pagination and filtering | | update_document | Update title and/or tags | | add_note | Add markdown note to knowledge base |

Memory Tools (Persistent Memory)

Enable with ENABLE_MEMORY=true (default). Requires scripts/setup-db-memory.sql or setup-db-memory-ollama.sql.

| Tool | Description | |------|-------------| | remember_fact | Store facts about entities (people, projects, concepts) | | build_knowledge | Store multiple facts and relations in a single batch call | | query_memory | Query the memory graph (mode: 'search' \| 'entity' \| 'list') | | relate_entities | Create relationships between entities | | forget_entity | Delete an entity and all its memories | | extract_memories | Extract entities and facts from text using LLM |

Conversation Tools (Conversation Memory)

Enable with ENABLE_CONVERSATIONS=true (default). Requires running one of the conversation schema scripts:

• scripts/setup-db-conversation.sql (OpenAI embeddings, 1536d)
• scripts/setup-db-conversation-ollama.sql (Ollama v1 - nomic-embed-text, 1024d)
• scripts/setup-db-conversation-ollama-v2.sql (Ollama v2 - nomic-embed-text-v2-moe, 768d)
• scripts/setup-db-conversation-google.sql (Google AI - gemini-embedding-2-preview, 3072d)

| Tool | Description | |------|-------------| | save_conversation_context | Save conversation summary and turns for recall | | query_conversations | Query past conversations (mode: 'search' \| 'get' \| 'list') | | delete_conversation | Delete a conversation session |

Insight Tools (Proactive Discovery)

Enable with ENABLE_INSIGHTS=true (default).

| Tool | Description | |------|-------------| | get_insights | View discovered cross-source connections and patterns | | discover_connections | Trigger an insight scan across the knowledge base | | dismiss_insight | Dismiss an insight from the queue |

Stats

| Tool | Description | |------|-------------| | get_stats | Statistics across all features (scope: 'all' \| 'knowledge' \| 'memory' \| 'conversations' \| 'insights') | | health_check | Check the health of all subsystems — returns pass/fail per component with overall status (verbose for latency and counts) |

Unified Tools

| Tool | Description | |------|-------------| | ask | Unified RAG search across all knowledge sources (documents, memory, conversations, insights) | | daily_briefing | Generate a daily briefing with recent additions, new insights, and resurfaced

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jeffgreendesign
• Source: jeffgreendesign/textrawl
• License: MIT
• Homepage: https://www.textrawl.com/

Install and usage instructions live in the source repository linked above.