Renfield

English | [Deutsch](README.de.md)

Renfield

Self-hosted AI assistant with voice satellites, 100+ tool integrations, and full offline capability.

[](https://python.org) [](https://fastapi.tiangolo.com) [](https://react.dev) [](https://ollama.ai) [](LICENSE) [](tests/)

---

Why Renfield?

• Voice-first with physical satellites — Raspberry Pi Zero 2 W + ReSpeaker HAT in every room (~$30/satellite). Wake word, STT, TTS — fully local.
• 10 MCP servers, 100+ tools, Agent Loop — Weather, search, news, calendar, smart home, media, DLNA, documents, email, workflows. ReAct agent chains tools autonomously for complex queries.
• Fully offline, self-hosted, GDPR-friendly — Runs entirely on your hardware. No cloud dependencies. Your data never leaves your network.

Screenshots

Chat Interface — Conversation history, streaming responses, voice input

Knowledge Base (RAG) — Document upload, hybrid search, knowledge bases

Integrations — 10 MCP servers, 100+ tools

Room Management — 17 rooms synced with Home Assistant, DLNA renderer assignment

DLNA Output Device — Assign DLNA renderers to rooms for album playback

Satellite Monitor — Live status of voice satellites

Memory — Long-term recall of preferences and facts

Features

Core

• Chat interface with streaming responses, conversation history, inline album art, collapsible agent steps
• ReAct Agent System with specialized roles, multi-step tool chaining, and media transport shortcuts
• Knowledge Graph — entity-relation triples with post-extraction validation, string-similarity dedup, and bulk cleanup API
• Conversational memory — long-term recall of preferences, facts, and instructions with contradiction detection
• Intent feedback learning — learns from corrections via semantic matching (3-scope: intent, tool, complexity)
• Voice I/O — Whisper STT + Piper TTS + SpeechBrain speaker recognition
• Presence detection — BLE scanning, voice recognition, and web auth track who's in which room
• Proactive notifications — webhook-based alerts from Home Assistant/n8n + generic MCP notification polling
• Paperless audit — automated LLM-based metadata validation with review queue, OCR quality scoring, and auto-fix
• Hook/Plugin system — async extension API for custom integrations without modifying core

Integrations (10 MCP Servers)

| Server | Description | Transport | |--------|-------------|-----------| | Weather | OpenWeatherMap forecasts | stdio | | Search | SearXNG metasearch | stdio | | News | NewsAPI headlines | stdio | | Calendar | Exchange, Google, CalDAV (multi-account) | stdio | | Jellyfin | Media server queries | stdio | | DLNA | Media renderer control with gapless queue | streamablehttp | | n8n | Workflow automation | stdio | | Home Assistant | Smart home control | streamablehttp | | Paperless | Document management | stdio | | Email | IMAP/SMTP | stdio |

Knowledge Base (RAG)

• Hybrid search — dense embeddings (pgvector) + BM25 full-text (German FTS, OR matching), combined via RRF
• Formats — PDF, DOCX, PPTX, XLSX, HTML, Markdown, TXT (with EasyOCR fallback for garbled PDFs)
• Knowledge bases — thematic collections with sharing and access control
• Agent tool — knowledge_search for combined RAG + Paperless search within the agent loop
• Knowledge Graph — entity-relation triples with post-extraction validation, string-similarity dedup, bulk cleanup API, admin dashboard

Multi-Room Voice Satellites

| Component | Cost | |-----------|------| | Raspberry Pi Zero 2 W | ~$18 | | ReSpeaker 2-Mics Pi HAT | ~$12 | | MicroSD, power supply, speaker | ~$28 | | Total per room | ~$58 |

• Local wake word detection (OpenWakeWord)
• Audio output routing to best device per room (Renfield, HA, DLNA)
• IP-based room context detection

Presence Detection

Multi-source room-level presence tracking:

| Source | Trigger | Latency | |--------|---------|---------| | BLE Scanning | Satellite detects phone/watch via Bluetooth | ~30s (hysteresis) | | Voice Presence | Speaker recognition identifies user | Instant | | Web Auth | Authenticated user on room-assigned device | Instant |

• Privacy-aware TTS — notifications respect room occupancy (public / personal / confidential)
• Automation hooks — enter_room, leave_room, first_arrived, last_left events fire webhooks for n8n / Home Assistant
• Presence dashboard — real-time room occupancy in the admin UI

Security & Access Control

• Role-permission based access control (RPBAC) with JWT
• Rate limiting, circuit breakers, trusted proxy support
• Docker secrets management for production

Platform

• Progressive Web App (desktop, tablet, mobile)
• Dark mode (light, dark, system)
• Multilingual (German, English)
• Prometheus metrics (opt-in)

Quick Start

git clone https://github.com/ebongard/renfield.git && cd renfield
cp .env.example .env                          # configure your settings
docker compose up -d                          # start the stack
docker exec -it renfield-ollama ollama pull qwen3:8b  # download LLM

Open http://localhost:3000 and start chatting.

> For GPU acceleration, use docker compose -f docker-compose.prod.yml up -d with NVIDIA Container Toolkit.

Architecture

Configuration

All settings via .env, loaded by Pydantic Settings. See [docs/ENVIRONMENTVARIABLES.md](docs/ENVIRONMENTVARIABLES.md) for the full reference.

LLM (Multi-Model)

Separate models for different tasks:

OLLAMA_URL=http://ollama:11434
OLLAMA_CHAT_MODEL=qwen3:14b       # chat responses
OLLAMA_INTENT_MODEL=qwen3:8b      # intent recognition
OLLAMA_RAG_MODEL=qwen3:14b        # RAG answers
OLLAMA_EMBED_MODEL=nomic-embed-text  # embeddings (768 dim)

Pluggable inference — not everything runs on Ollama. The LLM tier speaks an OpenAI-compatible protocol, so each task can route to a different backend (LLM_OPENAI_FOR_*). Ollama is the simple default; the reference deployment splits inference across dedicated GPU servers:

| Tier | Server | Model | |------|--------|-------| | Chat · Agent · RAG · Intent · KG · Memory | llama.cpp | Qwen3.6-35B-A3B (MoE) | | Embeddings | llama.cpp | Qwen3-Embedding-4B | | Vision | Ollama | qwen3-vl |

Voice runs on its own server. Speech-to-text (Whisper), text-to-speech (Piper), and speaker recognition (ECAPA-TDNN) run in a dedicated, GPU-resident voice-server microservice over a streaming WebSocket — set VOICE_SERVER_URL to use it (they run in-process for local dev otherwise).

Key Settings

AGENT_ENABLED=false               # ReAct agent loop (opt-in)
MEMORY_ENABLED=false              # long-term memory (opt-in)
AUTH_ENABLED=false                 # RPBAC auth (opt-in)
MCP_ENABLED=true                  # master switch for integrations
PRESENCE_ENABLED=false            # room presence detection (opt-in)
KNOWLEDGE_GRAPH_ENABLED=false     # entity-relation extraction (opt-in)
NOTIFICATION_POLLER_ENABLED=false # proactive MCP notifications (opt-in)
METRICS_ENABLED=false             # Prometheus /metrics (opt-in)

Development

make lint                    # lint all code (ruff + eslint)
make test                    # all tests
make test-backend            # backend tests (3,400+)
make test-frontend-react     # React tests (Vitest + RTL)
make test-coverage           # tests with coverage report

See [CONTRIBUTING.md](CONTRIBUTING.md) for the full development guide.

Docker Compose Variants

| File | Use Case | GPU | |------|----------|-----| | docker-compose.yml | Standard | No | | docker-compose.dev.yml | Development (Mac) | No | | docker-compose.prod.yml | Production | NVIDIA | | docker-compose.prod-cpu.yml | Production | No |

Documentation

| Document | Content | |----------|---------| | [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute | | [docs/FEATURES.md](docs/FEATURES.md) | Detailed feature documentation | | [docs/ENVIRONMENTVARIABLES.md](docs/ENVIRONMENTVARIABLES.md) | Full configuration reference | | [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Deployment guide | | [docs/SECRETSMANAGEMENT.md](docs/SECRETSMANAGEMENT.md) | Docker secrets for production | | [docs/SECURITY.md](docs/SECURITY.md) | Security headers, CSP, dependency security | | [docs/ACCESSCONTROL.md](docs/ACCESSCONTROL.md) | Role-based access control & MCP permissions | | [docs/SPEAKERRECOGNITION.md](docs/SPEAKERRECOGNITION.md) | Speaker recognition | | [docs/OUTPUTROUTING.md](docs/OUTPUTROUTING.md) | Audio output routing | | [docs/PROACTIVENOTIFICATIONS.md](docs/PROACTIVENOTIFICATIONS.md) | Webhook notifications & proactive alerts | | [docs/MULTILANGUAGE.md](docs/MULTILANGUAGE.md) | Internationalization (i18n) | | [docs/WAKEWORDCONFIGURATION.md](docs/WAKEWORDCONFIGURATION.md) | Wake word detection setup | | [docs/EXTERNALOLLAMA.md](docs/EXTERNALOLLAMA.md) | External Ollama instances | | [docs/SATELLITEMONITORING.md](docs/SATELLITEMONITORING.md) | Satellite monitoring | | [docs/SATELLITEOTAUPDATES.md](docs/SATELLITEOTAUPDATES.md) | Satellite over-the-air updates | | [docs/AUDIOCAPTURE4MIC.md](docs/AUDIOCAPTURE4MIC.md) | ReSpeaker 4-Mic array support | | [CLAUDE.md](CLAUDE.md) | Developer reference (architecture, patterns) |

Disclaimer

Renfield is an independent open-source project. It is not affiliated with, endorsed by, or connected to any third party, organization, company, or brand with the same or similar name.

Acknowledgments

• Ollama — Local LLM inference
• Whisper — Speech-to-text
• Piper — Text-to-speech
• SpeechBrain — Speaker recognition (ECAPA-TDNN)
• IBM Docling — Document processing for RAG
• pgvector — Vector similarity search
• Home Assistant — Smart home platform
• n8n — Workflow automation
• SearXNG — Metasearch engine
• OpenWakeWord — Wake word detection

License

MIT License — see [LICENSE](LICENSE) file.

Contributing

Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) and check the good first issues.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ebongard
• Source: ebongard/renfield
• License: MIT

Install and usage instructions live in the source repository linked above.