Moe Sovereign

MoE Sovereign

A Self-Hosted Multi-Model Orchestrator with Template-Based Expert Routingfor Sovereign AI Infrastructure

[](LICENSE) [](#deployment-targets) [](#) [](https://docs.moe-sovereign.org) [](#funded-research-eurohpc-lumi-g)

Documentation • Website • Issues • [Changelog](CHANGELOG.md)

---

Motivation

Commercial AI APIs process every request on infrastructure the customer neither owns nor can inspect. Training-data extraction, prompt logging, and retroactive policy changes are documented incidents. The European regulatory framework --- in particular GDPR Articles 25 and 32 --- mandates data protection by design, a requirement difficult to discharge with an opaque black box in a foreign jurisdiction.

MoE Sovereign is a fully self-hosted multi-model orchestrator with template-based expert routing that runs entirely on your own hardware. No data leaves your network. No cloud dependency. No vendor lock-in.

---

Architecture

flowchart TD
    subgraph Clients["Client Layer"]
        CC["Claude Code"]
        OW["Open WebUI"]
        API["Any OpenAI Client"]
    end

    subgraph Orchestrator["MoE Orchestrator (LangGraph)"]
        direction TB
        Cache{"L0/L1 CacheValkey + ChromaDB"}
        Planner["Plannerphi4:14b"]
        
        subgraph Experts["Parallel Expert LLMs"]
            E1["Code Reviewer"]
            E2["Researcher"]
            E3["Domain Expert"]
        end

        MCP["28 MCP ToolsAST-Whitelist + PPTX"]
        Graph["Neo4j GraphRAG"]
        Judge["Judge / Mergerllama3.3:70b"]
        GapCheck{"Gap DetectorCOMPLETE?"}
        Replan["Agentic Re-Planup to 3 rounds"]
    end

    subgraph Storage["Persistence Layer"]
        Neo4j[("Neo4jKnowledge Graph")]
        Chroma[("ChromaDBVector Cache")]
        Kafka["KafkaEvent Stream"]
        Valkey[("ValkeyState & Sessions")]
        PG[("PostgreSQLUsers & Checkpoints")]
    end

    Clients -->|"/v1/chat/completions/v1/messages/v1/responses"| Cache
    Cache -->|Miss| Planner
    Cache -->|Hit| Response
    Planner --> Experts
    Experts --> MCP
    MCP --> Graph
    Graph --> Judge
    Judge --> GapCheck
    GapCheck -->|"COMPLETE"| Response["Response"]
    GapCheck -->|"NEEDS_MORE_INFO"| Replan
    Replan -->|"inject gap context"| Planner
    Response -->|Ingest| Kafka
    Kafka --> Neo4j
    Response -->|Cache Write| Chroma
    Judge -.->|"Retry onlow score"| Planner

    style Orchestrator fill:#f0f4ff,stroke:#4a6fa5
    style Experts fill:#e8f5e9,stroke:#388e3c
    style Storage fill:#fff8e1,stroke:#f9a825

Pipeline Stages

| Stage | Description | |:---:|---| | 1. Cache | L0 query-hash (Valkey, 30 min TTL), L1 semantic similarity (ChromaDB, cosine < 0.15), and a conservative knowledge-bypass tier: similar-but-not-exact queries skip the LLM when the prior answer was high-confidence and still fresh (cosine < 0.25, confidence ≥ 0.85, within TTL) | | 2. Planner | Decomposes request into 1--4 subtasks with expert category assignment | | 3. Experts | T1 models (≤20B) screen with confidence gating; T2 (24--80B) engage only on low confidence | | 4. Tools | 28 MCP precision tools (math, subnet, date, legal, PPTX) via AST-whitelist --- zero hallucination | | 5. GraphRAG | Neo4j context enrichment with domain-scoped entity filters and trust-score decay. CAG layer intercepts static compliance domains (BAIT, VAIT, DORA, KRITIS) before the Neo4j query and injects pre-loaded authoritative text directly. Corrective RAG gate (Yan et al. 2024) scores each retrieved entity for query relevance and discards low-signal results before injection. Episode hints from past similar tasks are appended as routing context | | 6. Judge | Synthesises expert outputs, evaluates quality, retries on failure (up to 3 attempts) | | 7. Agentic Re-Plan | Lightweight gap detector checks completeness; if unresolved, injects findings into a new planner round (up to 3 agentic iterations) | | 8. Ingest | Validated knowledge flows back into Neo4j via Kafka for graph accumulation acceleration |

Module Structure

The orchestrator codebase is organised into focused packages. main.py is a thin entry point (~1 500 LOC) holding the FastAPI app, lifespan, middleware, and graph wiring. All domain logic lives in dedicated packages:

moe-infra/
├── main.py                    # FastAPI app, lifespan, middleware, graph wiring (~1 500 LOC)
├── config.py                  # All os.getenv() — typed config constants
├── state.py                   # Shared mutable globals (redis_client, _userdb_pool, …)
├── prompts.py                 # Static prompt text + routing detection regexes
├── metrics.py                 # Single Prometheus registry
├── parsing.py                 # Stateless parsers: JSON extraction, confidence, history truncation
├── context_budget.py          # Per-model context-window estimation
│
├── routes/                    # FastAPI APIRouters (one per concern)
│   ├── health.py              # /health, /metrics
│   ├── watchdog.py            # /api/watchdog/*, Starfleet feature toggles
│   ├── mission_context.py     # /api/mission-context
│   ├── graph.py               # /graph/*
│   ├── feedback.py            # /v1/feedback, /v1/memory/ingest
│   ├── admin_*.py             # Benchmark, ontology, stats admin endpoints
│   ├── models.py              # /v1/models
│   ├── ollama_compat.py       # /api/* (Ollama protocol)
│   └── anthropic_compat.py    # /v1/messages, /v1/responses, /v1/chat/completions
│
├── services/                  # Business logic — no FastAPI imports
│   ├── auth.py                # OIDC + API key validation + budget enforcement
│   ├── tracking.py            # Usage logging, request lifecycle, budget counters
│   ├── routing.py             # Expert template + per-template prompt resolution
│   ├── templates.py           # Expert template + Claude Code profile loading
│   ├── llm_instances.py       # ChatOpenAI singletons (judge, planner, ingest, search)
│   ├── inference.py           # Node selection, fallback chain, Thompson sampling
│   ├── helpers.py             # Progress reports, semantic memory, self-evaluation
│   ├── skills.py              # Server-side skill resolution + ADMIN_APPROVED hard-lock
│   ├── healer.py              # Ontology gap-healer (one-shot + dedicated subprocess)
│   ├── kafka.py               # Fire-and-forget Kafka publish helper
│   └── pipeline/              # OpenAI / Anthropic / Ollama / Responses API handlers
│       ├── chat.py            # OpenAI chat completions
│       ├── anthropic.py       # Anthropic Messages API + tool/MoE/reasoning handlers
│       ├── ollama.py          # Ollama-protocol streaming wrappers
│       └── responses.py       # OpenAI Responses API
│
├── graph/                     # LangGraph node implementations
│   ├── router_nodes.py        # cache_lookup, semantic_router, fuzzy_router, _route_cache
│   ├── tool_nodes.py          # mcp_node, graph_rag_node, math_node_wrapper
│   ├── planner.py             # planner_node + plan sanitization + topological levels
│   ├── expert.py              # expert_worker (parallel expert execution)
│   ├── research.py            # research_node + research_fallback + domain extraction
│   └── synthesis.py           # merger_node, thinking_node, resolve_conflicts_node, critic_node
│
├── pipeline/
│   ├── __init__.py            # LangGraph graph builder — assembles nodes into the pipeline DAG
│   └── state.py               # AgentState TypedDict (67 fields across 3 categories)
│
├── web_search.py              # SearXNG integration with domain-reliability scoring
├── math_node.py               # SymPy-backed math node (solve, integrate, differentiate)
├── graph_rag/                 # GraphRAG query, entity linking, ontology, corrections
├── federation/                # Push / pull federation client to MoE Libris hubs
├── mcp_server/                # 28 MCP precision tools (AST-whitelisted)
├── admin_ui/                  # Admin backend: experts, users, budgets, cleanup manager
├── prompts/systemprompt/      # 15 expert system prompts (English, "Respond in German.")
├── tests/                     # 195 unit + integration + smoke tests (all green)
└── benchmarks/                # Overnight benchmark suite, GAIA runner, result injection

The orchestrator started as an 11 190-line monolith in main.py. A 14-phase split (Q2 2026) decomposed it into the structure above without a single behavioural change — every phase ended with the full test suite green. See docs/ARCHITECTURE.md for the detailed module map.

---

Key Capabilities

A) Core AI & Orchestration

| | Capability | Description | |:---:|---|---| | 1 | Deterministic Expert Routing | Versioned, auditable templates --- not a probabilistic black box | | 2 | Two-Tier Escalation | T1 screens fast; T2 engages only when needed --- incl. a low-confidence rescue that lets a trivial query escalate to T2 when T1 returns weak/empty output (TRIVIAL_LOW_CONF_RESCUE_ENABLED) | | 3 | Neo4j GraphRAG | Trust-score self-healing, contradiction detection, domain-scoped filters | | 4 | Community Knowledge Bundles | Export/import learned knowledge as JSON-LD with regex-based privacy scrubbing (PII, secrets, hostnames) | | 5 | 51 MCP Precision Tools | AST-whitelisted --- 100% accuracy on deterministic tasks; includes wikidatasparql, pubmedsearch, crossreflookup, openalexsearch, duckduckgosearch, webbrowser (Splash JS rendering), waybackfetch, githubsearchissues with fuzzy label resolution | | 6 | VRAM-Aware Scheduling | Per-node VRAM limits, warm-model affinity, sticky sessions | | 8 | Claude Code Integration | Full Anthropic Messages API with 6 profiles and streaming thinking blocks | | 9 | Deployment Flexibility | One OCI image → LXC (tested), Docker Compose (tested), Podman rootless (tested), Helm/K8s (architecturally prepared, community validation requested) | | 10 | 9.3× Accumulation Speedup | 707 s → 76 s latency over 5 benchmark epochs | | 12 | Agentic Re-Planning Loop | After each synthesis the Judge checks completeness; unresolved gaps trigger a focused re-plan with injected context --- up to 3 autonomous iterations per request; domain-aware search cache prevents result poisoning across iterations | | 13 | PowerPoint Generation | MCP generate_pptx tool creates fully formatted .pptx presentations from structured content and delivers them as signed Garage (S3) download links | | 18 | Dynamic Sequential/Parallel Experts | Planner tasks support depends_on for multi-hop chains (e.g. find author → find their papers). Independent tasks run in parallel; dependent tasks execute sequentially with result injection via {result_of:id} placeholders | | 19 | Adaptive Context Budget | Context window limits per model auto-scale web-research blocks and GraphRAG budget. Fallback models (phi4:14b-fp16 16K, qwen3.6:35b 32K) receive proportionally smaller context slices | | 20 | GraphRAG On-Demand | Neo4j queries skipped for external research questions (papers, APIs, media) — only runs for internal knowledge queries or when the plan includes a knowledgehealing task | | 21 | OpenAI Responses API (/v1/responses) | Full Responses API streaming with correct SSE events (sequence_number, output_index, content_index) — enables Codex CLI, Continue.dev, and any OpenAI Responses API compatible agent out of the box | | 23 | Chess Analysis via Lichess | MCP tool chess_analyze_position queries Lichess cloud Stockfish (342M positions, depth 20–99) for best moves given a FEN string — no local engine required | | 25 | Formal Logic State Layer | Three-tier algebraic logic over the LangGraph state (de Vries 2007): paraconsistent conflict registry tolerates contradictory expert outputs without pipeline failure; intuitionistic ConstructiveProof[T] marks LLM claims as ⊥ until executor-verified; fuzzy T-norm routing replaces binary flags with continuous confidence scores — Gödel min and Łukasiewicz max(0,a+b−1) conjunctions configurable via env | | 26 | AIC Complexity Estimation | zlib compressibility as a Kolmogorov complexity proxy (Kolmogorov 1965) acts as a tie-breaker in estimate_complexity() — information-dense prompts (ratio This feature group requires the optional moe-codex > enterprise stack (Apache NiFi, Marquez/OpenLineage, lakeFS). It is not part of the > moe-sovereign core and is deployed as a separate compose stack. > See the moe-codex repository for setup instructions.

| | Capability | Description | |:---:|---|---| | 29 | OpenLineage Data Lineage (Marquez) | Five pipeline hook points (/v1/chat/completions, /v1/messages, /v1/responses, merger_node, kafka_ingest) emit OpenLineage 2.0.2 START/COMPLETE/FAIL events to a Marquez backend — fire-and-forget, no-op when MARQUEZ_URL is empty. Palantir Foundry-comparable lineage visibility for every MoE pipeline run | | 30 | Enterprise Stack Dashboard | Admin UI /enterprise page surfaces NiFi, Marquez and lakeFS reachability with live latency probes, plus the most recent OpenLineage runs from Marquez. Auto-refreshes every 30 s; gracefully hides when INSTALL_ENTERPRISE_DATA_STACK=false | | 31 | lakeFS Bundle Versioning | Every successful /graph/knowledge/import archives the JSON-LD bundle as a content-addressed commit on the moe-knowledge lakeFS repository — git-style audit log queryable via /api/enterprise/versioning/log, point-in-time bundle download via services.versioning.get_bundle_at() for rollback. Fire-and-forget; no-op when LAKEFS_ENDPOINT is empty | | 32 | NiFi ETL Submission | Knowledge events (Kafka ingest + bundle import) are forwarded to a configurable NiFi ListenHTTP processor (NIFI_INGEST_URL), so downstream NiFi flows can fan out to S3/Solr/Elastic/Snowflake without orchestrator changes. JSON in body, MoE metadata as X-MoE-* FlowFile attributes; admin dashboard surfaces NiFi system diagnostics (uptime, heap, threads, version) at /api/enterprise/etl/status | | 33 | Unified Data Catalog | Admin UI /catalog page aggregates datasets across all three back-ends in one searchable, source-filterable table — Marquez datasets per namespace, Neo4j entity-domain breakdown (entities/relations/syntheses), and lakeFS repositories with commit counts. Foundry-Catalog-equivalent cross-source browsing without leaving the admin UI | | 34 | Branch-based Approval Workflow | POST /v1/graph/knowledge/import/pending stages a bundle on a lakeFS pending/- branch instead of Neo4j; admins review pending bundles in /approval, then approve (= Neo4j import + lakeFS merge to main) or reject (= branch delete). Adds an explicit gate before any external knowledge enters the live graph | | 35 | Read-only Cypher Explorer | Admin UI /explorer page exposes an in-page Cypher editor restricted to read mode: regex-blacklist rejects CREATE/DELETE/SET/MERGE/REMOVE/DROP/ALTER/GRANT/REVOKE/FOREACH before the query reaches Neo4j, plus the driver runs in READ_ACCESS mode. Includes preset queries and a deep-link to the standalone Neo4j Browser | | 36 | Data Health Drift Detection | Every successful knowledge-bundle import is wrapped in a stats snapshot — services/data_health.compute_drift() flags entity_dedup_suppressed, zero_entities_added, entity_count_shrank, entity_overshoot, relation_overshoot, relation_to_entity_explosion. Events land in Redis moe:data_health:events (capped 500) and surface on the Enterprise dashboard with severity pills (ok / info / warn / crit). Threshold tunable via DATA_HEALTH_DRIFT_THRESHOLD (default 0.3) | | 37 | Embedded JupyterLite Notebook |

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: h3rb3rn
• Source: h3rb3rn/moe-sovereign
• License: Apache-2.0
• Homepage: https://moe-sovereign.org

Install and usage instructions live in the source repository linked above.