Llm Council

LLM Council Core

A multi-LLM deliberation system where multiple LLMs collaboratively answer questions through peer review and synthesis. Available as a Python library, MCP server, or HTTP API.

What is This?

Instead of asking a single LLM for answers, this MCP server:

1. Stage 1: Sends your question to multiple LLMs in parallel (GPT, Claude, Gemini, Grok, etc.)
2. Stage 2: Each LLM reviews and ranks the other responses (anonymized to prevent bias)
3. Stage 3: A Chairman LLM synthesizes all responses into a final, high-quality answer

Quick Deploy

Deploy your own LLM Council instance:

| Platform | Deploy | Best For | |----------|--------|----------| | Railway | [](https://railway.com/deploy/llm-council?referralCode=K9dsYj) | Production, webhooks | | Render | [](https://render.com/deploy?repo=https://github.com/amiable-dev/llm-council) | Evaluation, free tier |

Required Environment Variables:

• OPENROUTER_API_KEY - Your OpenRouter API key
• LLM_COUNCIL_API_TOKEN - A secure token for API authentication (generate with openssl rand -hex 16)

> Note: Railway is recommended for n8n integration (no cold-start). Render Free tier spins down after 15 minutes which may cause webhook timeouts.

For detailed deployment instructions, see the Deployment Guide.

Credits & Attribution

This project is a derivative work based on the original llm-council by Andrej Karpathy.

Karpathy's original README stated: > "I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like."

...the irony of producing a derivative work that packages the core concept for broader use via the Model Context Protocol!

Installation

pip install "llm-council-core[mcp]"

For core library only (no MCP server):

pip install llm-council-core

Setup

1. Choose Your Gateway

The council supports three gateway options for accessing LLMs:

| Gateway | Best For | API Keys Needed | |---------|----------|-----------------| | OpenRouter (default) | Easiest setup, 100+ models via single key | OPENROUTER_API_KEY | | Requesty | BYOK mode, analytics, cost tracking | REQUESTY_API_KEY + provider keys | | Direct | Maximum control, direct provider APIs | Provider keys (Anthropic, OpenAI, Google) |

Quick Start (OpenRouter):

# Sign up at openrouter.ai and get your API key
export OPENROUTER_API_KEY="sk-or-v1-..."

Direct Provider Access:

# Use your existing provider API keys directly
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."
export LLM_COUNCIL_DEFAULT_GATEWAY=direct

Requesty with BYOK:

export REQUESTY_API_KEY="..."
export ANTHROPIC_API_KEY="sk-ant-..."  # Your own key, routed through Requesty
export LLM_COUNCIL_DEFAULT_GATEWAY=requesty

2. Store Your API Keys Securely

Choose one of these options (in order of recommendation):

Option A: System Keychain (Most Secure)

Store keys encrypted in your OS keychain:

# Install with keychain support
pip install "llm-council-core[mcp,secure]"

# Store key securely (prompts for key, no echo)
llm-council setup-key

# For CI/CD automation, pipe from stdin:
echo "$OPENROUTER_API_KEY" | llm-council setup-key --stdin

Option B: Environment Variables

Set in your shell profile (~/.zshrc, ~/.bashrc):

# OpenRouter (default gateway)
export OPENROUTER_API_KEY="sk-or-v1-..."

# Or use direct provider APIs
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."

Option C: Environment File

Create a .env file (ensure it's in .gitignore):

# For OpenRouter
echo "OPENROUTER_API_KEY=sk-or-v1-..." > .env

# Or for direct APIs
cat > .env  **Security Note**: Never put API keys in command-line arguments or JSON config files that might be committed to version control.

### 3. Customize Models (Optional)

You can customize which models participate in the council using three methods (in priority order):

#### Option 1: Environment Variables (Recommended)

```bash
# Comma-separated list of council models
export LLM_COUNCIL_MODELS="openai/gpt-4,anthropic/claude-3-opus,google/gemini-pro"

# Chairman model (synthesizes final response)
export LLM_COUNCIL_CHAIRMAN="anthropic/claude-3-opus"

Option 2: YAML Configuration (Recommended)

Create llm_council.yaml in your project root or ~/.config/llm-council/llm_council.yaml:

council:
  # Tier configuration (ADR-022)
  tiers:
    default: high
    pools:
      quick:
        models:
          - openai/gpt-4o-mini
          - anthropic/claude-3-5-haiku-20241022
        timeout_seconds: 30
      balanced:
        models:
          - openai/gpt-4o
          - anthropic/claude-3-5-sonnet-20241022
        timeout_seconds: 90
      high:
        models:
          - openai/gpt-4o
          - anthropic/claude-opus-4-7
          - google/gemini-3-pro
        timeout_seconds: 180

  # Triage configuration (ADR-020)
  triage:
    enabled: false
    wildcard:
      enabled: true
    prompt_optimization:
      enabled: true

  # Gateway configuration (ADR-023, ADR-025a)
  gateways:
    default: openrouter
    fallback:
      enabled: true
      chain: [openrouter, ollama]  # Can use Ollama as fallback

    # Provider-specific configuration
    providers:
      ollama:
        enabled: true
        base_url: http://localhost:11434
        timeout_seconds: 120.0
        hardware_profile: recommended  # minimum|recommended|professional|enterprise

      openrouter:
        enabled: true
        base_url: https://openrouter.ai/api/v1/chat/completions

  # Webhook notifications (ADR-025a)
  webhooks:
    enabled: false  # Opt-in
    timeout_seconds: 5.0
    max_retries: 3
    https_only: true
    default_events:
      - council.complete
      - council.error

  observability:
    log_escalations: true

Priority: YAML config > Environment variables > Defaults

Option 3: JSON Configuration (Legacy)

Create ~/.config/llm-council/config.json:

{
  "council_models": [
    "openai/gpt-4-turbo",
    "anthropic/claude-3-opus",
    "google/gemini-pro",
    "meta-llama/llama-3-70b-instruct"
  ],
  "chairman_model": "anthropic/claude-3-opus",
  "synthesis_mode": "consensus",
  "exclude_self_votes": true,
  "style_normalization": false,
  "max_reviewers": null
}

Option 4: Use Defaults

If you don't configure anything, these defaults are used:

• Council: GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, Grok 4
• Chairman: Gemini 3 Pro
• Mode: consensus
• Self-vote exclusion: enabled

Finding Models:

• OpenRouter: openrouter.ai/models
• Anthropic: docs.anthropic.com/models
• OpenAI: platform.openai.com/docs/models
• Google: ai.google.dev/gemini-api/docs/models

Usage

With Claude Code

# First, store your API key securely (one-time setup)
llm-council setup-key

# Then add the MCP server (key is read from keychain or environment)
claude mcp add --transport stdio llm-council --scope user -- llm-council

Then in Claude Code:

Consult the LLM council about best practices for error handling

With Claude Desktop

First ensure your API key is available (via keychain, environment variable, or .env file).

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "llm-council": {
      "command": "llm-council"
    }
  }
}

> Note: No env block needed—the key is resolved from your system keychain or environment automatically.

With Other MCP Clients

Any MCP client can use the server by running:

llm-council

Available Tools

consult_council

Ask the LLM council a question and get synthesized guidance.

Arguments:

• query (string, required): The question to ask the council
• confidence (string, optional): Response quality level (default: "high")
• "quick": Fast models (mini/flash/haiku), ~30 seconds - fast responses for simple questions
• "balanced": Mid-tier models (GPT-4o, Sonnet), ~90 seconds - good balance of speed and quality
• "high": Full council (Opus, GPT-4o), ~180 seconds - comprehensive deliberation
• "reasoning": Deep thinking models (GPT-5.2, o1, DeepSeek-R1), ~600 seconds - complex reasoning
• include_details (boolean, optional): Include individual model responses and rankings (default: false)
• verdict_type (string, optional): Type of verdict to render (default: "synthesis")
• "synthesis": Free-form natural language synthesis
• "binary": Go/no-go decision (approved/rejected) with confidence score
• "tie_breaker": Chairman resolves deadlocked decisions
• include_dissent (boolean, optional): Extract minority opinions from Stage 2 (default: false)

Example:

Use consult_council to ask: "What are the trade-offs between microservices and monolithic architecture?"

Example with confidence level:

Use consult_council with confidence="quick" to ask: "What's the syntax for a Python list comprehension?"

Example with Jury Mode (binary verdict):

Use consult_council with verdict_type="binary" and include_dissent=true to ask: "Should we approve this PR that adds caching?"

council_health_check

Verify the council is working before expensive operations. Returns API connectivity status, configured models, and estimated response times.

Arguments: None

Returns:

• api_key_configured: Whether an API key was found
• key_source: Where the key came from ("environment", "keychain", or "config_file")
• council_size: Number of models in the council
• estimated_duration: Expected response times for each confidence level
• ready: Whether the council is ready to accept queries

Example:

Run council_health_check to verify the LLM council is working

How It Works

The council uses a multi-stage process inspired by ensemble methods and peer review:

User Query
    ↓
┌─────────────────────────────────────────────┐
│ STAGE 1: Independent Responses              │
│ • All council models queried in parallel    │
│ • No knowledge of other responses           │
│ • Graceful degradation if some fail         │
└─────────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────────┐
│ STAGE 1.5: Style Normalization (optional)   │
│ • Rewrites responses in neutral style       │
│ • Removes AI preambles and fingerprints     │
│ • Strengthens anonymization                 │
└─────────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────────┐
│ STAGE 2: Anonymous Peer Review              │
│ • Responses labeled A, B, C (randomized)    │
│ • XML sandboxing prevents prompt injection  │
│ • JSON-structured rankings with scores      │
│ • Self-votes excluded from aggregation      │
└─────────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────────┐
│ STAGE 3: Chairman Synthesis                 │
│ • Receives all responses + rankings         │
│ • Consensus mode: single best answer        │
│ • Debate mode: highlights disagreements     │
└─────────────────────────────────────────────┘
    ↓
Final Response + Metadata

This approach helps surface diverse perspectives, identify consensus, and produce more balanced, well-reasoned answers.

Advanced Features

Self-Vote Exclusion

By default, each model's vote for its own response is excluded from the aggregate rankings. This prevents self-preference bias.

export LLM_COUNCIL_EXCLUDE_SELF_VOTES=true  # default

Synthesis Modes

Consensus Mode (default): Chairman synthesizes a single best answer.

Debate Mode: Chairman highlights areas of agreement, key disagreements, and trade-offs between perspectives.

export LLM_COUNCIL_MODE=debate

Style Normalization (Stage 1.5)

Optional preprocessing that rewrites all responses in a neutral style before peer review. This strengthens anonymization by removing stylistic "fingerprints" that might allow models to recognize each other.

export LLM_COUNCIL_STYLE_NORMALIZATION=true
export LLM_COUNCIL_NORMALIZER_MODEL=google/gemini-2.0-flash-001  # fast/cheap

Stratified Sampling (Large Councils)

For councils with more than 5 models, you can limit the number of reviewers per response to reduce API costs (O(N²) → O(N×k)):

export LLM_COUNCIL_MAX_REVIEWERS=3

Reliability Features

The council includes built-in reliability features for long-running operations:

Tiered Timeouts: Graceful degradation under time pressure:

• Per-model soft deadline: 15s (start planning fallback)
• Per-model hard deadline: 25s (abandon slow model)
• Global synthesis trigger: 40s (must start synthesis)
• Response deadline: 50s (must return something)

Partial Results: If some models timeout, the council returns results from the models that responded, with a clear warning indicating which models were excluded.

Confidence Levels: Use the confidence parameter to trade off speed vs. thoroughness:

• quick: ~20-30 seconds (fastest models)
• balanced: ~45-60 seconds (most models)
• high: ~60-90 seconds (full council, default)

Progress Feedback: During deliberation, progress updates show which models have responded and which are still pending:

✓ claude-opus-4.7 (1/4) | waiting: gpt-5.1, gemini-3-pro, grok-4
✓ gemini-3-pro (2/4) | waiting: gpt-5.1, grok-4

Jury Mode (ADR-025b)

Transform the council from a "summary generator" to a "decision engine" with structured verdicts.

Verdict Types:

| Mode | Output | Use Case | |------|--------|----------| | synthesis (default) | Free-form synthesis | General questions, exploration | | binary | approved/rejected + confidence | CI/CD gates, PR reviews, policy checks | | tie_breaker | Chairman decides on deadlock | Contentious decisions |

Binary Verdict Mode:

Returns structured go/no-go decisions with confidence scores:

# MCP Tool
result = await consult_council(
    query="Should we approve this architectural change?",
    verdict_type="binary"
)

# Returns:
{
  "verdict": "approved",      # or "rejected"
  "confidence": 0.75,         # 0.0-1.0 based on council agreement
  "rationale": "Council agreed the change improves modularity..."
}

Tie-Breaker Mode:

When the council is deadlocked (top scores within 0.1 of each other), the chairman casts the deciding vote:

result = await consult_council(
    query="Option A vs Option B for caching strategy?",
    verdict_type="binary"
)

# If deadlocked, returns:
{
  "verdict": "approved",
  "confidence": 0.60,
  "rationale": "Chairman resolved deadlock based on...",
  "deadlocked": true  # Flag indicates tie-breaker was needed
}

Constructive Dissent:

Extract minority opinions from Stage 2 peer reviews:

result = await consult_council(
    query="Should we migrate to microservices?",
    verdict_type="binary",
    include_dissent=True
)

# Returns:
{
  "verdict": "approved",
  "confidence": 0.70,
  "rationale": "Majority supports migration...",
  "dissent": "Minority perspective: One reviewer noted scalability concerns with current team size."
}

Dissent Algorithm:

1. Collect all scores from Stage 2 peer reviews
2. Calculate median and standard deviation per response
3. Identify outliers (score = 0.7:

return True, verdict.get("rationale") else: r

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: amiable-dev
• Source: amiable-dev/llm-council
• License: MIT
• Homepage: https://llm-council.dev/

Install and usage instructions live in the source repository linked above.