Llmrouter

LLM Router

A unified gateway that aggregates multiple LLM providers behind a single endpoint. Clients use their preferred protocol (OpenAI or Anthropic Messages) and the gateway handles routing, protocol translation, and load balancing.

Features

• Multi-Provider: OpenAI, Claude, Gemini, Ollama, Mistral, ZAi — configure once, route by model name
• Protocol Translation: Clients speak OpenAI or Anthropic Messages; the gateway translates as needed, and also exposes a full Ollama-compatible API under /ollama
• Weight-Based Load Balancing: Distribute load across providers with configurable weights
• Smart Routing: Request the auto model and a Scriptling script picks the best provider/model based on tags, load, and request content
• MCP Aggregator: Combine tools, resources, and prompts from multiple remote MCP servers with namespace isolation, OAuth support, and per-tool visibility / allow / deny filtering
• Chat UI: Built-in interface at /chat with personas, conversation history, slash commands, @prompt / @resource menus, live MCP tool calling, markdown rendering, and skill:// resources auto-surfaced to the model
• Personas: System prompts, default models, and generation parameters — defined in the config file or managed through the admin UI
• Admin UI: Optional web interface at /admin to manage providers, personas, and MCP servers (create / edit / delete, enable / disable), browse models (with rescan) and tools, and test-run MCP tools directly from the browser
• Responses API: OpenAI-compatible responses storage (emulated for all providers)
• Conversations API: n8n-compatible conversation management
• Optional Auth: Bearer token protection for all endpoints

Installation

Homebrew (macOS & Linux)

brew tap paularlott/tap
brew install llmrouter

Download from GitHub Releases

Download the latest release for your platform from github.com/paularlott/llmrouter/releases:

| Platform | Architecture | Download | | -------- | --------------------- | ----------------------------- | | macOS | Intel (AMD64) | llmrouter-darwin-amd64.zip | | macOS | Apple Silicon (ARM64) | llmrouter-darwin-arm64.zip | | Linux | AMD64 | llmrouter-linux-amd64.zip | | Linux | ARM64 | llmrouter-linux-arm64.zip | | Windows | AMD64 | llmrouter-windows-amd64.zip | | Windows | ARM64 | llmrouter-windows-arm64.zip |

Extract the archive and place the binary in your PATH.

Build from Source

git clone https://github.com/paularlott/llmrouter.git
cd llmrouter
go build -o llmrouter .
./llmrouter server
./llmrouter -config /path/to/config.toml server

Configuration

[server]
host = "0.0.0.0"
port = 12345
token = "your-secret-token"   # Optional bearer token
admin_password = "admin123"   # Optional: enables admin UI at /admin
storage_path = "./data"  # Omit for memory-only storage (under [server])

[logging]
level = "info"    # trace | debug | info | warn | error
format = "console" # console | json

[responses]
ttl_days = 30

[conversations]
ttl_days = 30

[[providers]]
name = "openai"
provider = "openai"           # openai | claude | gemini | ollama | mistral | zai
token = "sk-..."
enabled = true
weight = 1.0                  # 0.0-2.0, default 1.0; higher = preferred
tags = ["capable", "expensive"]  # optional tags for smart routing

[providers.model_tags]        # optional per-model tags
"gpt-4o"      = ["capable", "expensive"]
"gpt-4o-mini" = ["fast", "cheap"]

[providers.model_aliases]     # optional alias -> real model name, resolved per-provider
"gpt4"     = "gpt-4o"
"gpt4-mini" = "gpt-4o-mini"

[[providers]]
name = "anthropic"
provider = "claude"
token = "sk-ant-..."
enabled = true
model_allowlist = ["claude-opus-4-5", "claude-sonnet-4-5"]  # Required for Claude
tags = ["capable"]

[providers.model_aliases]     # same alias "fast", different real model on this provider
"fast" = "claude-sonnet-4-5"

[providers.model_tags]
"claude-opus-4-5"   = ["capable", "expensive"]
"claude-sonnet-4-5" = ["capable", "fast"]

[[providers]]
name = "google"
provider = "gemini"
token = "your-google-key"
enabled = true
model_allowlist = ["gemini-2.5-flash-lite"]  # Optional: restrict to specific models
models = ["gemini-2.5-flash-lite"]           # Optional: override model list entirely (still health-checks the API)

[[providers]]
name = "local"
provider = "ollama"
base_url = "http://localhost:11434/v1"
enabled = true

[smart_routing]
enabled = false
script = "router.py"  # Scriptling script for routing decisions
default_model = "mistralai/ministral-3-3b"  # Fallback if script returns nothing
libdir = "./router_libs"  # Optional: directory of .py script libraries

[mcp]
[[mcp.remote_servers]]
namespace = "tools"
url = "https://tools.example.com/mcp"
token = "secret"
tool_visibility = "native"    # native | discoverable
tool_allowlist = ["search", "query"]  # Optional: only these tools are enabled
tool_denylist = ["delete"]           # Optional: these tools are disabled

Provider Types

| Provider | Default Base URL | Embeddings | Model Discovery | | --------- | --------------------------------------------- | ---------- | ---------------- | | openai | https://api.openai.com/v1 | Yes | Auto | | claude | https://api.anthropic.com/v1 | No | Must specify | | gemini | https://generativelanguage.googleapis.com/... | Yes | Auto | | ollama | https://ollama.com/v1/ | Yes | Auto | | mistral | https://api.mistral.ai/v1 | Yes | Auto | | zai | https://api.z.ai/api/paas/v4/ | Yes | Auto |

base_url is optional — each provider has a built-in default. Set it to override (e.g. local LM Studio).

model_allowlist restricts the provider to only the listed models. For Claude this is required (no discovery API). For other providers it is optional.

model_denylist excludes specific models from auto-discovery. Ignored when model_allowlist is set.

models overrides the model list entirely — the provider's /models API is still called (for health checks) but its response is discarded and the configured list is used instead. Useful for providers that return no models or an incomplete list.

model_aliases maps short or friendly names to real model IDs. Aliases appear in /v1/models alongside real models and support the same weight-based load balancing and round-robin behaviour. When multiple providers define the same alias, each provider maps it to its own real model — requests via that alias are distributed across all of them, and each provider receives its own real model name.

Smart Routing

When a client requests the model name auto, the router runs a Scriptling script to pick the best provider and model. If the script returns nothing or fails, default_model is used.

libdir is an optional directory of .py files. Each file is registered as a script library (named after the file without the .py extension) and is available for import in the routing script. The directory is watched for changes — any modification triggers a full VM pool rebuild.

[smart_routing]
enabled = true
script = "router.py"  # Scriptling script for routing decisions
default_model = "mistralai/ministral-3-3b"

[smart_routing.vars]  # Optional key-value pairs exposed to the script
openai_key = "sk-..."
my_endpoint = "https://api.example.com"

The auto model appears in /v1/models so clients can discover it.

Provider and Model Tags

Tags are arbitrary strings assigned to providers and individual models. The routing script uses them to select the right provider/model for each request.

[[providers]]
name = "mistral"
provider = "mistral"
token = "..."
enabled = true
tags = ["fast", "cheap"]           # provider-level tags

[providers.model_tags]
"mistralai/ministral-3-3b"      = ["small", "fast", "cheap"]
"mistralai/mistral-small-latest" = ["small", "fast"]

The routing script can then query by tag:

import router

req = router.get_request()

# Route tool-heavy requests to a capable model
if router.is_chat_completion() and len(req["tools"]) > 0:
    models = router.models_by_tag("capable")
else:
    models = router.models_by_tag("cheap")

if models:
    router.set_model(models[0])

Use router.model_tags(model_id) to narrow a candidate list by a secondary tag:

import router

# Start broad: all "capable" models
candidates = router.models_by_tag("capable")

# Narrow: prefer "super_fast" within that set
fast = [m for m in candidates if "super_fast" in router.model_tags(m)]
cheap = [m for m in candidates if "cheap" in router.model_tags(m)]

models = fast or cheap or candidates
if models:
    router.set_model(models[0])

See [docs/scriptling-router-library.md](docs/scriptling-router-library.md) for the full script API reference.

Script Libraries

Every routing script has access to all Scriptling standard libraries (json, re, math, random, hashlib, base64, uuid, datetime, time, urllib, etc.) plus the following extended and Scriptling-specific libraries:

| Library | Description | | -------------------------- | --------------------------------------------- | | requests | HTTP client | | secrets | Cryptographically strong random numbers | | html.parser | HTML/XHTML parser | | logging | Logging to the router log | | yaml | YAML parsing | | toml | TOML parsing | | sys | System parameters | | scriptling.ai | AI/LLM client for OpenAI-compatible APIs | | scriptling.ai.agent | Agentic AI loop with automatic tool execution | | scriptling.mcp | MCP tool interaction | | scriptling.toon | TOON encoding/decoding | | scriptling.similarity | String matching and similarity utilities | | scriptling.net.resolve | DNS resolution (IP, SRV, srv+http URLs) | | scriptling.template.html | HTML template rendering | | scriptling.template.text | Text template rendering | | scriptling.runtime | Background tasks, KV store, sync primitives |

Filesystem access (os, pathlib, glob), subprocess execution, and wait_for are not available in routing scripts.

Script Variables

Use [smart_routing.vars] to pass tokens or other config to the script without hard-coding them:

[smart_routing.vars]
openai_key = "sk-..."

import vars
import scriptling.ai as ai

client = ai.Client("", api_key=vars.openai_key)

Weight-Based Load Balancing

When multiple providers serve the same model, the router selects using score = active_completions / weight. Lower score wins.

| Weight | Effect | | ------ | ------------------------------------- | | 0.0 | Last resort only | | 1.0 | Normal (default) | | 2.0 | Preferred — gets 2× the traffic share |

MCP Tool Visibility

| Mode | Behavior | | -------------- | --------------------------------------------------- | | native | Tools appear in tools/list, directly callable | | discoverable | Hidden from list, searchable via tool_search only |

MCP Tool Filtering

Tools from remote MCP servers can be filtered using tool_allowlist or tool_denylist:

• tool_allowlist: When defined, only the listed tools are enabled. All other tools are disabled.
• tool_denylist: When defined, all tools are enabled except those in the list.

Note: If both are defined, tool_allowlist takes precedence and tool_denylist is ignored.

[[mcp.remote_servers]]
namespace = "github"
url = "https://github.example.com/mcp"
token = "secret"
tool_visibility = "native"
tool_allowlist = ["search_repos", "get_issue", "create_issue"]  # Only these 3 tools are enabled

[[mcp.remote_servers]]
namespace = "slack"
url = "https://slack.example.com/mcp"
token = "secret"
tool_visibility = "native"
tool_denylist = ["delete_message", "ban_user"]  # All tools enabled except these 2

Scripting Tools

Define custom MCP tools using .toml (metadata) and .py (implementation) file pairs. Tools are loaded from a configurable directory and automatically reloaded when files change.

./llmrouter server --tools-dir ./tools --plugin-dir ./plugins --libpath ./libs

Directory structure:

tools/
├── calculator.toml
├── calculator.py
├── weather.toml
└── weather.py

Tool metadata (.toml):

description = "Calculate the sum of two numbers"
keywords = ["math", "add", "sum"]

[[parameters]]
name = "a"
type = "int"
description = "First number"
required = true

[[parameters]]
name = "b"
type = "int"
description = "Second number"
required = true

Tool implementation (.py):

import scriptling.mcp.tool as tool

a = tool.get_int("a")
b = tool.get_int("b")

result = a + b
tool.return_string(f"{a} + {b} = {result}")

Configuration:

[scripting]
tools_dir = "./tools"              # Directory containing .toml/.py tool pairs
plugin_dirs = ["./plugins"]        # Directories containing plugin executables
lib_paths = ["./libs"]             # Additional directories for scriptling libraries

Tools can be marked as discoverable = true to hide them from tools/list and make them searchable via tool_search only.

Chat UI

When admin_password is set and a personas directory is configured, a built-in chat interface is available at /chat with conversation history, personas, slash commands, @prompt / @resource menus, MCP tool calling, and markdown rendering.

./llmrouter server --personas-dir ./personas --commands-dir ./commands --resources-dir ./resources

Skills: Resources with a skill:// URI prefix are automatically surfaced to the LLM. On every chat request, the router queries the MCP server for skill:// resources and appends their names and descriptions to the persona's system prompt. A virtual tool (lmchatkit__get_skill) is injected into the tool list — the model calls it to retrieve a skill's full instructions on demand. The tool routes to the standard MCP ReadResource API, so skills work from any source (files, remote servers, or custom providers). The tool is auto-approved (no user prompt) since it's a read-only context fetch. Skills are transient — the stored conversation is not modified; the augmentation is recomputed on each request.

Example skill resource directory structure:

resources/
└── skill/
    ├── golang.md    → skill://golang.md
    └── testing.md   → skill://testing.md

MCP OAuth Authentication

For MCP servers that require OAuth authentication, configure the OAuth fields instead of token:

[[mcp.remote_servers]]
namespace = "my-oauth-service"
url = "https://api.example.com/mcp"
auth_type = "oauth"
oauth_client_id = "your-client-id"
oauth_token_url = "https://auth.example.com/token"
oauth_access_token = "current-access-token"
oauth_refresh_token = "refresh-token"  # Optional: for token refresh

API Endpoints

All endpoints are available under both /v1/

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: paularlott
• Source: paularlott/llmrouter
• License: MIT

Install and usage instructions live in the source repository linked above.