Llmrix Router

LLMRix Model Router

A production-oriented multi-model routing and orchestration framework for Java.

OpenAI · DeepSeek · OpenRouter · Semantic routing · Contextual bandits · Fugu orchestration

Why LLMRix • Architecture • Modules • Quick Start • HTTP Protocol • Server • Client

LLMRix exposes multiple model providers through stable provider-neutral ModelClient and modality-specific model interfaces. It selects an eligible model using declared operations, features, input modalities, quality, cost, latency, quota, and health signals, then applies bounded retries and cooldown without leaking routing complexity into application code.

Use it as an embedded Java SDK, a Spring Boot starter, or an OpenAI-compatible routing service.

> Project status: General Availability (1.0.2). The Java API and configuration model are production-ready, with Semantic Versioning strictly enforced. Published to Maven Central.

Why LLMRix

• Three first-class providers: OpenAI, DeepSeek, and OpenRouter over one validated OpenAI protocol transport.
• Policy separated from execution: strategies rank model targets; the executor owns timeout, retry, quota, and cooldown correctness.
• Streaming-safe candidate switching: the router can try another configured model before output begins and never replays after output begins.
• Local or distributed state: zero-infrastructure local mode and Redis-backed health, leases, RPM, and TPM for multi-instance deployments.
• OpenAI-compatible edge: Chat, Responses, Embeddings, Rerank, Audio, Images, Videos, Models, and SSE endpoints.
• Framework-neutral client: Orion provides a small Java client plus optional Spring Boot auto-configuration.
• Observable by design: lifecycle events, Micrometer metrics, Spring Observations, request IDs, and health indicators.
• Composable advanced routing: semantic routing, contextual bandits, online shadow traffic, evaluation, and Fugu-style iterative orchestration.

Architecture

Clients enter through embedded Java, Spring Boot, or OpenAI-compatible HTTP APIs. The provider-neutral core filters and ranks model targets, executes calls with reliability controls, shares runtime state, and emits telemetry. The framework owns routing semantics and request correctness. Infrastructure remains responsible for TLS, WAF, load balancing, Redis HA, secret management, telemetry storage, and container orchestration.

Modules

| Artifact | Responsibility | |---|---| | llmrix-model-open | Shared model contracts, common model exceptions/authentication SPI, and reusable OpenAI-compatible transport/adapters. | | llmrix-model-router-core | Runtime facade and Builder, model targets, strategies, execution, state SPI, provider SPI, quota, health, and events. | | llmrix-model-router-integrations | Default OpenAI/DeepSeek/OpenRouter registrations, Redis, Bucket4j, ONNX, evaluation, shadow, and Fugu adapters. | | llmrix-model-router-spring-starter | Router properties, auto-configuration, OpenAI-compatible HTTP/SSE endpoints, HTTP authentication, request IDs, Actuator, Micrometer/Observation, and configuration metadata. | | llmrix-model-orion | Lightweight framework-neutral Java client for the routing server. | | llmrix-model-orion-spring-starter | Orion auto-configuration and Micrometer integration. | | llmrix-model-examples | Maven aggregator for executable examples and module-scoped tests. Not a production dependency. | | llmrix-model-router-core-examples | Core routing examples and tests. | | llmrix-model-router-integrations-examples | Provider and infrastructure integration examples and tests. | | llmrix-model-router-spring-starter-examples | Spring Boot starter, HTTP protocol, and observability tests. | | llmrix-model-router-server-examples | Runnable standalone Spring Boot server example and launch smoke test. | | llmrix-model-client-examples | Orion client and client starter tests. |

Requirements

• Java 17 or later; Java 21 is recommended.
• Spring Boot 3.x when using either starter.
• Redis is optional and required only for shared multi-instance runtime state.

Quick Start

Maven

  com.llmrix.model
  llmrix-model-router-core
  1.0.2

  com.llmrix.model
  llmrix-model-router-integrations
  1.0.2

Programmatic configuration

The same router can be built without Spring or YAML. The runtime Builder lives in Core; the integrations artifact registers the built-in OpenAI-compatible providers through the Core SPI. Integrations own provider credentials and may define multiple models:

try (LlmRouter router = LlmRouter.builder()
        .integration("openai", integration -> integration
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .model("gpt-4.1-mini", model -> model
                .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS)))
        .integration("deepseek", integration -> integration
            .apiKey(System.getenv("DEEPSEEK_API_KEY"))
            .model("deepseek-chat", model -> model
                .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS).traits(ModelTrait.CODE)))
        .route("general", route -> route
            .strategy("balanced")
            .quota(600L, 100_000L) // shared route RPM and TPM
            .models("openai/gpt-4.1-mini", "deepseek/deepseek-chat"))
        .build()) {
    ChatResponse response = router.chat("Review this Java code");
}

Route quotas are optional and apply to all targets in the route. The two-argument form is quota(requestsPerMinute, tokensPerMinute). When an authenticated request contains RoutingHints.AUTH_QUOTA_KEY, each key receives an independent quota partition; otherwise the route uses a shared partition. Target-level limits(...) remain independent provider-model limits.

Policy-based routing

RoutedChatModel model = RoutedChatModel.builder()
    .target("reasoning", reasoningModel, target -> target
        .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS).traits(ModelTrait.REASONING)
        .inputCostPerMillion(1.25)
        .outputCostPerMillion(10.00))
    .target("fast", fastModel, target -> target
        .operations(ModelOperation.CHAT).traits(ModelTrait.CODE)
        .inputCostPerMillion(0.27)
        .outputCostPerMillion(1.10))
    .strategy(Strategies.balanced())
    .timeout(Duration.ofSeconds(30))
    .maxRetries(1)
    .build();

ChatResponse response = model.chat(ChatRequest.builder()
    .userMessage("Find the race condition")
    .routingHints(RoutingHints.builder()
        .require(ModelTrait.CODE)
        .maxCostUsd(0.05)
        .build())
    .build());

Applications can call synchronously, asynchronously, or as a Flow.Publisher. Text, images, input audio, tools, structured response formats, usage, finish reasons, and common generation options are represented by provider-neutral Core types.

Routing Model

Every request follows one deterministic execution pipeline:

1. Validate the request and normalize routing hints.
2. Remove targets that violate capability, model, context, cost, quota, concurrency, or health constraints.
3. Rank eligible targets with the configured strategy.
4. Acquire runtime quota and concurrency leases.
5. Execute with a bounded per-attempt and total timeout.
6. Retry only retryable failures and only within the configured budget.
7. Mark failures, apply cooldown, and move to the next eligible model in the route pool.
8. Settle token usage, release leases, and publish lifecycle observations.

Built-in strategies include priority, round-robin, weighted random, balanced scoring, semantic scoring, and contextual bandit selection. Custom policies implement RoutingStrategy; custom runtime persistence implements RouterStateStore or BanditStateStore.

HTTP Protocol

The Spring Boot starter exposes an OpenAI-compatible HTTP API. The model field in every request identifies a configured Router route name (such as general, vision, or multimodal) rather than an upstream provider model ID.

Enable the HTTP API

llmrix:
  model:
    router:
      http:
        enabled: true
        auth:
          mode: api-key
          bootstrap-key: ${LLMRIX_MODEL_ROUTER_API_KEY}

export BASE_URL=http://127.0.0.1:8080
export API_KEY=your-llmrix-http-key

Endpoint Catalog

| Endpoint | Description | |---|---| | POST /v1/chat/completions | Synchronous and SSE streaming chat completions. | | POST /v1/responses | Core Responses API subset, with JSON and SSE streaming responses. | | POST /v1/embeddings | Text or token-array embeddings with float and base64 encoding. | | POST /v1/rerank | Query/document reranking with relevance scores. | | POST /v1/audio/transcriptions | Multipart audio transcription. | | POST /v1/audio/translations | Multipart audio translation. | | POST /v1/audio/speech | Text-to-speech with a binary audio response. | | POST /v1/images/generations | Image generation. | | POST /v1/images/edits | Multipart image editing. | | POST /v1/videos | Create a video generation task. | | GET /v1/videos/{video_id} | Retrieve video task status. | | GET /v1/videos/{video_id}/content | Download completed video content. | | DELETE /v1/videos/{video_id} | Delete a video task. | | POST /v1/videos/{video_id}/remix | Create a remix task. | | GET /v1/models | Available chat route identifiers. Operation-only routes are selected by their endpoint. |

The server example includes embeddings and rerank routes backed by free OpenRouter models. The request model is the Router route name, not the upstream model ID:

curl --location "${BASE_URL}/v1/embeddings" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"embeddings","input":"Text to embed"}'

curl --location "${BASE_URL}/v1/rerank" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"rerank","query":"refund policy","documents":["Refunds are available within 30 days.","Contact support by email."],"top_n":1}'

Chat Completions

Synchronous

curl --location "${BASE_URL}/v1/chat/completions" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "general",
    "messages": [
      {"role": "user", "content": "Introduce LLMRix Router in three sentences."}
    ],
    "temperature": 0
  }'

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "general",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "This is the model response."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 18, "completion_tokens": 12, "total_tokens": 30}
}

Streaming (SSE)

curl --no-buffer --location "${BASE_URL}/v1/chat/completions" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "general",
    "messages": [{"role": "user", "content": "Explain model routing."}],
    "stream": true
  }'

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"general","choices":[{"index":0,"delta":{"content":"Model"},"finish_reason":""}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"general","choices":[{"index":0,"delta":{"content":" routing"},"finish_reason":""}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"general","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

Protocol Conventions

Success envelope. Chat-style responses return id, object, model (the route name), choices, and a nullable usage block. Provider names and upstream response bodies are never exposed.

Error envelope. Controller-level errors use an OpenAI-shaped structure with error.message, error.type, error.code, and error.param. Authentication failures rejected before the controller may use a compact form without code or param.

HTTP status codes.

| Status | error.type | Meaning | |---:|---|---| | 400 | invalid_request_error | Invalid request body, parameter, or content format. | | 401 | authentication_error | Missing or invalid Router Bearer key. | | 402 | billing_error | The model service requires account capacity. | | 403 | permission_error | The request is not permitted for the selected model service. | | 404 | invalid_request_error | The route or requested model resource does not exist. | | 429 | rate_limit_error | Router quota, concurrency, or model-service rate limit was reached. | | 500 | server_error | Unclassified application execution failure. | | 503 | server_error | No model satisfies the request, or the model service is temporarily unavailable. |

Clients should branch on the HTTP status and error.type / error.code, not on complete message text. Third-party names, credentials, and raw upstream response bodies are never included in the public error message.

Multimodal Content

Chat Completions accepts text, image_url, video_url, input_audio, and file content parts. The selected model must declare the corresponding input-modalities value (vision, video, audio, or file). Speech and video content endpoints return binary data instead of a JSON wrapper.

Server Deployment

Server deployment, Spring Boot configuration, provider integrations, HTTP authentication, request ID propagation, startup commands, and additional curl examples are documented in [docs/server.md](docs/server.md). The full protocol reference with all endpoints, response shapes, and error formats is in [docs/api.md](docs/api.md).

Client Usage

The Orion Java client, typed model operations, multimodal requests, asynchronous and streaming calls, request options, and Spring Boot client starter are documented in [docs/client.md](docs/client.md).

Reliability and Streaming

• Retry applies only to failures classified as retryable; after a failed attempt, the router may continue through the configured model pool.
• A streaming request may switch targets before its first chunk, never after data is visible to the caller.
• Cancellation propagates to the active target and releases runtime state.
• Tool-bearing requests must not be blindly replayed because tools may have side effects.
• Upstream HTTP status is retained on provider-domain exceptions; non-HTTP failures use -1.
• Online shadow execution is isolated by sampling, timeout, and concurrency limits and skips tool requests by default.

Observability

Router and Fugu lifecycle listeners are the stable Core observability boundary. Optional Spring integration provides Micrometer counters/timers, first-token latency, Observation context, Actuator health, and request-ID correlation. Orion exposes its own dependency-free listener SPI and adapts it to Micrometer when used through Spring.

The framework emits telemetry but does not deploy Prometheus, Grafana, an OpenTelemetry Collector, or log storage.

Fugu Orchestration

FuguOrchestrator implements iterative Candidate/Role selection for solver-reviewer and refinement workflows. It supports rule-based or ONNX policies, bounded turns, retry/fallback, optional shared cooldown state, and a lifecycle Flow.Publisher. The lifecycle stream represents orchestration events, not model token chunks.

Training, reward modeling, and policy rollout stay offline. Runtime policy manifests are versioned and validated before inference.

Extension Points

| SPI | Use it to | |---|---| | ModelClient and modality-specific model contracts | Provider-neutral routing contract for chat, embeddings, rerank, audio, images, and video. | | RoutingStrategy | Implement business-specific target ord

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: llmrix
• Source: llmrix/llmrix-router
• License: MIT
• Homepage: https://www.llmrix.com

Install and usage instructions live in the source repository linked above.