Model Gateway Routing

Model Gateway / Routing

A gateway is a single control point between apps and many models/providers. It turns "which model?" and cross-cutting concerns into infrastructure.

What it centralizes (the reasons to build/buy one)

• Routing: pick a model per request by **cost / quality / latency / context

length / capability**, or A/B and canary new models.

• Fallback & resilience: retry/failover across providers on error or rate

limit; circuit-break a failing provider.

• Cost control: per-team/app budgets, quotas, and cost attribution;

route cheap queries to cheap models.

• Caching: exact + semantic cache to skip duplicate/near-duplicate calls.
• Security/governance: central API-key custody, PII redaction, audit logs,

policy (which teams may call which models).

• Observability: latency/tokens/cost/error metrics in one place.

Implementations: LiteLLM, Envoy AI Gateway, cloud AI gateways, or custom.

Design decisions

• Routing policy: static (rules) vs learned/heuristic (route by predicted

difficulty). Keep it explainable; mind added hop latency.

• Sync vs streaming: must pass through token streaming with low overhead.
• Statelessness: keep the gateway stateless + horizontally scalable; push

state (cache, budgets) to fast stores.

• Failure semantics: define what happens when all providers fail.

When to recommend

• Multiple models/providers, multiple teams, real cost/governance needs → yes.
• Single model, single team, prototype → a gateway is premature; call the model

directly and add the gateway when the second model/provider/team appears.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: claude-dev-suite
• Source: claude-dev-suite/claude-dev-suite
• License: MIT

Install and usage instructions live in the source repository linked above.