Ai Model Gateway

AI Model Gateway

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

Use When

• Designing or implementing the LLM gateway as a control-plane service for a multi-tenant SaaS.
• Replacing direct OpenAI / Anthropic / Bedrock SDK calls in feature code with a routed, audited, cost-attributed path.
• Adding multi-provider fallback to an existing AI feature (Anthropic → Bedrock → OpenAI).
• Enforcing per-tenant token caps, regional routing, and audit at one chokepoint instead of scattered library calls.

Do Not Use When

• The task is the wider AI architecture — start with ai-on-saas-architecture.
• The task is direct provider API exploration / spike — ai-llm-integration is the bare-metal SDK skill.
• The task is the prompt design — use ai-prompt-engineering.

Required Inputs

• The model tier catalogue (Free → distilled; Pro → mid-tier; Enterprise → flagship).
• The provider list, their SLAs, their regions, and contract limits.
• The tenant_ai_binding schema from ai-on-saas-architecture.
• Cost ceiling policy per plan (hard caps vs soft caps).
• Residency commitments per tenant region.

Workflow

1. Read this SKILL.md.
2. Define the internal gateway contract (§1) — HTTP + SDK shape, request envelope, response envelope.
3. Implement the request pipeline (§2): auth → model resolve → rate limit → safety in → provider call → safety out → cost capture → audit → respond.
4. Build the provider adapter layer (§3) so adding a provider is a # signed by internal auth

X-Tenant-Id: 8421 # required X-Feature-Id: support-copilot.answer # namespaced X-Trace-Id: trc_01HXY... # propagate

{ "promptid": "support.answer", "promptversion": "latest", // or pinned e.g. "v17" "variables": { "userquestion": "...", "kbpartitionid": "kbt8421" }, "retrieval": { // optional; gateway can call KB service "doretrieve": true, "topk": 6 }, "intent": "answerquestion", "userid": 990012, "maxtokensout": 800, "temperature": 0.2, "stream": false }

Response:

```json
{
  "request_id": "ai_req_01HXY...",
  "model_used": "Codex-3.7-sonnet",
  "region": "eu-west-1",
  "text": "...",
  "tokens_in": 1840,
  "tokens_out": 412,
  "usd_cost": 0.013824,
  "latency_ms": 1923,
  "fallback_used": false,
  "safety_findings": [],
  "grounding_score": 0.91,
  "citations": [{"chunk_id": "...", "source": "...", "score": 0.83}],
  "eval_sampled": false
}

A streaming variant uses Server-Sent Events. The final SSE event carries the full envelope (cost, latency, model, audit id).

§2 Request Pipeline

1. Authn (service JWT)        — verify signature; resolve calling service
2. Authz (tenant + feature)   — service is allowed to act for this tenant on this feature
3. Resolve binding            — read tenant_ai_binding for tenant
4. Entitlement check          — tenant's plan permits this feature/model
5. Kill-switch check          — ai_enabled = false → 403 fast
6. Rate limit                 — Redis token bucket per tenant per feature
7. Cap check                  — monthly USD/token cap not exceeded
8. Resolve prompt             — prompt registry returns (template, model_hint)
9. Render prompt              — template + variables (sanitised)
10. Safety in                 — prompt-injection classifier on user-supplied variables
11. (optional) Retrieval      — call KB service with tenant_id (no other path)
12. Provider call             — primary; retry once on transient
13. Safety out                — PII scrub, jailbreak detect, grounding check
14. Cost compute              — tokens × price table → usd_cost
15. Audit write               — synchronous; row in ai_requests
16. Cost event                — ai.cost.recorded onto event bus
17. Eval sample               — N% of requests written to eval queue
18. Respond                   — envelope to caller

The pipeline is the gateway. Each stage has a hard timeout; stage failures emit gateway.stage.failed traces.

§3 Provider Adapter Layer

class Provider(Protocol):
    name: str
    models: list[ModelDescriptor]
    regions: list[str]

    async def generate(self, req: NormalizedRequest) -> NormalizedResponse: ...

class AnthropicProvider:
    name = "anthropic"
    models = [
        ModelDescriptor("Codex-3.7-sonnet", ctx=200_000,
                        in_price=3e-6, out_price=15e-6),
        ModelDescriptor("Codex-3-haiku", ctx=200_000,
                        in_price=0.25e-6, out_price=1.25e-6),
    ]
    regions = ["us-east-1", "eu-west-1"]

    async def generate(self, req): ...

Adapter responsibilities:

• Translate normalised request → provider SDK call.
• Translate provider response/error → normalised response/error.
• Surface model capability flags (vision, tools, JSON-mode, streaming).
• Report region routing options.

Anything else (rate limit, retries with backoff, cost compute, audit) lives in the pipeline, not the adapter.

§4 Fallback Chains

Per tier, an ordered list of (provider, model, region) candidates.

tiers:
  enterprise:
    primary:     [anthropic, Codex-3.7-sonnet, region:tenant]
    fallback_1:  [bedrock,   anthropic.Codex-3-5-sonnet, region:tenant]
    fallback_2:  [openai,    gpt-4o, region:tenant_or_us]
  pro:
    primary:     [anthropic, Codex-3.7-sonnet, region:tenant]
    fallback_1:  [anthropic, Codex-3-5-haiku, region:tenant]
  free:
    primary:     [anthropic, Codex-3-haiku, region:any]
    fallback_1:  [openai,    gpt-4o-mini, region:any]

Triggering fallback:

• 5xx, timeout, or RateLimitError after one retry on primary.
• 429 with Retry-After > slo_budget.
• Model deprecation event.
• A safety vote from the in-line classifier (rare).

Record fallback_used=true in the audit row; alert when fallback ratio for a tier exceeds threshold.

§5 Per-Tenant Token / USD Ceilings

The gateway enforces caps at the atomic check-and-increment level using Redis (saas-rate-limiting-and-quotas algorithms). Three ceilings:

• Hard USD cap per month (tenant_ai_binding.monthly_usd_cap). On hit: 429 + quota:ai_usd. Upgrade path link in response.
• Hard token cap per day (rate-shaped). On hit: 429 + quota:ai_tokens_day.
• Soft cap at 80% — fires ai.budget.threshold event for in-product banner and sales-assist email; no enforcement.

For enterprise tenants on a true-up model, replace the hard cap with a paging threshold instead.

§6 Regional Routing

tenant_ai_binding.region drives:

• The provider region called (must match for residency).
• The region of the KB partition called.
• The S3 bucket of the audit payload.

When a region's preferred model is unavailable, the gateway consults a region policy:

• strict: 503 if the region cannot serve. Used for sovereignty.
• degraded: allow cross-region with a region_breach=true flag in audit; emit alert.
• permissive: allow cross-region silently (default for low-sensitivity tenants).

§7 Audit + Cost Capture

Synchronous, in-pipeline, atomic with the response. The gateway returns ONLY after the ai_requests row is committed and the ai.cost.recorded event has been published (or rolled back).

Two writes:

1. Postgres ai_requests row (the legal/compliance record).
2. Redis tenant cost counter increment (the realtime billing view).

A reconciliation job nightly compares Postgres rollups vs Redis to detect drift.

See references/token-accounting-pipeline.md.

§8 Kill-Switch Path

The gateway reads tenant_ai_binding.ai_enabled from a Redis-cached binding (TTL 30s; invalidated by ai.kill_switched event).

On ai_enabled = false:

{
  "error": {
    "code": "AI_DISABLED",
    "message": "AI features are disabled for this tenant. Contact support.",
    "kill_switch_reason": "tenant requested temporary disable"
  }
}

The back-office UI (saas-admin-backoffice-tooling) exposes the toggle plus a feature-scoped variant (tenant_ai_feature_disable table).

§9 SLA & Ops

| Metric | Target | |---|---| | Gateway availability | 99.95% | | Hot-path overhead | < 50ms p95 | | Audit-write success | 100% (any failure = reject request) | | Time to roll a new provider | < 1 week from contract to traffic | | Time to flip kill-switch | < 60 seconds | | Fallback ratio (enterprise tier) | < 1% sustained |

Dashboards: gateway QPS, p50/p95/p99 latency by stage, error rate by stage, fallback ratio by tier, cost burn by tenant top-20.

§10 Anti-Patterns

• Building per-feature gateway routes — turns the gateway into a feature factory.
• Streaming responses that don't emit a final envelope event — cost + audit lost on disconnect.
• Async audit writes via best-effort queue — failures cause compliance and billing drift.
• Rate limiting only at the provider — noisy tenants block others.
• Logging full prompts and responses unfiltered — PII exposure; encrypt audit payloads at rest.
• One provider, one model — first outage = total outage.
• Adapter logic leaking into the pipeline (provider-specific if anthropic: ... branches).

§11 Read Next

• ai-on-saas-architecture — broader context.
• ai-cost-per-tenant-attribution — what the gateway feeds.
• ai-usage-metering-and-billing — how the ledger turns into invoices.
• ai-prompt-injection-and-tenant-safety — the safety-in/safety-out logic the pipeline runs.
• ai-observability-and-debugging — traces, replays, debugging.
• saas-rate-limiting-and-quotas — algorithms the gateway uses.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: peterbamuhigire
• Source: peterbamuhigire/skills-web-dev
• License: MIT
• Homepage: https://techguypeter.com

Install and usage instructions live in the source repository linked above.