Azure Aigateway

Azure AI Gateway

Centralize LLM traffic through a controlled gateway for security, observability, policy enforcement, and cost control.

Use This Skill When

• The user needs one entry point for multiple models (Azure OpenAI, third-party LLMs, internal models)
• The user needs to enforce token budgets, rate limits, or SLA policies
• The user needs centralized auditing and compliance logging
• The user needs intelligent fallback (model outage handling)
• The user wants to avoid client-side model management complexity

Context: Gateway Maturity

Immature: Clients call models directly, no centralization Developing: Basic gateway, simple routing, minimal logging Managed: Multi-model routing, policy enforcement, token budgets, audit trails → Target Optimized: Real-time cost optimization, feedback-driven routing, self-healing, ML-based model selection

Required Inputs

• Model endpoints: Azure OpenAI? Hugging Face? Internal models? Multiple versions?
• Routing criteria: Latency? Cost? Specific models for specific tasks?
• Policy requirements: Rate limits? Token budgets? Approval workflows?
• Scale: Throughput (req/sec)? Concurrent users? Peak vs. average?
• Compliance: PII redaction? Audit trail retention? Data residency?
• Cost model: Budget cap? Cost-per-department? Showback?

Decision Tree

What's the primary goal of this gateway?
├─ Cost control → Route to cheaper model if accuracy acceptable
├─ Reliability → Route to alternative model on failure/timeout
├─ Governance → Enforce policies, audit all requests
├─ Performance → Route based on latency SLA
└─ Multi-tenancy → Tenant isolation, quota per tenant

Which models will the gateway route to?
├─ Single model (single Azure OpenAI deployment) → No routing needed
├─ Multiple Azure OpenAI models → Route by task/cost/latency
├─ Multiple providers (Azure OpenAI + third-party) → Handle auth differences
└─ Canary/blue-green (new model test) → Weighted routing

What policy enforcement is needed?
├─ None (simple routing only)
├─ Token budgets (cap usage per user/org)
├─ Rate limits (req/sec, concurrent, bursts)
├─ Approval workflows (route restricted requests for review)
└─ Compliance (PII redaction, output filtering, audit)

How much data volume?
├─  1000 req/sec → Distributed gateway, database-backed quotas

Workflow

Phase 1: Design Gateway Architecture

┌─ Client Application ─┐
│  (web, mobile, API)  │
└──────────┬───────────┘
           │ (HTTPS)
           │
    ┌──────▼─────────────────────────────────────┐
    │ AI Gateway (entry point)                   │
    │                                            │
    │  1. Authenticate client (API key, JWT)    │
    │  2. Check token budget (quota system)     │
    │  3. Log request (audit trail)             │
    │  4. Validate payload (safety checks)      │
    └──────┬─────────────────────────────────────┘
           │
    ┌──────▼──────────────────┐
    │ Routing & Policy Engine │
    │                         │
    │  ├─ Route to model 1    │
    │  ├─ Route to model 2    │
    │  └─ Fallback to model 3 │
    └──────┬──────────────────┘
           │
    ┌──────▼──────────────────────────────────────────┐
    │ Model Pools                                      │
    │                                                  │
    │  [Azure OpenAI GPT-4]  → $0.03 per 1K in      │
    │  [Azure OpenAI GPT-3.5]→ $0.0015 per 1K in   │
    │  [Third-party LLaMA]   → $0.01 per 1K in     │
    │  [Custom Model (AKS)]  → On-premise            │
    └──────────────────────────────────────────────────┘
           │
    ┌──────▼──────────────────────────────────┐
    │ Observability                           │
    │                                         │
    │  ├─ Application Insights (traces)      │
    │  ├─ Log Analytics (audit logs)         │
    │  ├─ Cost Management (spend tracking)   │
    │  └─ Alerts (budget overrun, failures)  │
    └─────────────────────────────────────────┘

Phase 2: Implement Gateway (Choose Stack)

Option 1: Azure API Management + Logic Apps (low-code)

Client → [API Management] → [Logic App] → [Model endpoints]

Benefits: 
- Built-in rate limiting, throttling
- Policy enforcement (transformations, auth)
- Analytics and metrics

Limitations:
- Limited model-specific logic
- Async processing harder

Option 2: Custom Service (Node.js, Python, Go)

# App Service or Container Apps
az containerapp create \
  --resource-group $RG \
  --name "ai-gateway" \
  --image "myacr.azurecr.io/ai-gateway:latest" \
  --target-port 8000 \
  --environment-variables \
    OPENAI_ENDPOINT="https://myai.openai.azure.com/" \
    OPENAI_KEY="@keyvault(openai-key)" \
    FALLBACK_MODEL="gpt-35-turbo"

Option 3: Azure Service Fabric / Kubernetes (scale-out)

Each node = independent gateway instance
Database = shared quota store (Redis, SQL)
Enables high-throughput, multi-region

Phase 3: Configure Routing Rules

1. Cost-optimized routing:

```python # Python pseudo-code def routetomodel(prompt, requiredaccuracy=0.9): # Try cheap model first try: response = gpt35turbo.call(prompt, timeout=2) if qualityscore(response) >= requiredaccuracy: return response except Timeout: pass

# Fallback to expensive but reliable model response = gpt4.call(prompt, timeout=5) return response

# Cost breakdown # 1000 requests: 700 via GPT-3.5 ($1.05) + 300 via GPT-4 ($9.00) = $10.05 # vs. Direct to GPT-4: 1000 × $0.03 = $30 ```

1. Latency-optimized routing:

``` SLA: Response self.limit: raise QuotaExceededError(f"Budget exceeded") redis.incr(f"quota:{userid}", tokensneeded)

# Cost cap: $100/day per customer # If usage exceeds budget, reject or queue for review ```

1. Approval workflows (for sensitive operations):

``` Request: "Summarize confidential document X" ├─ Payload > 10K tokens? → Route to human review ├─ Contains PII pattern? → Route to compliance team └─ User has restricted tag? → Require manager approval

Approved → Route to model Rejected → Return error with reason ```

1. Input/output filtering:

``` Input validation: ├─ Reject prompts with SQL injection patterns ├─ Reject attempts to override system prompt └─ Redact email/SSN before sending to model

Output filtering: ├─ Block generation of malware/illegal content ├─ Redact PII before returning to user └─ Flag hallucinated data if confidence low ```

Phase 5: Implement Observability

1. Log every request (for audit):

```csharp // C# - Log to Application Insights var properties = new Dictionary { { "ClientId", clientId }, { "Model", selectedModel }, { "TokensIn", tokensIn.ToString() }, { "TokensOut", tokensOut.ToString() }, { "Cost", cost.ToString() }, { "LatencyMs", latency.ToString() }, { "Status", response.StatusCode.ToString() } };

telemetryClient.TrackEvent("aigatewayrequest", properties); ```

1. Cost tracking:

``bash # Daily cost by model customMetrics | where name == "ai_gateway_cost" | extend Model = tostring(customDimensions.model) | summarize TotalCost = sum(value) by Model, bin(timestamp, 1d) ``

1. Alerts:

• Budget exceeded: >$500/day
• Model failure: 5+ consecutive timeouts
• Latency spike: p95 > 2000ms
• Abuse pattern: 1000+ req/min from single IP

Phase 6: Deploy & Manage

1. Deploy gateway:

``bash # Container Apps az containerapp up \ --source . \ --build-env-vars "GATEWAY_VERSION=1.0" \ --environment-variables "OPENAI_ENDPOINT=..." \ --secrets "openai-key=$(az keyvault secret show --vault-name $KV --name openai-key --query value)" ``

1. Monitor SLA:

`` Target SLA: 99.9% uptime ├─ Availability: Measure successful responses / total requests ├─ Latency p95: < 500ms ├─ Cost per request: < $0.002 └─ Error rate: < 0.1% ``

1. Update routes (zero-downtime):

`` Current: 100% → Model A Step 1: 90% → Model A, 10% → Model B (canary) Step 2: Monitor for 1 hour Step 3: 50% → Model A, 50% → Model B (if healthy) Step 4: 0% → Model A, 100% → Model B (complete) ``

Output Contract

1. Gateway Architecture

• Deployment topology (App Service, Container Apps, API Mgmt)
• Model endpoints and routing rules
• Fallback and failover strategy

1. Policy & Control Definitions

• Rate limits (req/sec, daily cap)
• Token budgets (cost cap per user/org)
• Approval workflows for restricted operations
• Input/output filtering rules

1. Security & Compliance

• Authentication (API key, JWT, mTLS?)
• Authorization (RBAC roles)
• PII/sensitive data handling
• Audit trail requirements

1. Monitoring & Cost

• Observability dashboards (latency, errors, cost)
• Alerts for anomalies and budget overrun
• Cost breakdown by model/user/department
• SLO targets (availability, latency, cost)

1. Operational Runbook

• Deployment and rollback procedures
• How to add/remove models
• Incident response (model outage, quota exhausted)

Guardrails

• Never bypass policy checks: All requests through gateway, no direct model access.
• Protect API keys: Store model endpoints and secrets in Key Vault, never in code.
• Implement rate limiting: Prevent token stuffing and DDoS attacks.
• Log for compliance: Maintain audit trail for sensitive operations.
• Plan for failover: At least 2 model endpoints, automatic switchover on failure.
• Monitor cost closely: Set budget alerts; can spiral quickly with large batch jobs.
• Test routing changes: Use canary deployment before full rollout.
• Enforce timeouts: Prevent hanging requests from consuming resources indefinitely.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: manu14357
• Source: manu14357/zskills
• License: MIT
• Homepage: https://zskills.vercel.app

Install and usage instructions live in the source repository linked above.