# Azure Aigateway > > - **Type:** Skill - **Install:** `agentstack add skill-manu14357-zskills-azure-aigateway` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [manu14357](https://agentstack.voostack.com/s/manu14357) - **Installs:** 0 - **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [manu14357](https://github.com/manu14357) - **Source:** https://github.com/manu14357/zskills/tree/main/skills/azure-aigateway - **Website:** https://zskills.vercel.app ## Install ```sh agentstack add skill-manu14357-zskills-azure-aigateway ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Azure AI Gateway Centralize LLM traffic through a controlled gateway for security, observability, policy enforcement, and cost control. ## Use This Skill When - The user needs one entry point for multiple models (Azure OpenAI, third-party LLMs, internal models) - The user needs to enforce token budgets, rate limits, or SLA policies - The user needs centralized auditing and compliance logging - The user needs intelligent fallback (model outage handling) - The user wants to avoid client-side model management complexity ## Context: Gateway Maturity **Immature**: Clients call models directly, no centralization **Developing**: Basic gateway, simple routing, minimal logging **Managed**: Multi-model routing, policy enforcement, token budgets, audit trails → **Target** **Optimized**: Real-time cost optimization, feedback-driven routing, self-healing, ML-based model selection ## Required Inputs - **Model endpoints**: Azure OpenAI? Hugging Face? Internal models? Multiple versions? - **Routing criteria**: Latency? Cost? Specific models for specific tasks? - **Policy requirements**: Rate limits? Token budgets? Approval workflows? - **Scale**: Throughput (req/sec)? Concurrent users? Peak vs. average? - **Compliance**: PII redaction? Audit trail retention? Data residency? - **Cost model**: Budget cap? Cost-per-department? Showback? ## Decision Tree ``` What's the primary goal of this gateway? ├─ Cost control → Route to cheaper model if accuracy acceptable ├─ Reliability → Route to alternative model on failure/timeout ├─ Governance → Enforce policies, audit all requests ├─ Performance → Route based on latency SLA └─ Multi-tenancy → Tenant isolation, quota per tenant Which models will the gateway route to? ├─ Single model (single Azure OpenAI deployment) → No routing needed ├─ Multiple Azure OpenAI models → Route by task/cost/latency ├─ Multiple providers (Azure OpenAI + third-party) → Handle auth differences └─ Canary/blue-green (new model test) → Weighted routing What policy enforcement is needed? ├─ None (simple routing only) ├─ Token budgets (cap usage per user/org) ├─ Rate limits (req/sec, concurrent, bursts) ├─ Approval workflows (route restricted requests for review) └─ Compliance (PII redaction, output filtering, audit) How much data volume? ├─ 1000 req/sec → Distributed gateway, database-backed quotas ``` ## Workflow ### Phase 1: Design Gateway Architecture ``` ┌─ Client Application ─┐ │ (web, mobile, API) │ └──────────┬───────────┘ │ (HTTPS) │ ┌──────▼─────────────────────────────────────┐ │ AI Gateway (entry point) │ │ │ │ 1. Authenticate client (API key, JWT) │ │ 2. Check token budget (quota system) │ │ 3. Log request (audit trail) │ │ 4. Validate payload (safety checks) │ └──────┬─────────────────────────────────────┘ │ ┌──────▼──────────────────┐ │ Routing & Policy Engine │ │ │ │ ├─ Route to model 1 │ │ ├─ Route to model 2 │ │ └─ Fallback to model 3 │ └──────┬──────────────────┘ │ ┌──────▼──────────────────────────────────────────┐ │ Model Pools │ │ │ │ [Azure OpenAI GPT-4] → $0.03 per 1K in │ │ [Azure OpenAI GPT-3.5]→ $0.0015 per 1K in │ │ [Third-party LLaMA] → $0.01 per 1K in │ │ [Custom Model (AKS)] → On-premise │ └──────────────────────────────────────────────────┘ │ ┌──────▼──────────────────────────────────┐ │ Observability │ │ │ │ ├─ Application Insights (traces) │ │ ├─ Log Analytics (audit logs) │ │ ├─ Cost Management (spend tracking) │ │ └─ Alerts (budget overrun, failures) │ └─────────────────────────────────────────┘ ``` ### Phase 2: Implement Gateway (Choose Stack) **Option 1: Azure API Management + Logic Apps (low-code)** ``` Client → [API Management] → [Logic App] → [Model endpoints] Benefits: - Built-in rate limiting, throttling - Policy enforcement (transformations, auth) - Analytics and metrics Limitations: - Limited model-specific logic - Async processing harder ``` **Option 2: Custom Service (Node.js, Python, Go)** ```bash # App Service or Container Apps az containerapp create \ --resource-group $RG \ --name "ai-gateway" \ --image "myacr.azurecr.io/ai-gateway:latest" \ --target-port 8000 \ --environment-variables \ OPENAI_ENDPOINT="https://myai.openai.azure.com/" \ OPENAI_KEY="@keyvault(openai-key)" \ FALLBACK_MODEL="gpt-35-turbo" ``` **Option 3: Azure Service Fabric / Kubernetes (scale-out)** ``` Each node = independent gateway instance Database = shared quota store (Redis, SQL) Enables high-throughput, multi-region ``` ### Phase 3: Configure Routing Rules 1. **Cost-optimized routing**: ```python # Python pseudo-code def route_to_model(prompt, required_accuracy=0.9): # Try cheap model first try: response = gpt35_turbo.call(prompt, timeout=2) if quality_score(response) >= required_accuracy: return response except Timeout: pass # Fallback to expensive but reliable model response = gpt4.call(prompt, timeout=5) return response # Cost breakdown # 1000 requests: 700 via GPT-3.5 ($1.05) + 300 via GPT-4 ($9.00) = $10.05 # vs. Direct to GPT-4: 1000 × $0.03 = $30 ``` 2. **Latency-optimized routing**: ``` SLA: Response self.limit: raise QuotaExceededError(f"Budget exceeded") redis.incr(f"quota:{user_id}", tokens_needed) # Cost cap: $100/day per customer # If usage exceeds budget, reject or queue for review ``` 3. **Approval workflows** (for sensitive operations): ``` Request: "Summarize confidential document X" ├─ Payload > 10K tokens? → Route to human review ├─ Contains PII pattern? → Route to compliance team └─ User has restricted tag? → Require manager approval Approved → Route to model Rejected → Return error with reason ``` 4. **Input/output filtering**: ``` Input validation: ├─ Reject prompts with SQL injection patterns ├─ Reject attempts to override system prompt └─ Redact email/SSN before sending to model Output filtering: ├─ Block generation of malware/illegal content ├─ Redact PII before returning to user └─ Flag hallucinated data if confidence low ``` ### Phase 5: Implement Observability 1. **Log every request** (for audit): ```csharp // C# - Log to Application Insights var properties = new Dictionary { { "ClientId", clientId }, { "Model", selectedModel }, { "TokensIn", tokensIn.ToString() }, { "TokensOut", tokensOut.ToString() }, { "Cost", cost.ToString() }, { "LatencyMs", latency.ToString() }, { "Status", response.StatusCode.ToString() } }; telemetryClient.TrackEvent("ai_gateway_request", properties); ``` 2. **Cost tracking**: ```bash # Daily cost by model customMetrics | where name == "ai_gateway_cost" | extend Model = tostring(customDimensions.model) | summarize TotalCost = sum(value) by Model, bin(timestamp, 1d) ``` 3. **Alerts**: - Budget exceeded: >$500/day - Model failure: 5+ consecutive timeouts - Latency spike: p95 > 2000ms - Abuse pattern: 1000+ req/min from single IP ### Phase 6: Deploy & Manage 1. **Deploy gateway**: ```bash # Container Apps az containerapp up \ --source . \ --build-env-vars "GATEWAY_VERSION=1.0" \ --environment-variables "OPENAI_ENDPOINT=..." \ --secrets "openai-key=$(az keyvault secret show --vault-name $KV --name openai-key --query value)" ``` 2. **Monitor SLA**: ``` Target SLA: 99.9% uptime ├─ Availability: Measure successful responses / total requests ├─ Latency p95: < 500ms ├─ Cost per request: < $0.002 └─ Error rate: < 0.1% ``` 3. **Update routes (zero-downtime)**: ``` Current: 100% → Model A Step 1: 90% → Model A, 10% → Model B (canary) Step 2: Monitor for 1 hour Step 3: 50% → Model A, 50% → Model B (if healthy) Step 4: 0% → Model A, 100% → Model B (complete) ``` ## Output Contract 1. **Gateway Architecture** - Deployment topology (App Service, Container Apps, API Mgmt) - Model endpoints and routing rules - Fallback and failover strategy 2. **Policy & Control Definitions** - Rate limits (req/sec, daily cap) - Token budgets (cost cap per user/org) - Approval workflows for restricted operations - Input/output filtering rules 3. **Security & Compliance** - Authentication (API key, JWT, mTLS?) - Authorization (RBAC roles) - PII/sensitive data handling - Audit trail requirements 4. **Monitoring & Cost** - Observability dashboards (latency, errors, cost) - Alerts for anomalies and budget overrun - Cost breakdown by model/user/department - SLO targets (availability, latency, cost) 5. **Operational Runbook** - Deployment and rollback procedures - How to add/remove models - Incident response (model outage, quota exhausted) ## Guardrails - **Never bypass policy checks**: All requests through gateway, no direct model access. - **Protect API keys**: Store model endpoints and secrets in Key Vault, never in code. - **Implement rate limiting**: Prevent token stuffing and DDoS attacks. - **Log for compliance**: Maintain audit trail for sensitive operations. - **Plan for failover**: At least 2 model endpoints, automatic switchover on failure. - **Monitor cost closely**: Set budget alerts; can spiral quickly with large batch jobs. - **Test routing changes**: Use canary deployment before full rollout. - **Enforce timeouts**: Prevent hanging requests from consuming resources indefinitely. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [manu14357](https://github.com/manu14357) - **Source:** [manu14357/zskills](https://github.com/manu14357/zskills) - **License:** MIT - **Homepage:** https://zskills.vercel.app Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-manu14357-zskills-azure-aigateway - Seller: https://agentstack.voostack.com/s/manu14357 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.