# Ai Latency Optimizer

> >

- **Type:** Skill
- **Install:** `agentstack add skill-patonkikh-apes-ai-latency-optimizer`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [patonkikh](https://agentstack.voostack.com/s/patonkikh)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [patonkikh](https://github.com/patonkikh)
- **Source:** https://github.com/patonkikh/APES/tree/main/skills/ai/ai-latency-optimizer

## Install

```sh
agentstack add skill-patonkikh-apes-ai-latency-optimizer
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# AI Latency Optimizer

# Purpose

Reduce AI inference latency: pipeline profiling, parallelization, caching, model selection, and streaming strategies to meet SLA targets.

**Input:** Workflow spec or architecture, latency SLA (p50/p95), current latency baseline (optional), quality constraints  
**Output:** Latency Optimization Plan with bottleneck analysis, optimizations ranked by impact, and SLA compliance projection
**Examples:** See [examples.md](examples.md) for worked input/output.

---

# Workflow

## Step 1: Profile latency budget

Decompose end-to-end latency:

| Stage | p50 (ms) | p95 (ms) | % of total |
|-------|----------|----------|------------|
| Preprocessing | | | |
| Retrieval (RAG) | | | |
| LLM TTFT | | | |
| LLM generation | | | |
| Tool calls | | | |
| Postprocessing | | | |

Identify bottleneck stage (>40% of p95 or exceeding individual SLA).

## Step 2: Set SLA targets

| Metric | Target | Measurement point |
|--------|--------|-------------------|
| Time to first token (TTFT) | | Client-visible |
| End-to-end p50 | | Client-visible |
| End-to-end p95 | | Client-visible |

Distinguish streaming (TTFT matters) vs batch (total time matters).

## Step 3: Apply latency levers

| Lever | Latency impact | Quality/cost trade-off |
|-------|----------------|------------------------|
| Parallel tool/retrieval calls | High | None if independent |
| Prompt/context reduction | Medium | Possible accuracy loss |
| Faster model tier | High | Quality risk |
| Speculative decoding | Medium | Hardware dependent |
| Edge/regional deployment | Medium | Infra cost |
| Streaming response | High perceived | UX only |
| Precomputed embeddings | Medium | Storage cost |
| Connection pooling / warm instances | Medium | Infra cost |

## Step 4: Design parallel execution

For independent stages:

```text
Input → [Retrieval || Tool prefetch || Classify] → Merge → Generate
```

Document synchronization point and timeout for slowest branch.

## Step 5: Plan streaming strategy

| Content type | Stream | Buffer strategy |
|--------------|--------|-----------------|
| Long narrative | Yes | Token-by-token |
| Structured JSON | Partial | Stream after schema validation |
| Tool calls | No | Wait for complete args |

## Step 6: Validate against SLA

Project p50/p95 after optimizations. Confirm quality metrics unchanged via eval gates.

Run Validation checklist.

---

# Decision Rules

| Condition | Action |
|-----------|--------|
| SLA undefined | Stop; request p50/p95 targets |
| Single stage >60% of p95 | Optimize bottleneck first before micro-optimizations |
| Parallelization adds race conditions | Serialize conflicting stages; parallelize only independent work |
| TTFT SLA 20% | Document trade-off; require explicit approval |

---

# Validation

- [ ] Latency profile with per-stage p50/p95
- [ ] Bottleneck stage identified
- [ ] SLA targets documented (TTFT and e2e)
- [ ] ≥3 optimizations evaluated with impact estimates
- [ ] Parallel execution plan for independent stages
- [ ] Streaming strategy defined for user-facing paths
- [ ] Projected p95 meets SLA after changes
- [ ] Quality impact assessed (no eval regression)

---

# Anti-patterns

- **Optimizing non-bottleneck** — shaving 10ms off postprocess when LLM takes 3s.
- **Cold start ignored** — measuring warm-cache latency only.
- **Sequential by default** — chaining retrieval and tools that could run in parallel.
- **Fake streaming** — buffering full response then simulating stream.
- **Latency without monitoring** — no production p95 tracking after deploy.

---

# Best Practices

- Measure client-side latency, not just server-side model time.
- Set per-stage timeouts to fail fast and degrade gracefully.
- Use regional inference endpoints close to users.
- Pre-warm models/instances before traffic spikes.
- Pair with ai-cost-optimizer when latency fixes increase spend.

---

# Output Structure

```markdown
# Latency Optimization Plan: [System Name]

## Current Profile
| Stage | p50 | p95 | % total |
|-------|-----|-----|---------|

## SLA Targets
| Metric | Target | Current | Gap |
|--------|--------|---------|-----|

## Bottleneck Analysis
[Root cause and contributing factors]

## Optimizations
| Lever | Est. p95 reduction | Trade-off | Priority |
|-------|-------------------|-----------|----------|

## Parallel Execution
[Diagram and sync points]

## Streaming Plan
[What streams, what waits]

## Projected Performance
| Metric | Before | After | Meets SLA |
|--------|--------|-------|-----------|

## Monitoring
| Metric | Source | Alert |
|--------|--------|-------|
```

---

# Next Skills

| Outcome | Recommended Skill |
|---------|-------------------|
| Re-validate quality | `ai/ai-evaluation-builder` |
| Balance cost vs latency | `ai/ai-cost-optimizer` |
| Redesign workflow structure | `ai/ai-workflow-builder` |
| Optimize context size | `ai/context-engineering` |
| Infra scaling | `architecture/scalability-advisor` |

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [patonkikh](https://github.com/patonkikh)
- **Source:** [patonkikh/APES](https://github.com/patonkikh/APES)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-patonkikh-apes-ai-latency-optimizer
- Seller: https://agentstack.voostack.com/s/patonkikh
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.