Vllm Caching

vLLM tiered KV caching

Target audience: operators running vLLM on H100/H200-class datacenter GPUs in production. Assumes CUDA 12/13, Kubernetes or bare container deployment, multi-GPU tensor parallel.

Why this matters

Long-context workloads (coding agents, RAG, research agents averaging 50k–200k tokens) are almost always KV-cache bound, not compute bound. The GPU sits idle waiting for free KV slots. Tiered caching — HBM → CPU DRAM → local NVMe → remote — extends effective capacity without adding GPUs and converts repeated sessions from "re-prefill every turn" to "reload in milliseconds."

The single biggest throughput lever for long-context serving, once the model fits, is a well-sized DRAM tier. It routinely delivers 50–100× speedups on cache hits vs re-prefill, and coding-agent traffic has strong prefix locality that makes hits nearly universal after the first turn.

Version gates — check these FIRST

Operators constantly ask "is this available?" when it either isn't in their version, or has been for a while and they missed it. Always confirm the vLLM version before recommending a config.

| Feature | First release | Notes | |---|---|---| | Native CPU KV offload (vllm/v1/kv_offload/) | v0.11.0 (2025-10-02) | Infrastructure + scheduler integration | | CLI flags --kv-offloading-size / --kv-offloading-backend | v0.11.1 (2025-11-18) | Before this, required editing config objects | | LMCache bundled in official x86 vllm/vllm-openai image | v0.14.0 (2026-01-20) | arm64 had it from v0.10.2; x86 was intentionally stripped July 2025–Jan 2026 due to torch version conflict — many ops teams had to pip install LMCache at container start during that window, which is now unnecessary | | --calculate-kv-scales deprecated | pre-v0.19 (still present in v0.21.0) | Flag emits a deprecation warning but still accepted. FP8 KV without shipped scales falls back to scale=1.0 (see pitfalls). Verified 2026-04-24 against vllm/config/cache.py on main. | | KV Offload + Hybrid Memory Allocator (HMA) | v0.21.0 (2026-05-15) | Native offload now integrates with HMA — full enablement (#41445) + sliding-window groups (#41228) + multi-connector HMA (#39571), plus Qwen3.5/Mamba hybrid support (#35520). Re-test hybrid models on v0.21.0 before assuming --disable-hybrid-kv-cache-manager is still required (see Hybrid-models section). |

Latest stable as of 2026-05-28: v0.21.0 (2026-05-15). v0.20.0 GA'd 2026-04-27, followed by v0.20.1 (05-04), v0.20.2 (05-10).

Known-good tags: v0.14.0+, v0.19.0, v0.19.0-cu130, v0.20.x, v0.21.0, and model-specific tags like glm51-cu130 all ship with INSTALL_KV_CONNECTORS=true baked in — LMCache, NIXL, and Mooncake pre-installed. Confirm bundling on a specific tag with the two-step check below before trusting it.

Two-step bundling verification (build flag + runtime import)

Build flag = "we tried to install it." Runtime import = "the package actually loads." Different things — the torch-conflict era of mid-2025 had cases where the build flag said yes but import lmcache failed at runtime.

Step 1 — build-flag check (no pull, ~1s):

${CLAUDE_SKILL_DIR}/scripts/inspect-vllm-image.sh 

Prints LMCache/NIXL/Mooncake: YES (built with INSTALL_KV_CONNECTORS=true) if the build layer ran the kv_connectors install.

Step 2 — runtime-import check (pull if needed, ~30s after pull):

~/.claude/skills/lmcache-mp/scripts/verify-bundling.sh 

Starts a sleep-overridden container, exec's a Python probe that confirms lmcache, nixl, mooncake import cleanly; checks the LMCache MP adapter classes and the ParallelStrategy version-hazard symbol; lists all registered KV connectors; loads each connector class through the factory's own thunk. Run this against any tag before trusting it in production. Verified vllm/vllm-openai:v0.19.1 2026-04-26: vllm 0.19.1, lmcache 0.4.3, nixl 0.9.0, mooncake-transfer-engine 0.3.10.post1, all imports clean, all four KV-offload connectors load.

lmcache version compatibility: vLLM main imports ParallelStrategy from the lmcache MP adapter. That symbol does NOT exist in lmcache 0.4.3 (verified against the v0.4.3 tag); it was added in 0.4.4. v0.19.1 image ships 0.4.3 — fine for v0.19.x vLLM source which doesn't need the symbol, but if you mix vLLM main with the bundled lmcache, expect ImportError: cannot import name 'ParallelStrategy'. Pin the pair or pip install -U lmcache>=0.4.4 inside the container.

For other diagnostics (LMCache logs, gdscheck, Prometheus metrics), see references/diagnostics.md.

Decision tree — which backend

Ask these in order:

1. Single node, CPU DRAM tier only, no disk? → --kv-offloading-backend native. Zero extra deps, included since v0.11.1. Start here unless there is a concrete reason to add complexity.
2. Single node, want NVMe as a third tier? → LMCache in-process via LMCacheConnectorV1 (--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' + LMCACHE_LOCAL_DISK env vars).
3. Multiple vLLM pods on the same node want a SHARED KV cache, OR cache CPU work should not contend with the inference GIL, OR cache memory needs to scale independently of GPU pods? → LMCache MP mode (separate-pod LMCache server, vLLM connects via ZMQ using LMCacheMPConnector). Defer to the lmcache-mp skill for the DaemonSet+Deployment pattern, image pair, ZMQ protocol, L2 adapter cascade.
4. Disaggregated prefill across nodes (separate prefill and decode pods)? → NixlConnector via --kv-transfer-config. Tunes TTFT and ITL independently.
5. RDMA-backed KV transfer between nodes, high-throughput datacenter fabric? → MooncakeConnector.
6. NIXL between prefill/decode pods AND local CPU overflow on each? → MultiConnector, composing NixlConnector + OffloadingConnector.

Do not reach for LMCache or NIXL or Mooncake just because they exist. Native offload handles the 80% case with zero operational surface area. LMCache MP adds an extra pod and another image — only justified by the multi-pod-shared-cache or GIL-isolation cases above.

For concrete config recipes for any of the above, see references/connectors.md. Load that reference once a backend has been selected — it contains copy-paste-ready invocations with all required env vars, GDS host prerequisites (cuda-keyring, open kernel modules, Secure Boot), and the MultiConnector JSON format.

> NIXL deep-dive — NIXL itself (transfer library, 13 backend plugins, agent API, telemetry, ETCD/side-channel metadata, plugin authoring) lives in the dedicated nvidia-nixl skill. This skill covers vLLM-side wiring of NixlConnector / LMCache-P2P-over-NIXL only. Reach for nvidia-nixl when picking transports (UCX_TLS, GDS, Mooncake, libfabric…), tuning UCX, debugging nixl_agent directly, or writing custom plugins.

> LMCache MP deep-dive — the standalone-server multiprocess mode (LMCacheMPConnector, lmcache server, lmcache/standalone:nightly, DaemonSet+Deployment, L2 adapters: nixlstore / fs / mooncakestore / s3) is its own operational shape. The dedicated lmcache-mp skill covers it.

Sizing math — READ BEFORE RECOMMENDING A NUMBER

--kv-offloading-size is TOTAL across all TP ranks, in GiB. This is the OPPOSITE of SGLang's equivalent flag (which is per-rank). Teams migrating from SGLang routinely under-allocate by a factor of TP.

If an SGLang config was 200 GB per GPU × 8 GPUs = 1600 GB, then on vLLM use --kv-offloading-size 1600. Not 200.

For per-token KV formulas, worked slot-math examples, Dell XE9680 hardware reference numbers, and NVMe-vs-DRAM-vs-prefill load-time comparisons, see references/hardware-sizing.md.

When using LMCache as the backend, keep LMCACHE_MAX_LOCAL_CPU_SIZE and --kv-offloading-size consistent. LMCache env vars win on the backend side; inconsistency leads to scheduler miscounting available slots and either over-subscribing or under-using the cache.

Critical pitfalls

OffloadingConnector requires --disable-hybrid-kv-cache-manager

Add this flag whenever --kv-offloading-size is set. Without it the engine fails at startup with:

ValueError: Connector OffloadingConnector does not support HMA but HMA is enabled.
Please set `--disable-hybrid-kv-cache-manager`.

The Hybrid Memory Allocator (HMA) is the default scheduler in vLLM v0.18+ and is mutually exclusive with OffloadingConnector. This is the single most common silent blocker for first-time KV-offload deploys — the error message names the fix but it appears on no cookbook page or release note (verified 2026-04-25 against release notes for v0.18.0 → v0.20.0). Verified live on RTX 4060 Ti 16 GB + cu130-nightly + Qwen3-4B; pod boots clean once the flag lands.

The same flag is required by LMCacheConnectorV1, LMCacheMPConnector, and the new hybrid-aware path being built — see the next section.

Hybrid models (Qwen3.5, Gemma4, Mamba+attention) are a moving target

vLLM's Hybrid Memory Allocator (HMA) is mutually exclusive with all current KV-offload connectors. Disabling HMA fixes startup, but introduces secondary problems on hybrid-attention models:

• 0% prefix cache hit rate on Gemma4 + speculative decoding (EAGLE/DFlash/MTP) — reported as vLLM#40624 (open as of 2026-04-26, last update 2026-04-23). Caused by a per-manager EAGLE-drop spiral when the hybrid coordinator sees ≥3 attention groups (Gemma4 full + sliding + DFlash draft = 3 specs). Workaround: --disable-hybrid-kv-cache-manager + lower --max-model-len to fit non-HMA allocation.
• LMCacheConnectorV1 still doesn't support hybrid models (LMCache#3106, open, last update 2026-05-27, and LMCache#2845). The LMCache storage path materializes a single-shape MemoryObj.tensor and fails on mixed attention groups (Gemma-4 sliding + full, Qwen3.5 delta-net + attention). This is independent of the vLLM-native path below — the v0.21.0 HMA work fixed vLLM's own offload, NOT the LMCacheConnectorV1 path.
• Native offload + hybrid shipped in v0.21.0 (2026-05-15). The [kv_offload+HMA][N/N] PR series fully merged: #41445 [13/N] full HMA enablement + #41228 sliding-window groups + #39571 multi-connector HMA, with Qwen3.5/Mamba hybrid model support via #35520. The native OffloadingConnector now integrates with the Hybrid Memory Allocator.

Today's recommendation for hybrid-model + offload:

1. On v0.21.0+ with native offload, re-test before disabling HMA. The v0.21.0 HMA enablement means --kv-offloading-backend native may now coexist with the hybrid allocator on Qwen3.5/Mamba-class models — boot with native offload and only fall back to --disable-hybrid-kv-cache-manager + reduced --max-model-len if startup still errors. Confirm the exact still-gated paths against the v0.21.0 release notes for the specific model.
2. LMCacheConnectorV1 on hybrid models remains blocked (#3106). If a DRAM/NVMe tier on a hybrid model is required, prefer the native v0.21.0 path over LMCache, or use NixlConnector 1P1D (different topology).
3. Pure-transformer models (Qwen3-14B, Llama-3, Mistral-7B) are unaffected by this section — the standard offload recipes work on any v0.14.0+ tag.

When rechecking, the canonical probe is:

gh issue view 3106 --repo LMCache/LMCache --json state,updatedAt   # LMCacheConnectorV1 hybrid block — still open 2026-05-27
gh issue view 40624 --repo vllm-project/vllm --json state,updatedAt # Gemma4 + spec-decode prefix-cache hit rate
gh release view v0.21.0 --repo vllm-project/vllm | grep -i "HMA\|hybrid"  # confirm shipped HMA scope per model

--cpu-offload-gb is NOT the same as --kv-offloading-size

| Flag | What it offloads | Unit | Effect | |---|---|---|---| | --cpu-offload-gb | Model weights | Per-GPU | Hurts prefill throughput; lets larger-than-HBM models fit | | --kv-offloading-size | KV cache | Total across TP | Helps TTFT on cache hits; no effect on compute throughput |

Recommending --cpu-offload-gb when the user asked about KV tiering is a serious error. They are different subsystems with opposite throughput implications.

FP8 KV cache without shipped scales

--calculate-kv-scales is deprecated (still accepted as of v0.21.0, emits a warning, scheduled for removal). Setting it has no effect — vLLM now always loads scales from the checkpoint. If --kv-cache-dtype fp8 is set on a model whose checkpoint doesn't ship calibrated k_scale/v_scale, vLLM defaults to scale=1.0, which can clip pathological activations — especially on long code contexts where specific tokens produce large activations in specific layers.

Symptoms: subtle quality degradation, often only past 128k context. "Usually works fine" is the common operator experience because most activations fit, but the risk is real.

Recommend: stay on BF16 KV and use offload to claw back the memory. The prefill savings from offload hits dwarf whatever would be saved by moving KV from BF16 to FP8.

NVSwitch doesn't accelerate host-to-GPU offload

Operators new to tiered caching sometimes assume NVSwitch/NVLink helps. It doesn't — those interconnects are GPU↔GPU only. KV offload traffic is CPU DRAM → PCIe Gen5 x16 → GPU HBM, one link per GPU. See references/hardware-sizing.md for the bandwidth numbers on XE9680-class hardware.

NVMe tier is NOT a hot path

Loading a 100k-token BF16 context from NVMe takes ~5–7 seconds (Gen4 U.2, buffered) — still beats a ~10 second re-prefill, but DRAM beats NVMe by 20×. Size the DRAM tier to hold the hot working set; treat NVMe as overflow for sessions from "earlier today" that might come back.

When the user says "it doesn't help"

Most common root causes, in order:

1. Concurrency is already at the memory wall, not the cache. On an 8× H200 with a 300–500 GB weight footprint serving 200k+ contexts, 1–2 live slots is physics — compute jumped ~6× over the previous generation but memory bandwidth only ~1.6×. Adding offload doesn't raise live concurrency; it raises the warm-set size (sessions that skip re-prefill on return). Confirm the expectation is "more warm sessions," not "more simultaneous running."
2. Hit rate is low. No prefix reuse in the workload → offload gains nothing. Check prefix_cache_hits_total / prefix_cache_queries_total. Non-agentic, non-RAG traffic often has low prefix locality.
3. --enable-prefix-caching is missing. Without it, vLLM doesn't try to reuse prefixes in the first place; offload has nothing to hit against.
4. Flag sized as per-GPU (SGLang habit). Check vllm:cache_config_info — the number reported is what vLLM actually allocated. If it's 1/TP of what was expected, this is the cause.
5. Cold cache. First N requests pay full prefill cost while filling the cache. Measure steady-state, not ramp-up.
6. Max context set higher than needed. If --max-model-len is 200k but typical requests are 8k, the scheduler reserves capacity for 200k; effective concurrency is starved. Tune --max-model-len to the 95th or 99th percentile of real traffic, not the theoretical maximum.
7. Actually compute-bound. Short prompts + many tokens out = decode-dominated. Offload can't help; more GPUs or a smaller model is the fix.

Open bugs to know before recommending offload

Issue states verified 2026-05-28. All checked when authoring a new offload deploy.

| Issue | Repo | State | Affects | Avoidance | |---|---|---|---|---| | #40259 | vllm-project/vllm | open | KV offload + EAGLE3 + Expert Parallel cuMemcpyDtoHAsync segfault on 8× H20-3e | Don't combine offload with EP+EAGLE3 until fix lands | | #2942 | LMCache/LMCache | open | LocalCPUBackend.allocate() deadlocks when `use_ho

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: air-gapped
• Source: air-gapped/skills
• License: MIT

Install and usage instructions live in the source repository linked above.