Deploy Kimi K26 On Rtx Pro 6000

Deploy Kimi-K2.6 (INT4 QAT or NVFP4) on 8× RTX PRO 6000 Blackwell Server Edition (sm_120)

Serve Kimi-K2.6 (1T MoE; MLA; 256K; MoonViT vision) in a user-chosen quantization, with an official-image Docker container — OpenAI-compatible API on :30000, TP=8, weights bind-mounted read-only from local NVMe, all in VRAM. Both the quantization and the engine are chosen at deploy time (steps 2–3) with a hardware-based recommendation:

• INT4 QAT — moonshotai/Kimi-K2.6; compressed-tensors → Marlin (auto); vLLM or SGLang; stock images, no patch. Recommended on sm120 (official, simplest, both engines verified).
• NVFP4 — nvidia/Kimi-K2.6-NVFP4; ModelOpt FP4 (--quantization modelopt_fp4); vLLM only (SGLang NVFP4 = NaN on sm120). Needs a patched CUDA-13 image (build_nvfp4_image.sh) + offline remote-code prep (prep_remote_code.sh). On sm120 it gives no throughput win (PCIe-comm-bound: Marlin ≈ native b12x, native is actually ~12% slower) — prefer it on datacenter Blackwell (sm100/B200) where native FP4 (cutedsl) is tuned, or when you specifically need the NVFP4 checkpoint.

Hardware target: 8× RTX PRO 6000 Blackwell Server Edition (GB202, 96 GB, sm_120) — ~595 GB of weights + KV cache need the full 8×96 GB pool; PCIe-only, no NVLink. Same-chip Workstation/Max-Q variants should behave identically (unverified). The host needs only the NVIDIA driver (≥570, open kernel module, incl. nvidia-persistenced), Docker, and nvidia-container-toolkit — no CUDA toolkit, no Python packages (the download check uses stock python3 + curl).

Why Docker-only (current best practice): the official images ship precompiled sm120 kernels with their own CUDA + glibc, so the host-JIT failure class a native venv fights (glibc≥2.41 rsqrt header conflict, ninja, JIT pre-warm against host CUDA) doesn't exist here, and the deploy reproduces across hosts — both of those sm120 fixes are empirically confirmed unnecessary in-container (image glibc 2.39; the CuTe-DSL norm ICE does not reproduce). GPU access uses CDI (--device nvidia.com/gpu=all, plain runc — not the legacy --gpus runtime hook). Treat step 6 as the go/no-go gate before fronting traffic.

Workflow

1. Verify host — nvidia-smi: GPU model/count/VRAM (this drives the quantization + engine

recommendations in steps 2–3); persistence daemon active (systemctl is-active nvidia-persistenced — ships with the driver and still matters for containers: it keeps GPU state resident across container restarts; ad-hoc fallback sudo nvidia-smi -pm 1); ~650 GB free on local NVMe; GPU containers work via CDI (spec: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml): ``bash docker run --rm --device nvidia.com/gpu=all --entrypoint nvidia-smi ` (Legacy alternative to CDI: --gpus all` — see REFERENCE.md.)

1. Choose the quantization — ask with AskUserQuestion: "Which Kimi-K2.6 quantization?", options

INT4 QAT and NVFP4 (the tool adds "Other"), marking one (Recommended) by the step-1 GPU:

• sm_120 (RTX PRO 6000 Blackwell — reference hardware) → recommend INT4 QAT. Official, both

engines verified, stock images (no patch). NVFP4 here buys no throughput — the box is PCIe-comm-bound (no NVLink), so Marlin-of-INT4 ≈ Marlin-of-NVFP4, and the native FP4 path (flashinfer_b12x) measured ~12% slower than Marlin in a 2026-06-11 A/B; it also needs a patched CUDA-13 image + offline remote-code prep (step 4b). Pick NVFP4 only if you specifically need that checkpoint.

• Datacenter Blackwell (sm_100 / B200) → recommend NVFP4 — native FP4 (FlashInfer cutedsl) is

tuned there; the FP4 tensor cores are the real win. (Not verified by this skill — upstream path.)

• Hopper / Ada / Ampere (pre-Blackwell, no FP4 tensor cores) → INT4 QAT (NVFP4 would only dequant).

Sets the checkpoint + flags — INT4: moonshotai/Kimi-K2.6 (compressed-tensors→Marlin, auto-detected, no --quantization). NVFP4: nvidia/Kimi-K2.6-NVFP4 (--quantization modelopt_fp4, fp8 KV, --disable-custom-all-reduce, --moe-backend marlin|flashinfer_b12x).

1. Choose the engine — AskUserQuestion "Which engine?":

• NVFP4 → vLLM only (don't ask; SGLang NVFP4 = NaN on sm_120, sgl #18954 — serve_docker_sglang.sh

hard-refuses it).

• INT4 → SGLang vs vLLM, mark one (Recommended) by hardware (both verified 5/5, 2026-06-11):
• sm_120 → recommend SGLang — faster at every concurrency (+34% @ c1 … +29% @ c128 vs vLLM) and

serves full 256K at mem-fraction 0.85; vLLM here needs util 0.95 and ~131K with bf16 KV.

• Other hardware → recommend vLLM — model-card primary path, precompiled kernels, no JIT dep.

The (QUANT, FRAMEWORK) pair is the whole deploy identity; it goes in the env file (step 7), not the service name.

3b. NVFP4 only — choose the MoE backend (skip entirely for INT4) — AskUserQuestion "Which NVFP4 MoE backend?":

• marlin (Recommended on sm_120) — measured faster at every concurrency (~12% end-to-end;

the box is PCIe-comm-bound, so the native-FP4 GEMM speedup never reaches the wire) and leaves more KV (b12x reserves extra workspace). The safe throughput pick.

• flashinfer_b12x — the native FP4 tensor-core path (vLLM PR #40082; needs the CUDA-13

patched image step 4b builds anyway). Pick it to exercise the FP4 tensor cores, or on hardware where native FP4 is tuned (datacenter Blackwell uses flashinfer_cutedsl instead). Confirm the dispatch after launch with scripts/assert-native.sh kimi-k26 (NATIVE FP4 vs MARLIN fallback). Sets MOE_BACKEND in the env file (step 7); serve_docker_vllm.sh defaults to marlin if unset. (A/B numbers: the llm-inference-benchmark skill.)

3c. Choose the KV-cache dtype — AskUserQuestion "Which KV-cache dtype?", options derived from the (engine, quant) just chosen; mark the first (Recommended):

• SGLang + INT4 → fp8_e4m3 (Recommended) — verified on the reference host: KV pool

2× (116K→232K tokens, full-256K single request fits), verify 5/5 incl. vision, throughput parity through c64 and +13.5% at c128. | auto (bf16 — engine default, most conservative numerics, pool 116K) | fp8_e5m2 (more exponent range, less mantissa — accepted by SGLang's Triton MLA backend, runtime-unverified here).

• vLLM + INT4 → fp8 (Recommended) — ~2× KV pool, and **required to reach 256K-class

context** on 96 GB GPUs (bf16 KV tops out ~131K MAX_MODEL_LEN). | auto (bf16 — the verified 0.95/131K config).

• vLLM + NVFP4 → fp8 (Recommended — effectively required): the NVFP4 checkpoint ships

FP8 KV scales; serve_docker_vllm.sh already defaults it for QUANT=nvfp4. | auto (ignores the shipped scales; bigger KV bytes — only for debugging numerics). (Why the vLLM rows list no fp8_e5m2: on sm120 vLLM serves Kimi through the TRITONMLA attention backend, whose KV menu is {auto/bf16, fp8 = fp8_e4m3} only — fp8_e5m2, nvfp4, turboquant_* belong to non-MLA backends and hard-fail backend validation at startup. Verified in-image, vLLM 0.22.1. SGLang uses a different MLA kernel, so its row is governed by its own supported set — see the SGLang+INT4 row.) KV dtype changes numerics → re-run verify.sh after switching. Sets KV_CACHE_DTYPE in the env file (step 7). (Measured impact tables: the llm-inference-benchmark skill.)

1. Download checkpoint (~595 GB) into the HF hub cache, pinned to a commit. Respect HF_HOME

(default ~/.cache/huggingface; set HF_HOME to a big-NVMe cache root) — never hardcode paths. hf/Xet may deadlock → the script falls back to parallel curl, verifies size/count vs the paginated HF tree API (curl + python3 stdlib only), writes refs/main: ``bash bash scripts/download.sh moonshotai/Kimi-K2.6 # INT4 bash scripts/download.sh nvidia/Kimi-K2.6-NVFP4 # NVFP4 ``

4b. NVFP4 only — build the patched image + fix offline remote-code (skip entirely for INT4): ``bash bash scripts/build_nvfp4_image.sh # -> kimi-k26-nvfp4-vllm:cu130-mla bash scripts/prep_remote_code.sh nvidia/Kimi-K2.6-NVFP4 # de-symlink snapshot .py (run as cache owner, not root) ` Why: native FP4 MoE needs **CUDA 13**, and Kimi MLA on sm_120 can only use **TRITON_MLA**, whose grouped-decode kernel OOMs at graph capture (smem 102400 > 101376) until the numstages patch; and offline trustremotecode (transformers ≥5.10) can't resolve the custom module's relative imports from the symlinked cache (FileNotFoundError: …/blobs/tooldeclarationts.py`). Both fixes are baked into those two scripts. (REFERENCE.md → "NVFP4 on sm120".)

1. Pull the pinned image, then launch (foreground; DETACH=1 = -d --restart unless-stopped).

QUANT selects the checkpoint + flags; the container is always kimi-k26: ```bash docker pull lmsysorg/sglang:v0.5.12.post1-cu130 # INT4+SGLang (FULL image; Marlin JIT needs ninja+nvcc) docker pull vllm/vllm-openai:v0.22.1 # INT4+vLLM (NVFP4+vLLM uses the locally-built image)

QUANT=int4 bash scripts/servedockersglang.sh # INT4 on SGLang (run the pre-launch gates first — REFERENCE.md) QUANT=int4 bash scripts/servedockervllm.sh # INT4 on vLLM QUANT=nvfp4 IMAGE=kimi-k26-nvfp4-vllm:cu130-mla bash scripts/servedockervllm.sh # NVFP4 on vLLM `` Container: CDI GPUs, --ipc=host --network host (host-net required for NCCL transport / IB-RoCE RDMA; server binds 0.0.0.0, Caddy upstreams over loopback → see REFERENCE "Access model"), weights :ro, memlock/nofile ulimits, HFHUBOFFLINE=1. Load ~10–15 min from NVMe (~4–5 min warm cache); ready on "The server is fired up and ready to roll!" (SGLang) / "Application startup complete" (vLLM) — docker logs -f kimi-k26`.

1. Verify — health, models, text, tool-call, and vision (sent as a base64 data URL):

``bash bash scripts/verify.sh bash scripts/assert-native.sh kimi-k26 # NVFP4: confirm the MoE backend (NATIVE FP4 vs MARLIN fallback) ` Vision debug (dumps content + reasoning_content for a red PNG): python3 scripts/vision-probe.py --model kimi-k2.6. Optional throughput check: use the **llm-inference-benchmark** skill — the canonical bench_sweep.sh`, the sweep methodology, and this hardware's recorded baselines all live there.

1. Productionize — one static kimi-k26.service driven by /etc/kimi-k26.env (selects

FRAMEWORK/QUANT/IMAGE). Only one 595 GB variant fits the 8-GPU pool, so the service name never changes — switch quant/engine by editing the env file + sudo systemctl restart kimi-k26 (no disable/enable). Use the unit or DETACH=1's restart policy, never both. The launcher goes on the root disk (/usr/local/bin) so the service never depends on a /data mount being present at boot. ``bash sudo install -m755 scripts/serve_docker_vllm.sh scripts/serve_docker_sglang.sh /usr/local/bin/ # launcher (root disk) sudo mkdir -p /var/lib/kimi-k26 # service state dir (WorkingDirectory) sudo cp scripts/kimi-k26.env.example /etc/kimi-k26.env # EDIT: FRAMEWORK, QUANT, IMAGE, HF_HOME (BARE values!) sudo cp scripts/kimi-k26.service /etc/systemd/system/kimi-k26.service sudo systemctl daemon-reload && sudo systemctl enable --now kimi-k26 # journalctl -u kimi-k26 -f ` ⚠ **Keep /etc/kimi-k26.env values bare** — systemd EnvironmentFile folds an inline # comment into the value (mangles HFHOME → LocalEntryNotFoundError). fp8 KV: add KVCACHEDTYPE (vLLM fp8, SGLang fp8e4m3) — verified, ~2× KV pool. TLS + Bearer-API-key reverse proxy (engine-agnostic): scripts/setup_proxy.sh. Access model: the --network host server binds **0.0.0.0**, so raw :30000 answers unauthenticated on loopback/ LAN/tailnet — the auth boundary is **Caddy :443** (off-box clients) + the router (no public IP), not the bind. **Caddy upstreams over loopback (127.0.0.1), never the LAN IP** — a LAN-IP upstream is a DHCP time-bomb (lease moves → Caddy 502s with an empty body; cost us two outages). **No host firewall** (setup_proxy retires any legacy kimi-fw/kimi-netguard). The Caddyfile sets admin off, so apply proxy changes with sudo systemctl restart caddy, not reload`. See REFERENCE.md "Access model".

Key facts (don't relearn these the hard way)

• --ipc=host is non-negotiable for TP=8: NCCL needs shared memory and Docker's default 64 MB

/dev/shm breaks it. NCCL "unhandled system error"/SIGBUS right after the load ⇒ check this first.

• INT4 auto-detects (compressed-tensors → Marlin MoE) — no --quantization flag.
• NVFP4 is vLLM-only on sm_120 and needs --quantization modelopt_fp4 + the patched CUDA-13 image

(build_nvfp4_image.sh): native FP4 (--moe-backend flashinfer_b12x) requires CUDA 13 ("b12x fused MoE requires CUDA 13") and the TRITONMLA num_stages smem patch (else graph-capture OOM 102400 > 101376). It genuinely dispatches the FP4 GEMM (no silent dequant) but measured ~12% slower than Marlin on this PCIe-comm-bound box, so --moe-backend marlin is the throughput pick. SGLang NVFP4 = NaN on sm120 — don't. Offline load also needs prep_remote_code.sh or it dies with FileNotFoundError …/blobs/tool_declaration_ts.py (transformers ≥5.10 + symlinked cache).

• Tool calls: SGLang needs only --tool-call-parser kimi_k2; vLLM needs both

--tool-call-parser kimi_k2 and --enable-auto-tool-choice (missing ⇒ no tool_calls).

• The memory knobs are NOT equivalent across engines (measured): SGLang's

--mem-fraction-static = weights+KV pool with transients outside it — 0.85 is the verified ceiling here, 0.90 OOM-crashes the server (vision tower and big-batch transients allocate outside the pool). vLLM's --gpu-memory-utilization caps the total footprint — 0.85 leaves ~0.5 GB for KV (won't start); 0.95 is the working setting, and 256K context needs fp8 KV. FP8 KV (KV_CACHE_DTYPE env in both scripts) doubles the KV pool at zero throughput cost.

• Kimi-K2.6 is a thinking model (reasoning by default) → answer in reasoning_content/content;

disable per request with chat_template_kwargs:{"thinking":false}.

• Vision: send images as base64 data URLs. Server-side image_url URL-fetch gets 403 from

UA-filtering hosts (e.g. Wikimedia) — fixture issue, not a MoonViT failure. vLLM additionally wants --mm-encoder-tp-mode data (SGLang needs nothing extra). Debug an empty vision reply with scripts/vision-probe.py — it shows whether the answer landed in reasoning_content (a `-block artifact, fixed with more max_tokens`) vs a real MoonViT failure.

• MoE weight load is CPU-bound and slow (~10–15 min); high CPU + 0% GPU + quiet logs = normal loading.
• no kernel image is available on sm_120 ⇒ the image predates Blackwell support — bump the tag.
• Cutover: wait for the GPUs to actually free before relaunching. When swapping variants/engines (or

off an ad-hoc deployment), a ~595 GB / 8-GPU teardown takes 30–60 s; starting too soon OOMs the workers at executor init (Engine core initialization failed, before weight load) and a failed init can leak GPU memory that turns into a crash-loop. The serve scripts now wait (WAIT_GPU_FREE); if a loop already leaked, nvidia-smi --query-compute-apps, kill -9 the orphan, confirm GPUs → 0, restart. To preserve the client contract across a migration, keep MODEL_NAME stable (it's the OpenAI model id).

• **Cadd

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: soulmachine
• Source: soulmachine/skills
• License: MIT

Install and usage instructions live in the source repository linked above.