Vllm Chat Templates

vLLM chat templates — operator triage

Target audience: operators deploying vLLM in production. Assumes OpenAI-API-compatible frontend (/v1/chat/completions or /v1/responses), multi-GPU, mid-2024 through 2026 model families.

Scope — three sibling skills, three layers

Chat-template bugs span three layers. This skill owns the prompt-rendering layer; route to a sibling when the problem is output-side.

| Layer | Direction | Skill | What it covers | |---|---|---|---| | Chat template (Jinja) | messages → prompt string | this skill | template precedence, --chat-template, chat_template_kwargs, examples/tool_chat_template_*.jinja, bundled fallbacks | | Reasoning parser | model output → reasoning_content + content | vllm-reasoning-parsers | --reasoning-parser, extract_reasoning, is_reasoning_end, ` splitting | | **Tool parser** | model output → tool_calls[] | vllm-tool-parsers | --tool-call-parser`, streaming state machines, partial-JSON parsing |

When in doubt: if the operator complains about what the server receives, it's this skill; if about what the client receives, it's one of the parser skills. Template and parser bugs often present the same symptom ("reasoning_content is null"), so all three skills name each other in diagnostics. This file stays on the Jinja side.

Why this matters

Chat templates are the Jinja layer between structured messages and the raw prompt string the model sees. Tool-calling and reasoning models are extremely sensitive to this layer — a one-character whitespace difference in a template can send tool-call success rate from 99% to 0%, silently.

The hardest class of bug in vLLM deployments is template silent failure: the server starts, requests return 200s, but the model emits garbled tool calls, or reasoning_content is null, or enable_thinking=false has no effect. No exception, no log line, just wrong outputs. Operators often chase parser bugs when the real fix is a template override (or vice versa — override a template when the real fix is a parser flag).

This skill's job: given a symptom, identify whether it's a template, a parser, a kwarg, or an API-endpoint issue, and point at the known fix.

Template resolution precedence (memorize this)

vLLM picks a chat template in this order. First hit wins. Source: vllm/renderers/hf.py:96-145 (resolve_chat_template()).

1. --chat-template CLI flag — highest. Path to a .jinja file, literal Jinja string, or a named template ("tool_use") that the tokenizer knows.
2. AutoProcessor chat template — only consulted when the request has tools=None. Multimodal models often expose their template via the processor, not the tokenizer. Source: hf.py:110-116.
3. AutoTokenizer chat_template — the one shipped in tokenizer_config.json. Source: hf.py:119-126.
4. Bundled fallback in vllm/transformers_utils/chat_templates/ — vLLM ships defaults for ~10 vision/OCR model families (blip-2, chameleon, clip, colpali, deepseekocr/ocr2/vlv2, fuyu, minicpmv, paligemma, qwen, siglip/siglip2). Source: registry.py:64-75.

If none resolves, ChatTemplateResolutionError raised at hf.py:477 with: "As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one."

Consequences operators miss

• Passing --chat-template does NOT override the tokenizer's template if the path is invalid — vLLM raises "The supplied chat template ... appears path-like, but doesn't exist!" (chat_utils.py:1105-1109). Check ls first.
• Tool-calling disables the AutoProcessor step. If a multimodal model has a processor template but the request has tools, vLLM skips it and falls through to tokenizer. This is intentional — tool templates must be explicit.
• --tokenizer-mode mistral silently ignores --chat-template. Warning only; see Mistral section in references/model-families.md. Issue vllm-project/vllm#25401 — closed as COMPLETED 2025-10-09. Behavior is at least louder in current vLLM; verify on your version before assuming silent-drop. Verified 2026-04-24.
• Some templates (e.g. gpt-oss) appear hard-coded — operator-supplied --chat-template has no effect (vllm-project/vllm#23015, closed NOT_PLANNED 2026-04-20). Intentional: gpt-oss goes through the Harmony path via /v1/responses; --chat-template is not the intended escape hatch. Verified 2026-04-24.

Triage: symptom → suspect

Use this table first. Load references/model-families.md for the specific model bug + flags.

| Symptom | First suspect | Second suspect | |---|---|---| | reasoning_content is null or missing from response | --reasoning-parser flag missing/wrong | Template strips opening ` before model generates it (DeepSeek-R1) | | toolcalls empty array [] but model emitted text | --tool-call-parser wrong for model family | toolchoice="auto" (try "required" or a named tool) | | Tool calls have malformed JSON / arguments empty | Wrong tool-call parser (Hermes vs pythonic vs XML vs model-specific) | Template doesn't inject tools correctly | | enablethinking=False has no effect | Kwarg dropped by vLLM allowlist (need PR #27622 or ≥v0.11.1) | Downstream harness not forwarding chattemplatekwargs | | Tool calling works on /v1/completions but not /v1/chat/completions | Chat template doesn't render tool result (GLM-5.1 #39611, fixed 2026-04-12) | Endpoint-specific parser path | | Prompt in server log doesn't match model card | Wrong template resolving — check precedence order above | --chat-template pointing at file that doesn't exist (silent for some tokenizer modes) | | content leaks into final output | --reasoning-parser missing | skipspecialtokens=true stripping control tokens before parser (#38855) | | Garbled prompt with repeated placeholders | Template has placeholder AND multimodal parser adds one | interleavestrings mismatch — see chatutils.py:1186-1241 | | IndexError / Jinja exception on multi-turn tool calls | Tool-call ID format regex too strict (Kimi-K2, Mistral) | Historical toolcalls=[] in assistant msg crashes template (Llama #13978) | | Server starts, first request works, subsequent requests fail | State-ful template bug: leading whitespace accumulating (DeepSeek V3.1 #28804) | Concurrent-request regression (Gemma-4 tokens #39392) | | "this model is reasoning but reasoning_effort=none" | gpt-oss #37909 — "none" not honored | Wrong reasoning parser | | Works on SGLang / TGI, broken on vLLM | vLLM applies template differently (HF precedence + kwarg allowlist) | Check --tokenizer-mode` differences |

Template, parser (output), and kwarg (chat_template_kwargs) are orthogonal layers — diagnose one at a time, don't change multiple at once.

The flag matrix (common case)

Per-family flag matrix lives in references/flags-matrix.md. Quick cheat-sheet for the production-critical ones:

| Model family | --chat-template | --tool-call-parser | --reasoning-parser | |---|---|---|---| | Qwen3 dense / MoE | tokenizer default | hermes | qwen3 | | Qwen3-Coder | tokenizer default | qwen3_xml | — | | DeepSeek-R1 | examples/tool_chat_template_deepseekr1.jinja | deepseek_v3 | deepseek_r1 | | DeepSeek-V3-0324 | examples/tool_chat_template_deepseekv3.jinja | deepseek_v3 | — | | DeepSeek-V3.1 | examples/tool_chat_template_deepseekv31.jinja | deepseek_v31 | deepseek_r1 | | DeepSeek-V3.2 | (uses --tokenizer-mode deepseek_v32) | deepseek_v32 | deepseek_r1 | | GPT-OSS 20B/120B | prefer /v1/responses endpoint | openai (broken on chat-completions #22578) | harmony-native | | Kimi-K2 | tokenizer default (post-fix commit) | kimi_k2 | kimi_k2 | | Llama-4 Maverick/Scout | examples/tool_chat_template_llama4_pythonic.jinja | llama4_pythonic | — | | Mistral (HF format) | examples/tool_chat_template_mistral_parallel.jinja | mistral | — | | Mistral (mistral tokenizer-mode) | ignored (don't set) | mistral | — | | Gemma-3 | examples/tool_chat_template_gemma3_pythonic.jinja | pythonic | — | | Gemma-4 | examples/tool_chat_template_gemma4.jinja | gemma4 (flaky concurrent) | gemma4 | | Phi-4-mini | examples/tool_chat_template_phi4_mini.jinja | llama3_json | — | | Phi-4-reasoning | tokenizer default | — | not deepseek_r1 (#18141) | | Hermes / Hunyuan / Granite / Toolace / XLAM | examples/tool_chat_template_.jinja | hermes/matching | — |

Commit-gated models (Kimi-K2) require minimum HF revision — check references/model-families.md before recommending.

Top 15 patterns that silently break production

Consult this list for any "broken chat template" symptom. Most reports reduce to one of these.

1. Chat template silently ignored. Mistral in --tokenizer-mode mistral (#25401), reportedly gpt-oss (#23015). Warning-only. Symptom: edit template, nothing changes. Fix: switch to HF tokenizer mode, or accept that the built-in Mistral template is canonical.

1. skip_special_tokens=true strips control tokens before reasoning parser. Gemma-4 (#38855), Qwen3 reasoning, gpt-oss harmony. Fix: extra_body={"skip_special_tokens": false} per request.

1. **` opening tag prepended to prompt → never appears in output → reasoning_content=null.** DeepSeek-R1 (HF #144, vLLM #12999). Template forces thinking by inserting \n in the prefill; parser sees only . Fix: post-process prepend `, or use patched template.

1. Empty string '' wrapped to [{'type':'text','text':''}]. Broke Kimi-K2 templates expecting strings. Any template that doesn't type-check message.content is at risk. Fix: updated chat template or vLLM version.

1. Tool-call ID regex mismatch. Mistral requires 9 digits; Kimi-K2 required functions.X:idx exactly. vLLM generates longer IDs → IndexError → all tool calls silently dropped. Fix: Kimi-K2 fixed in PR #27565; Mistral template truncates to last 9 digits.

1. add_generation_prompt dropped by kwarg allowlist. PR #25794 restricted **kwargs forwarding for security; tokenizers relying on non-standard params broke. Fix: PR #27622 whitelists standard HF params — requires vLLM ≥ v0.11.1.

1. Jinja template from disk triggers _try_extract_ast() exception. #14884: \n escape sequences double-converted when shell-interpolated. Fix: pass the file path, not the contents; don't shell-escape.

1. tool_calls=[] in historical assistant message crashes template. #13978 (Llama). Templates assume nonempty if present. vLLM now drops empty tool_calls (chat_utils.py:1586-1589).

1. tool_choice="auto" vs "required" divergence. auto = free-form generation + regex parse (schema-free, brittle). required/named = structured-outputs FSM (schema-valid). Many "tool calling is broken" reports vanish when switching to required.

1. Multi-turn whitespace accumulation. DeepSeek V3.1 (#28804, closed NOT_PLANNED 2026-03-23) — leading whitespace grows each turn. Look for this when assistant replies get progressively indented. Upstream not planning a fix; apply client-side mitigation (strip leading whitespace on each turn) or use deepseek_v31 parser commits post-issue report. Verified 2026-04-24.

1. reasoning_effort="none" silently breaks output. gpt-oss #37909. Fix: omit or use "low".

1. **Concurrent requests → ` tokens.** Gemma-4 #39392. Not reproducible sequentially. Fix: version bump; interim workaround --max-num-seqs 1`.

1. --chat-template-content-format auto misroutes tool results. GLM-5.1 #39614 (CLOSED/COMPLETED 2026-04-25): tool result as {"type":"text","text":"..."} hits an "else" branch in a template checking .name → renders ` instead of result. Fixed upstream; the --chat-template-content-format openai/string` workaround is only needed on vLLM before the fix.

1. /v1/chat/completions vs /v1/responses divergence. gpt-oss tool calling works only on /v1/responses (#22578). GLM-5.1 (#39611, CLOSED/COMPLETED 2026-04-12) — tool results were ignored on chat-completions but work on completions; fixed upstream, so on patched vLLM tool results now render on /v1/chat/completions.

1. API field name mismatch. reasoning (vLLM older) vs reasoning_content (DeepSeek/OpenAI-style). Clients that hard-code one break on version bump. vLLM settled on reasoning_content (#28472).

What chat_template_kwargs accepts

Threaded as extra_body={"chat_template_kwargs": {...}} in OpenAI client. Source: vllm/renderers/params.py:70-125, hf.py:407-435.

• Any Jinja variable referenced in the resolved template — detected via AST parse at hf.py:429.
• HF apply_chat_template params auto-detected from tokenizer signature (hf.py:387-404).
• add_generation_prompt (bool)
• continue_final_message (bool) — mutually exclusive with add_generation_prompt=True
• documents (list of {title, contents}) for RAG templates
• enable_thinking (Qwen3 and some others)
• Reserved (will raise ValueError): chat_template, tokenize (hf.py:416-421).
• Unknown kwargs silently filtered (hf.py:434-435) — if enable_thinking doesn't appear in the detected Jinja vars, it's dropped without warning. This is a confusing failure mode: operator sets it, sees no effect, can't find an error.

Merging: server defaults (serving.py:111) ← request kwargs (params.py:28-40) — request wins.

Reasoning parser vs chat template (they're confused a lot)

• Chat template (this skill) controls the prompt — where `` tags go going in to the model, and how tool definitions get rendered.
• Reasoning parser (vllm-reasoning-parsers) extracts ` content from *output* into reasoningcontent. Template-side causes of null reasoningcontent are pattern #3 above (opening ` prepended in prefill); all other root causes live in the parser skill.
• Tool-call parser (vllm-tool-parsers) extracts tool_calls[] from output. Template-side causes of empty tool_calls are wrong tool template or tool_choice="auto" vs "required" (pattern #9); parser-selection and streaming state-machine debugging live in that skill.

Raw-output capture (--disable-log-requests=False + no --tool-call-parser/--reasoning-parser) determines which layer is at fault: if the model emits the expected tags/JSON, the prompt is fine and the problem is parser-side; if not, the template is misrendering.

Multimodal template notes

Models use model-specific placeholders (`, , etc.) — vLLM's internal placeholder is / / (chat_utils.py:96-100`).

• If request has images but template has no placeholder, vLLM prepends them to final prompt (chat_utils.py:1233-1240). Some models tolerate this; others produce garbage.
• If template has more placeholders than media items: ValueError: "Found more '' placeholders in input prompt than actual multimodal data items." (chat_utils.py:1226-1229).
• interleave_strings=True (via multimodal_config.interleave_mm_strings) enables in-order substitution; default is prepend-all.

For Pixtral, Mistral-Small-3.1 multimodal, and other MM-tool-calling combinations, see references/model-families.md.

When nothing works — full debugging workflow

Load references/debugging.md. The short version:

1. Log the actual prompt. --disable-log-requests=False + log level DEBUG. Compare against model card example. If they differ, template is wrong.
2. Bypass vLLM. Load the same tokenizer in a Python REPL, call tokenizer.apply_chat_template(messages, add_generation_prompt=True), compare. If same bad output, template is wrong. If different, vLLM's kwarg forwarding is eating something.
3. Capture raw model output. Set --tool-call-parser to none, read what the model actually emits. Then work backward to pick the right parser.
4. Pin versions. Record vllm --version, transformers.__version__, model revision SHA. Many template bugs track

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: air-gapped
• Source: air-gapped/skills
• License: MIT

Install and usage instructions live in the source repository linked above.