Transformers Config Tokenizers Expert

Transformers config + tokenizers expert

Target: engineers writing a preflight tool (or a vLLM/sglang operator) that must decide, before handing a HuggingFace snapshot to an inference engine, which files win, which tokens are structural, and which class will actually instantiate.

Almost every major 2026 release has shipped with drift between tokenizer_config.json, generation_config.json, config.json, and the Rust-backed tokenizer state. The skill exists so a preflight tool can answer that drift authoritatively — not guess.

---

Stance

• Cite, don't paraphrase. Every load-bearing claim has a file:line

or URL citation in references/. Point at the source.

• Version-gate. Transformers v5 (GA 2026-01-26) renamed the

tokenizer classes and changed serialization shapes. Pre-5.0 and post-5.0 diverge — check transformers.__version__ before claiming.

• Rust is truth. For any model with tokenizer.json, the

authoritative added-token state is tokenizer.backend_tokenizer.get_added_tokens_decoder(). Python-side all_special_ids / special_tokens_map / added_tokens_decoder are views; treat them as such.

• Engines disagree. vLLM and sglang both union-merge

generation_config.eos_token_id, but apply it through different pipelines (see engine-knobs.md). Predict per engine, not in the abstract.

---

Triage: symptom → layer → reference

Use this table first. Deep dives live in references/.

| Symptom | Layer | Open | |---|---|---| | tokenizer.eos_token_id disagrees with generation_config.eos_token_id | Config drift | config-files.md#eos-drift | | Engine stops on token X, template emits token Y | Config drift | config-files.md#eos-drift + engine-knobs.md#stop-token-merge | | AutoTokenizer.from_pretrained wants trust_remote_code=True | Class selection | tokenizer-classes.md#tiktoken-path | | KeyError: 'TokenizersBackend' on import | Version gate | tokenizer-classes.md#version-aliases | | AttributeError: 'list' object has no attribute 'keys' on extra_special_tokens | Cross-version serialization | precedence-rules.md#extra-special-tokens-shape | | all_special_ids misses DeepSeek ` tokens | Discovery precedence | precedence-rules.md#backend-fallback | | addedtokensdecoder absent from tokenizerconfig.json | v5 consolidation | config-files.md#v5-consolidation | | Cannot use chat template functions because tokenizer.chattemplate is not set | Template file not wired | chat-template-contract.md#gemma-4-issue-45205 | | chattemplatekwargs silently dropped at request time | Allowlist filter | engine-knobs.md#chat-template-kwargs-allowlist | | enablethinking=false has no effect | Allowlist filter (pre-v0.11.1) | engine-knobs.md#pr-27622 | | Tool-call arguments render as "None" instead of null | Template scalar bug | hall-of-shame.md#gemma-4-ap-45 | | Turn primers (, ) leak into output | EOS list contains turn markers (GLM-5.1) | hall-of-shame.md#glm-5-1-three-id-eos | | Streaming chunks arrive as word fragments | sglang servingchat.py double-slice (#22549) OR vLLM skipspecialtokens=False | engine-knobs.md#incremental-detokenizer | | applychattemplate crashes with UndefinedError on tc.arguments.items() | Arguments arrived as JSON string, not dict | hall-of-shame.md#glm-5-1-ap-45 | | Kimi emits [EOS] but engine expects (or vice versa) | Kimi EOS split-brain | hall-of-shame.md#kimi-k2-6-half-fix` |

---

The precedence cheat sheet (memorize)

Five sources exist for "is this token structural?" They disagree. Reach for them in this order when writing preflight code:

1. tokenizer.backend_tokenizer.get_added_tokens_decoder() — Rust

truth. dict[int, AddedToken]. Every added token, with special flag, lstrip/rstrip/normalized attrs. Source: tokenization_utils_tokenizers.py:488-495 (v5), passthrough. Only available for TokenizersBackend.

1. tokenizer.added_tokens_decoder — Python mirror. For

TokenizersBackend it's a passthrough to #1. For PythonBackend it's deserialized from tokenizer_config.json["added_tokens_decoder"].

1. tokenizer.all_special_tokens / all_special_ids — the narrow

union of SEVEN named role slots (SPECIAL_TOKENS_ATTRIBUTES at tokenization_utils_base.py:1040-1047) + extra_special_tokens. Does not include any added token registered with special=False. This is why DeepSeek's reserved-token slabs and GLM-5.1's ``-as-turn-primer are invisible.

1. tokenizer.special_tokens_map — only the seven role slots as

dict[str, str]. No extras, no added_tokens. Legacy shape.

1. tokenizer.extra_special_tokens — list internally

(self._extra_special_tokens = [] at tokenization_utils_base.py:1074). v5.0.0rc0 serialized this as a list into tokenizer_config.json, crashing ` vs on Instruct | No | | deepseek-ai/DeepSeek-V3 | LlamaTokenizerFast | TokenizersBackend | tokenizer.json (7.85 MB LFS); addedtokensdecoder NOT in tokenizerconfig.json | No | | deepseek-ai/DeepSeek-R1 | LlamaTokenizerFast | TokenizersBackend | / only in chattemplate.jinja, NOT in addedtokensdecoder | No | | microsoft/phi-4 | GPT2Tokenizer | TokenizersBackend | EOS is ; BOS is (inverted vs Qwen-Base) | No | | mistralai/Mistral-Small-24B-Instruct-2501 | LlamaTokenizer | MistralCommonBackend if tekken.json present, else fast | [INST]/[/INST]` at ids 3/4 | No |

Full taxonomy + auto_map mechanics: references/tokenizer-classes.md.

---

Chat-template Jinja rendering contract

Environment built at transformers/utils/chat_template_utils.py:234:

jinja_env = ImmutableSandboxedEnvironment(
    trim_blocks=True, lstrip_blocks=True,
    extensions=[AssistantTracker, jinja2.ext.loopcontrols]
)
jinja_env.filters["tojson"] = tojson                     # ensure_ascii=False default
jinja_env.globals["raise_exception"] = raise_exception   # throws TemplateError
jinja_env.globals["strftime_now"] = strftime_now         # LOCAL TZ, not UTC

Four gotchas operators hit:

1. tojson defaults to ensure_ascii=False — stdlib Jinja's

default is True. Templates that dump CJK/emoji tool schemas rely on this override. A preflight tool that renders in a naive Jinja env will produce HTML-escaped output the model never trained on.

1. strftime_now uses local time. Llama-3.1/3.2 templates inject

a date header; the host's timezone determines the value. A container running in UTC produces different prompts than a laptop in Europe.

1. ImmutableSandboxedEnvironment blocks mutation. Templates

cannot .pop() messages or write to passed objects. Workarounds copy into locals.

1. loopcontrols enables {% break %} and {% continue %}.

Some templates depend on these; a stripped-down renderer missing the extension raises TemplateSyntaxError.

add_generation_prompt semantics, continue_final_message, apply_chat_template resolution order, AssistantTracker offsets: references/chat-template-contract.md.

---

Engine knob precedence (vLLM + sglang)

Short form:

• vLLM chat_template_kwargs: CLI --default-chat-template-kwargs

→ OpenAIServingChat.__init__ default → _prepare_extra_chat_template_kwargs merges with dict-union (request wins) → safe_apply_chat_template → resolve_chat_template_kwargs allowlist filter at vllm/renderers/hf.py:352-377 → tokenizer.apply_chat_template(**resolved). Allowlist fix PR #27622 shipped in v0.11.1 (2025-11-18). Pre-v0.11.1 silently dropped kwargs for tokenizers whose apply_chat_template uses **kwargs (Kimi K2).

• sglang chat_template_kwargs: literal dict update at

serving_chat.py:524-527. No allowlist. Any key reaches apply_chat_template. Closer to pre-27622 vLLM.

• vLLM trust_request_chat_template: default False. Rejects

per-request chat_template or chat_template_kwargs unless set True. Enforced at engine/serving.py:415-425.

• sglang no equivalent: request kwargs always accepted; only

three sites hardcode overrides to skip_special_tokens=False (gpt-oss/gemma4 models, request.tools present, mistral reasoning_effort).

• adjust_request (vLLM): runs at

render/serving.py:372-383, reasoning parser first then tool parser. Can mutate tools, stop, structured_outputs, response_format before to_sampling_params.

• sglang has no adjust_request analog. The three hardcoded

skip_special_tokens=False overrides at serving_chat.py:306/315/397 are the equivalent.

• Stop-token merge:
• vLLM: update_from_generation_config at sampling_params.py:540-560

appends generation_config.eos_token_id list to stop_token_ids unless ignore_eos=True.

• sglang: model_config._get_hf_eos_token_id at model_config.py:580-598

unions hf_config.eos_token_id and hf_generation_config.eos_token_id into Set[int].

• Incremental detokenizer word boundaries: vLLM has fast

(DecodeStream from tokenizers) and slow (detokenize_incrementally with prefix_offset/read_offset diff + U+FFFD guard) paths at vllm/v1/engine/detokenizer.py and vllm/tokenizers/detokenizer_utils.py:98-167. sglang uses DetokenizerManager subprocess with four-offset DecodeStatus at sglang/srt/managers/detokenizer_manager.py:57-63. sglang #22510 was a serving_chat.py double-slice bug (fixed PR #22549, not the detokenizer — despite skip_special_tokens=False being a red herring in the initial report).

Deep dive with file:line per knob: references/engine-knobs.md.

---

Hall of shame (verified 2026)

Pre-loaded real incidents. Each entry in references/hall-of-shame.md has the exact file(s), token IDs, and — where known — the bead ID or commit SHA. Summary:

• GLM-5.1 — three-ID EOS [154820, 154827, 154829] in

generation_config.json. IDs 154827/154829 are ` / turn primers. Engines unioning this list stop on turn boundaries; skipspecialtokens=False leaks them into output. extraspecialtokens as **list**, not dict. TokenizersBackend class name — fails import on transformers ); tokenizer_config.json kept [EOS] (163585). vLLM reads tokenizer_config, sglang reads generation_config. Different engines stop on different tokens. No tokenizer.json; tiktoken` package required.

• Kimi-K2.6 nested-config trap — quantization_config lives at

config["text_config"]["quantization_config"], NOT top-level. Top has only dtype: bfloat16 and an empty/absent quant block. A reader that grabs config["quantization_config"] returns {} and concludes "BF16, no quantization" — wrong. Reality: compressed-tensors, num_bits: 4, group_size: 32, format: pack-quantized (W4A16 routed-MoE INT4 with BF16 carve-outs for lm_head, self_attn.*, shared_experts.*, dense MLP). Total checkpoint 595 GB ≠ ~1 TB BF16. Always walk nested keys — text_config, vision_config, audio_config, language_config are common multimodal/MoE homes. Same trap on K2.5 (same nesting), Llama-4 vision configs, GLM-4V, Qwen3-VL.

• Qwen3-0.6B — `` is simultaneously turn terminator AND

EOS. Qwen3.5-Base flips EOS to ` — preflight hardcoding ` emits runaway completions on base variants.

• DeepSeek-V3 — added tokens live only in tokenizer.json (7.85

MB LFS). tokenizer_config.json has no added_tokens_decoder. `/ on R1 live only in chat_template.jinja`, not as added tokens.

• Phi-4 inversion — EOS `, BOS `.

Opposite of Qwen-Base. Don't regex on string.

Full incidents with citations: references/hall-of-shame.md.

---

Drop-in snippets

references/snippets.py — copy-paste Python for preflight init-time questions:

| Function | Answers | |---|---| | discover_added_tokens(tokenizer, snapshot_dir=None) | Every added token ID, walked Rust→Python→config→tokenizer.json | | resolve_marker_to_id(tokenizer, marker_str) | ID(s) for ` / / etc. Length >1 = vocab collision | | isturnmarkereos(snapshotdir) | [(eosid, content, whereintemplate)] for EOS entries that the template emits as turn primers (leak-on-stream set) | | crossreffiles(snapshotdir) | Drift findings: EOS mismatch, extra_special_tokens shape, special_tokens_map drift, template sidecar-vs-inline | | versiongatetokenizerclass(cfg) | Minimum transformers version (TokenizersBackend → >=5.0; PreTrainedTokenizerFast → >=4.0 alias) | | buildchattemplateenv() | Minimal faithful ImmutableSandboxedEnvironment for offline render testing | | verifycommitreachable(repoid, sha) | Guards against GLM-5.1-FP8-style orphan-commit traps via HF /refs | | findnestedquantizationconfig(config) | Walks textconfig, visionconfig, etc. — catches Kimi-K2.6 W4A16 hidden under textconfig.quantizationconfig while top-level looks BF16. Returns [(dottedpath, value)]. | | summarizequantconfig(qc) | One-line render of a quantizationconfig dict — compressed-tensors numbits=4 groupsize=32 format=pack-quantized ignorepatterns=4 kvcachescheme=None. Surfaces kvcache_scheme:null (no shipped K/V scales → scale=1.0 fallback risk on --kv-cache-dtype fp8`). |

---

Reference map

• references/config-files.md — catalogue per file, drift matrix
• references/tokenizer-classes.md — v5 taxonomy, auto_map, aliases
• references/precedence-rules.md — five-source discovery w/ file:line
• references/chat-template-contract.md — Jinja env, globals, add_generation_prompt
• references/engine-knobs.md — vLLM + sglang tokenizer-adjacent flags
• references/hall-of-shame.md — verified 2026 incidents
• references/snippets.py — drop-in preflight Python
• references/sources.md — dated external references (freshen target)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: air-gapped
• Source: air-gapped/skills
• License: MIT

Install and usage instructions live in the source repository linked above.