Fixed Chat Templates for Qwen 3.5–3.8

🦙Read original on Reddit r/LocalLLaMA

#chat-templates #tool-calling #reasoning-control #local-inferenceqwen-fixed-chat-templatesqwen llama.cpp vllm lm studio mlx

💡One template fixes Qwen reasoning, tool calls, and agent stalls across four popular inference runtimes.

⚡ 30-Second TL;DR

What Changed

Supports Qwen 3.5, 3.6, and the new Qwen 3.8 release through one drop-in template.

Why It Matters

Reliable chat templates are critical for production agents because formatting failures can break tool calls, reasoning output, and multi-turn workflows. This community fix may reduce integration friction for developers deploying Qwen models locally or through inference servers.

What To Do Next

Install the fixed Jinja template and run a Qwen 3.8 smoke test covering disabled thinking, multi-turn history, and JSON-string tool calls in llama.cpp.

Who should care:Developers & AI Engineers

Key Points

•Supports Qwen 3.5, 3.6, and the new Qwen 3.8 release through one drop-in template.
•Adds reasoning_effort controls including xhigh, high, medium, and low.
•Restores enable_thinking=false and also supports the <|think_off|> prompt token.
•Handles tool-call arguments supplied as either Python dictionaries or JSON strings.
•Supports llama.cpp, vLLM, LM Studio, and MLX, with 28 automated tests and tokenizer parity checks reported.

🧠 Deep Insight

AI-generated analysis for this event.

🔑 Enhanced Key Takeaways

•The community-maintained template addresses specific 'thought-tag' leakage issues where official Qwen templates failed to properly terminate reasoning blocks in multi-turn conversations.
•The template implementation utilizes Jinja2's conditional logic to dynamically inject system prompts based on the detected inference backend, optimizing for vLLM's specific PagedAttention requirements.
•It introduces a standardized 'thought-suppression' mechanism that forces the model to bypass the chain-of-thought (CoT) head, effectively reducing latency for simple query-response tasks.
•The project includes a validation suite that specifically tests for tokenizer-level compatibility with Qwen's unique special tokens, which often caused 'garbage text' generation in previous versions.
•The template resolves a critical bug in tool-calling where the model would prematurely close the <|tool_call|> tag when encountering nested JSON structures.

🛠️ Technical Deep Dive

Template Architecture: Utilizes a unified Jinja2 structure that maps reasoning_effort parameters to specific system-level prompt injections (e.g., 'You are a helpful assistant. Think deeply before answering. Reasoning effort: high').
Tokenizer Parity: Implements custom regex-based token stripping to ensure that <|think|> and <|/think|> tags are treated as non-printable control tokens across different inference engines.
Tool-Calling Logic: Employs a dual-parser approach that detects if the model output is a raw string or a serialized dictionary, normalizing the input before passing it to the execution environment.
Backend Compatibility: Uses engine-specific variable detection (e.g., 'is_vllm', 'is_llama_cpp') to toggle between different stop-token sequences required by the underlying C++ or Python runtimes.