๐ฆReddit r/LocalLLaMAโขStalecollected in 5h
Testing Qwen 3.6 35B Multi-token Prediction Capabilities
๐กReal-world testing of MTP models shows 1.5x speed gains for local LLM coding tasks.
โก 30-Second TL;DR
What Changed
MTP models provide ~1.5x token generation speed increase
Why It Matters
MTP technology is proving to be a game-changer for local LLM usability, allowing for faster coding and complex project management on home hardware.
What To Do Next
Experiment with MTP-enabled llama.cpp builds if you are hitting latency bottlenecks in your local coding agent workflows.
Who should care:Developers & AI Engineers
Key Points
- โขMTP models provide ~1.5x token generation speed increase
- โขSuccessfully tested 300k context window on 32GB VRAM
- โขRequires specific llama.cpp server prototypes for MTP support
- โขMoE models may face stability issues at extreme context depths
๐ง Deep Insight
Web-grounded analysis with 20 cited sources.
๐ Enhanced Key Takeaways
- โขQwen 3.6 35B A3B is a sparse Mixture-of-Experts (MoE) model, featuring 35 billion total parameters but activating only 3 billion parameters per token during inference, which contributes to its efficiency for local deployment.
- โขThe model is natively multimodal, capable of processing text, image, and video inputs, and incorporates a 'thinking preservation' feature that allows it to retain reasoning context across conversational turns, enhancing its utility for complex agentic tasks.
- โขQwen 3.6 35B A3B boasts a native context window of 262,144 tokens, which can be extended to over 1 million tokens using YaRN scaling, enabling it to handle repository-scale reasoning and large documents.
- โขMulti-Token Prediction (MTP) in Qwen 3.6 is implemented as a built-in speculative decoding mechanism, utilizing multiple output heads within a single transformer to predict several future tokens simultaneously, thereby eliminating the need for a separate draft model and simplifying local inference setups.
- โขOn agentic coding benchmarks, Qwen 3.6 35B A3B demonstrates superior performance, scoring 73.4% on SWE-bench Verified, significantly outperforming competitors like Google's Gemma 4 26B A4B which scored 52.0% on the same benchmark.
๐ Competitor Analysisโธ Show
| Feature/Benchmark | Qwen 3.6 35B A3B | Gemma 4 26B A4B | Qwen 3.6 27B (Dense) |
|---|---|---|---|
| Architecture | Sparse MoE (35B total, 3B active parameters) | Sparse MoE (26B total, 4B active parameters) | Dense (27B total parameters) |
| Multi-token Prediction | Built-in MTP | Supports MTP (via llama.cpp beta) | Supports MTP (via llama.cpp beta) |
| Context Window | 262K native, 1M with YaRN | Not explicitly stated, but competitive with long context models | 262K native, 1M with YaRN |
| Modality | Natively multimodal (text, image, video) | Multimodal reasoning | Multimodal |
| Key Focus | Agentic coding, repository-level reasoning, tool use | General-purpose multimodal reasoning, multilingual | Agentic coding, vision, chat tasks |
| SWE-bench Verified | 73.4% | 52.0% | 75.0% (Qwen3.5-27B, indicative) |
| Terminal-Bench 2.0 | 51.5% | 34.2% (Gemma4-26BA4B) | 41.6% (Qwen3.5-27B, indicative) |
| Token Generation Speed | ~120+ tokens/sec (RTX 4090, llama.cpp/vLLM), 3.5-4x faster than 27B | ~300 tok/s (M2 Ultra, Q8), 40% faster with MTP (M5Max) | Slower than 35B-A3B (3.5-4x slower) |
| VRAM (Q4_K_M) | ~21 GB (fits 24GB GPU) | Not explicitly stated for 26B A4B, but 31B BF16 is ~20GB | ~18 GB (fits 18GB RAM setups) |
| License | Apache 2.0 | Open-source (Apache 2.0 for some variants) | Apache 2.0 |
| API Pricing | $0.15/1M input, $1/1M output (OpenRouter) | Not explicitly stated for 26B A4B, but generally competitive | Not explicitly stated |
๐ ๏ธ Technical Deep Dive
- Model Architecture: Qwen 3.6 35B A3B employs a hybrid sparse Mixture-of-Experts (MoE) architecture. It features 35 billion total parameters, but only approximately 3 billion are actively engaged per token during inference, significantly enhancing computational efficiency.
- Attention Mechanism: The architecture integrates Gated DeltaNet linear attention layers with standard Gated Attention layers.
- Multi-Token Prediction (MTP) Implementation: MTP is built directly into the model, utilizing a shared transformer backbone with multiple output heads. Each head is trained to predict a different future token (e.g., token 2, token 3, token 4) simultaneously, allowing for the drafting of multiple tokens in a single forward pass, which are then verified. This internal mechanism removes the need for a separate draft model.
- Context Window: The model natively supports a context length of 262,144 tokens and can be extended to 1,010,000 tokens through YaRN (Yet another RoPE extension) scaling.
- Thinking Preservation: A notable feature is 'thinking preservation,' which enables the model to retain its reasoning traces from previous conversational turns. This is particularly beneficial for multi-step agentic workflows, where maintaining coherent context is crucial.
- Expert Configuration: The MoE layer consists of 256 experts, with 8 routed experts and 1 shared expert actively selected per token.
- Tokenizer and Layers: The model uses a tokenizer with a vocabulary size of 248,320 and is composed of 40 layers.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Local AI development will experience accelerated innovation and adoption.
The ability of models like Qwen 3.6 35B MTP to deliver frontier-level performance and large context windows efficiently on consumer hardware significantly lowers the barrier to entry for developers and researchers, fostering a more dynamic local AI ecosystem.
Agentic AI applications will become more sophisticated and reliable on edge devices.
Features such as 'thinking preservation' and extended context windows directly address critical challenges in maintaining state and coherence for complex, multi-step agentic workflows, enabling more robust local AI agents.
The competitive landscape between open-source and proprietary cloud-based LLMs will intensify.
With open-source models achieving comparable or superior performance to proprietary cloud offerings in specific domains (e.g., agentic coding) while offering local deployment advantages, the market will likely see increased pressure on cloud providers to innovate further or adjust pricing.
โณ Timeline
2023-04
Alibaba launched Qwen (Tongyi Qianwen) in beta.
2023-09
Qwen opened for public use after regulatory clearance.
2024-06
Qwen2 model series released.
2025-04
Qwen3 model family released, including dense and MoE variants, trained on 36 trillion tokens.
2026-02
Qwen3.5 series released, featuring native multimodal capabilities and Multi-Token Prediction (MTP).
2026-03-31
Qwen 3.6 Plus, a proprietary cloud model with a 1 million token context window, was released.
2026-04-16
Qwen 3.6 35B A3B, an open-source, open-weight MoE model with MTP, 262K native context, and agentic coding focus, was released.
2026-05-04
llama.cpp merged beta support for Multi-Token Prediction (MTP), initially for Qwen3.5 MTP compatible models.
๐ Sources (20)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ