Test Qwen Models on llama.cpp Not Ollama

💡Qwen issues? Blame Ollama—switch to llama.cpp for real results

⚡ 30-Second TL;DR

What Changed

Long CoT, tool calls, garbage outputs specific to Ollama/LMStudio

Why It Matters

Encourages switching runtimes, improving new model evaluations and reducing false negatives on capabilities.

What To Do Next

Deploy llama.cpp server to retest new Qwen models for accurate performance.

Who should care:Developers & AI Engineers

Background and context from public sources — not the original article. 9 sources cited.

•Qwen3 models, including Qwen3-30B-A3B, achieve 342K token context length and 20–22 tokens/sec throughput on 80GB GPUs using llama.cpp's Vulkan backend[1].
•Qwen3-Coder-Next requires specific flags like --tool-call-parser qwen3_coder and Q6_K quantization in llama.cpp or vLLM to resolve tool calling issues[2].
•MLX on Mac has KV cache inconsistencies for Qwen3 during conversation branching, making llama.cpp the superior choice over MLX or optimized LM Studio[2][3].
•Qwen3 supports 1M token context via hybrid attention (Gated DeltaNet + MoE) with near-linear compute scaling, natively handled by llama.cpp GGUF[4].

📊 Competitor Analysis▸ Show

Feature	llama.cpp	Ollama	LM Studio	vLLM	SGLang
Qwen3 Tool Calling	Full support with custom parsers	Issues in tags	Lacks presence penalty	Native MoE/expert routing	Native MoE/expert routing
Context Length	Up to 342K+ (Vulkan)	Limited by runtime bugs	Hardware-dependent	1M hybrid attention	High-throughput long context
Quantization	Q4_K_XL, Q5_K_M, Q6_K	Basic GGUF	Optimized MLX on Mac	Server-optimized	Server-optimized
Benchmarks (tokens/sec)	20-22 on 80GB GPU	Inferior for most cases	Slow on branching convos	High for agents	High for agents
Pricing	Free/open-source	Free/open-source	Free (proprietary UI)	Free/open-source	Free/open-source

•Qwen3 uses hybrid attention alternating Gated DeltaNet (linear attention, 3:1 ratio with full attention) for 1M token contexts with near-linear scaling[4].
•MoE architecture (e.g., Qwen3-235B-A22B) natively supported in llama.cpp via GGUF quantization (Q4_K_XL, Q5_K_M, Q6_K)[1][2][4].
•Vulkan backend in llama.cpp optimizes Qwen3-30B-A3B inference speed/memory; ROCm/Vulkan enables AMD GPU support (e.g., Radeon 7900 XTX)[1][2].
•Tool calling fixed with --tool-call-parser qwen3_coder, updated llama.cpp/vLLM, and recommended sampling parameters[2].