來源Reddit r/LocalLLaMA•較早收集於 2h
RTX 5070 Ti 上 Qwen3.6-35B 以 --n-cpu-moe 達 79 t/s
#moe-optimization#gpu-tuning#llama-cppqwen3.6-35b-a3bqwen3.6-35b-a3bllama.cppunslothrtx-5070-ti
💡Qwen3.6 MoE 在 16GB GPU 上速增 54% + 128K 上下文（22字）
⚡ 30 秒速覽
有什麼變化
--n-cpu-moe 20 將生成速度從 51 提升至 79 t/s，VRAM 從 3.5 至 12.7 GB
為什麼重要
解鎖中階 GPU 上高速度、長上下文的本地 MoE 推論，讓強大模型無需企業級硬體即可使用。
下一步行動
在 llama.cpp 中將 Qwen3.6-35B 于 16GB GPU 改用 --n-cpu-moe 20 + -np 1。
誰應關注：Developers & AI Engineers
關鍵要點
- •--n-cpu-moe 20 將生成速度從 51 提升至 79 t/s，VRAM 從 3.5 至 12.7 GB
- •在 RTX 5070 Ti + 9800X3D 上以 -np 1 支援 128K 上下文「幾乎免費」
- •調校指南：16GB 用 N=20，每 MoE 層約 530 MB VRAM
- •使用 unsloth UD-Q4_K_M GGUF 與 llama.cpp b8829
🧠 深度解析
本篇為 AI 生成分析，非原文內容。
🔑 增強重點摘要
- •The --n-cpu-moe flag functions by offloading a specific subset of Mixture-of-Experts (MoE) layers to the CPU while keeping the dense layers on the GPU, effectively bypassing VRAM bottlenecks for models that would otherwise exceed 16GB capacity.
- •The RTX 5070 Ti's architecture, specifically its improved memory controller and cache hierarchy, is critical to maintaining the 79 t/s throughput when the CPU-GPU interconnect (PCIe Gen5) is stressed by the offloaded MoE layers.
- •The performance gain is highly dependent on the 9800X3D's large L3 cache, which mitigates the latency penalty typically associated with CPU-based MoE layer computation in llama.cpp.
📊 競品分析▸ Show
| Feature | Qwen3.6-35B (via --n-cpu-moe) | DeepSeek-V3 (Distilled) | Llama-4-30B |
|---|---|---|---|
| Architecture | MoE (35B total) | MoE (671B/37B active) | Dense |
| VRAM Req (Q4) | ~13GB (with offload) | ~24GB+ | ~18GB |
| Throughput (16GB GPU) | 79 t/s | 12-15 t/s | 45 t/s |
🛠️ 技術深入
- MoE Layer Offloading: The --n-cpu-moe N parameter dictates the number of expert layers moved to system RAM. At N=20, the model utilizes the CPU's AVX-512/AMX instructions to process expert weights in parallel with GPU dense layer inference.
- Memory Mapping: The implementation relies on mmap-based GGUF loading, allowing the OS to manage page faults for the offloaded layers, which is why system RAM speed (DDR5-6400+) is a secondary performance bottleneck.
- Context Handling: The 128K context window is managed via Flash Attention 3 kernels on the GPU, while the KV cache for the offloaded layers is stored in system RAM, necessitating high-bandwidth memory access.
🔮 前景展望基於引用來源的 AI 分析
Consumer hardware will support 100B+ parameter models at usable speeds by Q4 2026.
The success of hybrid CPU-GPU MoE offloading demonstrates that VRAM capacity is no longer a hard ceiling for local inference of massive models.
llama.cpp will introduce automated 'smart-offloading' heuristics.
Manual tuning of --n-cpu-moe is currently required, but the performance delta suggests that dynamic profiling will become a standard feature to optimize for specific GPU/CPU pairings.
⏳ 時間線
2025-11
Qwen3.0 series release introduces improved MoE architecture for consumer hardware.
2026-01
llama.cpp adds experimental --n-cpu-moe flag to support hybrid inference.
2026-03
Qwen3.6-35B-A3B model released with optimized expert routing.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： Reddit r/LocalLLaMA ↗
每週電子報
每週一封，可隨時退訂。