๐Ÿ•ท๏ธ
SetupAI
็น
๐Ÿฆ™Stalecollected in 7h

NVFP4 Support Imminent in Llama.cpp GGUF

NVFP4 Support Imminent in Llama.cpp GGUF
PostLinkedIn
๐Ÿฆ™Read original on Reddit r/LocalLLaMA
#quantization#gpu#inferencellama.cppllama.cppggufnvfp4blackwellvllm

๐Ÿ’ก2.3x faster Llama.cpp inference on Blackwell GPUs soon

โšก 30-Second TL;DR

What Changed

NVFP4 merge expected in hours or under a week

Why It Matters

Unlocks efficient local inference on consumer GPUs, reducing memory needs for practitioners.

What To Do Next

Monitor llama.cpp GitHub PRs for NVFP4 merge and test on Blackwell GPU.

Who should care:Developers & AI Engineers

Key Points

  • โ€ขNVFP4 merge expected in hours or under a week
  • โ€ข2.3x speed boost, 30-70% model size savings
  • โ€ขRAM offload support for Blackwell GPUs unlike vLLM

๐Ÿง  Deep Insight

Background and context from public sources โ€” not the original article. 8 sources cited.

๐Ÿ”‘ Enhanced Key Takeaways

  • โ€ขNVFP4 (Native Variable Float Precision 4-bit) is an experimental NVIDIA format leveraging Blackwell GPU's MXFP4 capabilities, with a dedicated pull request #17906 submitted for llama.cpp integration[6].
  • โ€ขCommunity discussions on GitHub highlight ongoing efforts to enable NVFP4 model loading in llama.cpp, including attempts to convert DeepSeek-R1-0528-FP4 safetensors via convert_hf_to_gguf.py[7][8].
  • โ€ขEarly benchmarks on Blackwell PR show competitive performance with ggml-org/gpt-oss-120b-GGUF models using llama-bench, achieving 24.00 ยฑ 1.40 tokens/s at 32k context[6].
๐Ÿ“Š Competitor Analysisโ–ธ Show
Featurellama.cpp (NVFP4)vLLM
Blackwell FP4 OptimizationNative MXFP4 paths via PR[6]Missing optimized paths[6]
RAM OffloadingSupported on Blackwell[1]Not specified for Blackwell
Benchmarks24 t/s on 120B model (experimental)[6]Slower on llama-server equivalent[6]

๐Ÿ› ๏ธ Technical Deep Dive

  • โ€ขNVFP4 refers to NVIDIA's MXFP4 (Microscaling FP4), a 4-bit floating-point format native to Blackwell GPUs (B100/B200), enabling e4m3 or e5m2 precision for weights with dynamic scaling[6].
  • โ€ขImplementation via llama.cpp PR #17906 adds experimental native support, requiring specific ggml-org quant models like gpt-oss-120b-GGUF and flags like --gpu-layers 999 for full offload[6].
  • โ€ขConversion challenges noted for NVFP4 safetensors (e.g., DeepSeek-R1-0528-FP4) using convert_hf_to_gguf.py, which now supports Mixture-of-Experts and lazy loading to avoid full RAM usage[1][8].

๐Ÿ”ฎ Future ImplicationsAI analysis grounded in cited sources

Llama.cpp will outperform vLLM on Blackwell for FP4-quantized models by early 2026
PR #17906 provides native MXFP4 paths absent in vLLM, with initial benchmarks showing superior token throughput on large models[6].
NVFP4 enables 120B+ models on consumer Blackwell GPUs with <50GB VRAM
Combined with GGUF's lazy conversion and quantization, it extends prior ternary/AWQ reductions to FP4, fitting massive models via RAM offload[1][6].

โณ Timeline

2023-08
GGUF format introduced as extensible successor to GGML in llama.cpp
2025-11
convert_hf_to_gguf.py replaces convert.py with MoE and lazy conversion support[1]
2026-01
Experimental MXFP4 benchmarks run on llama.cpp for Blackwell GPUs[6]
2026-02
GitHub discussions and issues emerge on NVFP4 integration for llama.cpp[7][8]
2026-03
PR #17906 submitted for native NVFP4 support in llama.cpp[6]
๐Ÿฆ™Read original article on Reddit r/LocalLLaMA
๐Ÿ“ฐ

Weekly AI Recap

Read this week's curated digest of top AI events โ†’

๐Ÿ‘‰Related Updates

Same topic

Explore #quantization

Same product

More on llama.cpp

Same source

Latest from Reddit r/LocalLLaMA

๐Ÿค–

SHADOW-250M: A 60 MB Long-Context LLM

Reddit r/MachineLearningโ€ขAug 22
๐Ÿฆ™

Tuning Qwen3.8 for 16GB VRAM

Reddit r/LocalLLaMAโ€ขAug 22

AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ†—

This is a summary, not the original. Read the source, or get the weekly briefing.

Weekly AI briefing

One email a week. Unsubscribe anytime.