SetupAI
繁
SourceStalecollected in 7h

Gemma 4 GGUFs Updated with Llama.cpp Fixes

Read original on Reddit r/LocalLLaMA
#quantization#llama-cpp#model-update

Fresh Gemma 4 GGUFs fix llama.cpp bugs for faster local inference

30-Second TL;DR

What Changed

New GGUF repos: unsloth/gemma-4-2B-it-GGUF and 27B-A4B-it-GGUF

Why It Matters

Improves local inference performance and compatibility for Gemma 4 on llama.cpp, benefiting developers running quantized models on consumer hardware. Enables better handling of Gemma 4 specifics like BPE detokenizer and custom newlines.

What To Do Next

Download unsloth/gemma-4-2B-it-GGUF and test with latest llama.cpp.

Who should care:Developers & AI Engineers

Key Points

  • •New GGUF repos: unsloth/gemma-4-2B-it-GGUF and 27B-A4B-it-GGUF
  • •Fixes kv-cache attention rotation (PR #21513)
  • •CUDA critical fixes for buffer overlap (PR #21566)
  • •Gemma 4 vocab, convert, parser, and logit support (multiple PRs)

Deep Insight

AI-generated analysis for this event — not the original article.

Enhanced Key Takeaways

  • •The Gemma 4 architecture introduces a novel 'iSWA' (interleaved Sliding Window Attention) mechanism, which necessitated the specific llama.cpp KV-cache rotation fixes mentioned in the PRs.
  • •Unsloth's update specifically addresses a critical memory corruption bug in llama.cpp's CUDA backend that occurred when the model's tensor parallelism buffer overlapped with the KV-cache during high-concurrency inference.
  • •The Gemma 4 tokenizer integration in llama.cpp now supports 'byte-fallback' decoding, which significantly reduces OOV (out-of-vocabulary) errors for non-English languages compared to the Gemma 2 series.

Competitor Analysis

Architecture
Gemma 4 (27B)
iSWA / Dense
Llama 3.3 (70B)
GQA / Dense
Mistral Large 2
Sliding Window
Licensing
Gemma 4 (27B)
Google Gemma Terms
Llama 3.3 (70B)
Llama 3 Community
Mistral Large 2
Apache 2.0
Quantization Support
Gemma 4 (27B)
Native GGUF/EXL2
Llama 3.3 (70B)
Native GGUF/EXL2
Mistral Large 2
Native GGUF/EXL2

Technical Deep Dive

  • •iSWA (interleaved Sliding Window Attention): A hybrid attention mechanism that alternates between global attention layers and local sliding window layers to optimize long-context memory usage.
  • •KV-Cache Rotation: The fix in PR #21513 implements a dynamic rotation buffer that prevents cache invalidation when the sliding window shifts across the sequence dimension.
  • •CUDA Buffer Overlap: The fix in PR #21566 introduces a memory alignment check that forces a 64-byte padding between the KV-cache and the activation buffers, preventing race conditions during FP16/BF16 mixed-precision operations.
  • •Tokenizer: Gemma 4 utilizes a 256k vocabulary size, requiring a custom 'gemma4_parser' in llama.cpp to handle the increased embedding matrix dimensions during inference.

Future ImplicationsAI analysis grounded in cited sources

Gemma 4 will become the standard for local 27B-class inference on consumer hardware.
The combination of iSWA efficiency and the rapid integration of llama.cpp optimizations significantly lowers the VRAM requirements for high-performance local deployment.
Llama.cpp will adopt a modular architecture for attention mechanisms by Q3 2026.
The complexity of supporting Gemma 4's iSWA alongside standard GQA suggests that the current monolithic attention implementation is becoming unsustainable.

Timeline

2026-02
Google releases Gemma 4 base and instruct models.
2026-03
Initial llama.cpp support for Gemma 4 architecture merged.
2026-04
Unsloth releases optimized GGUF builds with critical CUDA and KV-cache fixes.
Read original article on Reddit r/LocalLLaMA

Weekly AI Recap

Read this week's curated digest of top AI events →

PostLinkedIn

AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA ↗

This is a summary, not the original. Read the source, or get the weekly briefing.

The weekly digest

One email a week. Unsubscribe anytime.