Tag: #llama-cpp14 results
Gemma 4's default vision budget is too low (280 tokens), impairing OCR and details. Use llama.cpp to set --image-min-tokens 560 and --image-max-tokens 2240 for SOTA vision performance. Increase --batch-size to 4096, consuming more VRAM but enabling superior OCR over rivals.
Benchmarks reveal --n-cpu-moe 20 flag boosts Qwen3.6-35B-A3B speed 54% on 16GB GPUs by optimizing VRAM for MoE layers. Achieves 79 t/s gen with 128K context on consumer hardware. Includes full llama.cpp command and VRAM tuning guide.
Unsloth released updated Gemma 4 GGUF models (2B and 27B) in response to recent llama.cpp pull requests. Key fixes include KV-cache for heterogeneous iSWA, CUDA buffer overlap checks, and Gemma 4-specific tokenizer and parser improvements. Links to Hugging Face repos provided.
Tutorial shares setup for 50 tokens/second on RTX 3090 using MTP GGUF Qwen3.6-27B with llama.cpp PR #22673 and 100k context. Includes exact server command with Q4 cache, spec-type mtp, and optimizations. MAC support via mtplx noted.
User achieves 665% speed increase on Devstral small with llama.cpp speculative decoding (--spec-type ngram-map-k, ngram-size 24). Varies by model: Gemma 2x, Qwen 3.6 only 40% initially, improved to 140% with tweaks. Edit adds repeat-penalty and ngram-mod for better results.
Detailed guide to run Claude Code CLI fully offline with Qwen3.5 27B on llama.cpp, including env configs to disable telemetry and server params. Shares results from 7 coding tasks showing speed degradation at high context and compaction issues.
DeepSeekOCR and codefuse-ai/F2LLM-v2 models are now supported in llama.cpp. Update to version b8530 for DeepSeekOCR and b8526 for F2LLM-v2. Users seek guidance on feature extraction and embedding models.
Benchmarks on Strix Halo (Ryzen AI Max+ 395) compare quants of Minimax M2.5, Step 3.5 Flash, Qwen3-Coder-Next, GLM 4.6V/4.7 Flash at 30k context. Run via llama.cpp on ROCm 7.2 with 128GB RAM. More models available on request.
User encounters Nemotron 3 Super interpreting its own reasoning as user input, causing infinite loops in llama-server with Aider. Issue persists even without --special flag, using --jinja. Example shows 8k tokens of meta-recursive thinking.
Tested aging Titan X Pascal GPU for local AI tasks with llama.cpp and OpenCode. Achieved 500 tokens/sec prompt processing and 25 tokens/sec generation, rivaling newer AMD 9070 XT at half the gen speed. Proves old hardware remains viable for inference.