Tag: #gguf28 results
GGUF quantizations (Q4 to Q1) of Minimax M2.5 severely underperform the original model, unlike robust Qwen 3.5 quants. Evaluations on H200 took 10-20 hours per model due to gibberish generation. Key lesson: quantization robustness varies by model.
User asks if llama.cpp with llama-server allows disabling reasoning for specific requests while keeping it enabled by default. Running unsloth/gemma-4-26B-A4B-it-GGUF model. Aimed at faster responses for chatbots.
Reddit user reports Claude-4.6-Opus fine-tunes of local models like Qwen 3.5 consistently underperform base models in intelligence and reasoning. Issues persist across quantizations in llama.cpp setups. Anecdotal evidence urges avoiding such models.
User shares detailed recipe for quantizing GGUF models like Gemma-4-26B-A4B, requiring 500GB storage and architecture-specific configs. Thanks quantizers like unsloth, bartowski; links full REPRODUCE.md on Hugging Face.
User tests show Qwen3.5-27B (Q6_K_XL) rivals Nemotron-120B, Qwen3.5-122B, and GPT-5.4 on dev tasks. Achieves 803pp/25tg on 2x RTX 3090 with 256k context. Replaces API subscriptions for daily coding.
Reddit user asks for the best Qwen3.5-27B quantized models (Q4-Q5, 20-24GB) optimized for coding tasks. Mentions candidates like Unsloth, bartowski, and mradermacher. Seeks advice on proper comparison methods.
Unsloth released Qwen3.5-4B in GGUF format, optimized for low-end hardware. The 4B model features 32 layers, gated DeltaNet/Attention, and context up to 1M tokens. Includes vision encoder and multi-step training.
Jan team releases Jan-Code-4B, a 4B code-tuned model based on Jan-v3-4B-base-instruct for local coding tasks like generation, editing, debugging. It improves over baseline on coding benchmarks and serves as Haiku replacement in Claude Code via Jan Desktop. Model links on HuggingFace with recommended params provided.