Tag: #gguf28 results
Comprehensive Q4 quantization comparison of Qwen3.5-27B GGUF files using KLD against BF16 baseline on custom chat and Wikitext2 datasets. unsloth_Qwen3.5-27B-UD-Q4_K_XL leads with lowest KLD of 0.005087. IQ4_XS variants top efficiency for VRAM-size balance.
An uncensored version of the new Qwen3.5-4B model has been released in GGUF format with zero refusals out of 465 tests. It retains full capabilities, supports multimodal inputs, and offers various quants from 2.6GB to 7.9GB. Upcoming uncensored variants for larger Qwen3.5 sizes are in progress.
mradermacher uploaded Qwen3.5-122B-A10B-heretic GGUF to Hugging Face, removing refusals for unrestricted use. Shared on Reddit as noteworthy for local runners. Not the uploader's creation.
Unsloth released Qwen3.5-9B-GGUF on Hugging Face, a 9B causal LM with vision encoder. It features advanced architecture like Gated DeltaNet and supports up to 1M token context. Trained with multi-step MTP for pre- and post-training.
A developer demonstrated LFM2.5-2.6B running at approximately 17 tokens per second on a OnePlus 13 using only the phone’s CPU. The setup uses a Q4_K_M GGUF model, a custom 450 KB inference engine, and an ADB-based device probe suite, with a target of around 30 tokens per second.
A developer shares insights from training a 216M parameter SLM from scratch, highlighting the critical importance of tokenizer quality over architecture tweaks. The project details the challenges of GGUF export and the impact of dataset composition on conversational performance.
A user shared a new AutoRound GGUF quantization of the Qwen 3.6 27B model, highlighting its efficiency in coding tasks. The post invites community feedback on the model's performance compared to other quantization methods.
Detailed guide for running Qwen3.5/3.6 models with NextN MTP speculative decoding in llama.cpp on a single RTX 3090 Ti, achieving ~150 tok/s on MoE models. Requires cherry-picking open PRs #22400 and #22673; includes build instructions and recommended GGUF quants.
Savant Commander 48B is a custom Qwen3-based 4x12B MOE merging distills from Claude, Gemini, OpenAI, DeepSeek, and more with hand-coded routing. Users control activation via prompts and test differences easily. GGUF versions include regular and Heretic uncensored, available on Hugging Face.
OmniCoder-9B is hailed as the smartest coding and tool-calling model for 8GB GPUs. It generated a full toolkit from a simple request. Available on Hugging Face in GGUF format for use with llama-server and VSCode.