Tag: #low-vram11 results
User achieved DFlash speculative decoding on Qwen3.5-35B-A3B using llama.cpp on RTX 2080 SUPER 8GB VRAM via MoE CPU offload. Speed improved from 26.8 to 35.6 tok/s, a 33% gain. Optimal draft-max was 6 with 99% acceptance.
A new PR in vLLM introduces dynamic expert caching with LRU policy, enabling 16G MoE models on 8G VRAM by keeping active experts in VRAM and offloading others to RAM. On cache misses, computation shifts to CPU while reshuffling experts to minimize latency. Upcoming features include mxfp4 quantization and disk streaming.
Rose is a new stateless PyTorch optimizer with ultra-low VRAM (less than 8-bit AdamW), fast convergence, and strong generalization, licensed Apache 2.0. Benchmarks on MNIST show competitive accuracy despite sometimes higher training loss but lower validation loss. Easy to use for ML training with minimal memory overhead.
Qwen3.5-35B-A3B-UD-Q4_K_XL on RTX 4060 8GB gaming laptop with llama.cpp achieves 26 t/s generation at 100K context and 331 t/s prompt processing. Performance scales well from 5K to 100K contexts. User eyes RX 7900 XTX upgrade for larger models.
User praises OmniCoder-9B Q4_K_M GGUF running at 40+ tps on 8GB VRAM in Opencode, completing tasks flawlessly at 100K context. Outperforms expectations vs quota-limited cloud tools. Config shared for llama-server setup.
User fine-tuned Gemma 3 4B using DPO on a psychotherapy dataset with QLoRa and PeFT on an RTX 3050Ti laptop. Seeks local evaluation benchmarks to compare against base model. Emphasizes it's experimental, not for medical advice.
User reports running Qwen3 Coder Next in MXFP4 on RTX 3060 12GB (8GB VRAM effective) with 131k context at sustained 23 tokens/second. Configuration shared for web dev tasks, replacing paid Claude. Requires 64GB RAM; ideal for SaaS coding delegation.
User reports GLM-4.7 (pro plan) has become unreliable, possibly due to quantization. Seeks local model recommendations for 4GB VRAM and 24GB RAM hardware mimicking GLM-4.7 performance.
Reddit user seeks Ollama-compatible models that match or beat Claude Opus on ultra-low 32MB VRAM hardware like GeForce 256 and Pentium 3. Goal is vibe coding for a custom AI wrapper. Community discussion likely humorous given hardware limits.
User praises Qwen 27B as top LLM for low VRAM/1 GPU setups, running well on 24-48GB. Seeks alternatives but finds none comparable currently. Highlights its strong performance in local inference.