Gemma 4 26b Schizophrenic in Coding Test

🔑 Enhanced Key Takeaways

• •Community consensus on r/LocalLLaMA suggests the 'schizophrenic' behavior in Gemma 4 26b is likely linked to a regression in the model's instruction-following fine-tuning (IFT) layer rather than a fundamental architectural flaw.
• •Users have identified that the model frequently hallucinates non-existent libraries or switches between programming languages mid-response when tasked with complex multi-file or single-page application generation.
• •Early benchmarking by the community indicates that while Gemma 4 26b excels in creative writing tasks, its performance on coding benchmarks like HumanEval has dropped significantly compared to the previous Gemma 3 iteration.

🛠️ Technical Deep Dive

• •Architecture: Utilizes a modified Transformer decoder-only architecture with Multi-Query Attention (MQA) for improved inference speed.
• •Parameter Count: 26 billion parameters, optimized for consumer-grade hardware with 24GB VRAM using 4-bit quantization.
• •Training Data: Trained on a mixture of synthetic data and filtered web-crawl data, with a specific focus on multilingual capabilities.
• •Issue Root Cause: Preliminary analysis suggests a 'mode collapse' during the final stage of RLHF (Reinforcement Learning from Human Feedback), causing the model to lose coherence in structured output tasks.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline