Ternary Bonsai: 1.58-Bit LLMs Launched

🔑 Enhanced Key Takeaways

• •Ternary Bonsai utilizes a custom activation quantization scheme called 'Dynamic Range Scaling' (DRS) to mitigate the precision loss typically associated with ternary weight quantization.
• •The models are optimized for edge deployment via a custom kernel implementation that leverages bit-manipulation instructions on ARM NEON and Apple Silicon, bypassing standard matrix multiplication bottlenecks.
• •PrismML has open-sourced the training recipe, which utilizes a two-stage distillation process where a dense FP16 teacher model guides the ternary student through a straight-through estimator (STE) during backpropagation.

🛠️ Technical Deep Dive

• •Weights are stored in a packed 2-bit format (using 2 bits per parameter to represent {-1, 0, 1}), achieving the theoretical 1.58-bit limit.
• •The architecture employs a modified RMSNorm that is computed in FP16 to maintain numerical stability during the forward pass.
• •The inference engine uses a 'dequantization-on-the-fly' approach, where ternary weights are expanded to FP16 registers only at the moment of computation to minimize cache pressure.
• •Training utilizes a custom loss function that penalizes weight distribution drift away from the ternary constraints, ensuring the model remains within the {-1, 0, 1} manifold.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline