SSMs Struggle in 25M Param Training

🔑 Enhanced Key Takeaways

• •The 'Parameter Golf' experiments highlight a fundamental trade-off where SSMs' linear scaling properties in inference do not translate to parameter efficiency in ultra-low-capacity regimes (sub-50M parameters).
• •The identified Mamba-3 Triton kernel slowdown is attributed to register pressure during the scan operation, which prevents optimal occupancy when fused with standard activation functions.
• •The observed quantization degradation is linked to the higher sensitivity of SSM state-space matrices to low-precision rounding compared to the attention-based projections in Transformers.

🛠️ Technical Deep Dive

• •Mamba-3 architecture utilizes a modified selective scan mechanism that attempts to balance hardware utilization with sequence length flexibility.
• •The 16% backward pass slowdown is specifically tied to the implementation of the associative scan operator in Triton, which struggles with shared memory (SMEM) bank conflicts when handling the state-space transition matrices.
• •The torch.compile quantizer bug stems from incorrect handling of non-linear state updates during the graph capture phase, leading to precision loss in the hidden state accumulation.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline