Kimi's Attention Residuals Fix LLM Depth Issues

🔑 Enhanced Key Takeaways

• •AttnRes employs a single learned pseudo-query vector per layer to compute softmax attention weights over prior layer outputs, enabling selective retrieval akin to RNN recurrence fixes.
• •Block AttnRes typically uses around eight blocks for optimal performance, where intra-block residuals use cheap addition and inter-block attention operates on summaries.
• •On downstream benchmarks, AttnRes boosts include +7.5 points on GPQA-Diamond, +3.6 on Minerva Math, and significant gains on HumanEval code generation.
• •Optimized implementations feature cross-stage caching and a two-phase computation strategy, adding minimal overhead to training or inference speed.

🛠️ Technical Deep Dive

• •Each layer uses a learned pseudo-query vector to compute softmax attention over key-value pairs derived from previous layer outputs, replacing fixed residual addition with input-dependent aggregation.
• •Block AttnRes partitions layers into groups (e.g., 8 blocks), summing intra-block outputs into summaries for efficient cross-block attention, reducing memory and communication costs.
• •Implementation optimizations include cross-stage caching of block summaries and a two-phase computation (forward pass with attention, backward with gradients), ensuring near-baseline speed.
• •In the 48B Mixture-of-Experts Kimi Linear model, AttnRes maintains bounded output magnitudes and uniform gradient norms, eliminating PreNorm dilution observed in baselines.

🔮 Future ImplicationsAI analysis grounded in cited sources