Musk Praises Kimi's Attention Residuals

Architecture & Mechanism:

• Replaces fixed residual connections (h = h_l + F(h_l)) with depth-wise attention: V = [b₀; b₁; …; bₙ] (stacked blocks), K = RMSNorm(V) (normalized keys), α = softmax(K · w_l) (learned attention weights), h = Σ αᵢ · Vᵢ (weighted combination)[4]
• Two AttnRes operations per layer: one before self-attention, one before MLP, each with independent learned pseudo-query weights (w_l)[4]
• Pseudo-query vectors initialized to zero, ensuring uniform 1/N attention weights at training start and avoiding early instability[1]
• RMSNorm applied to keys to prevent blocks with large accumulated magnitudes from dominating attention logits[3][4]

Scaling Properties:

• Block AttnRes matches baseline validation loss at 1.25x compute efficiency[3][4]
• Mitigates gradient flow issues by distributing gradient norms uniformly across layers[1][2]
• Keeps output magnitudes bounded across network depth, addressing hidden-state growth problems in standard residuals[2]

Implementation Details:

• Softmax attention operates over depth dimension (dim=0), not sequence dimension—this is attention over layers, not tokens[4]
• Rust implementation available on burn framework, supporting CPU, CUDA, Metal, and wgpu backends[4]
• Inference overhead less than 2% with optimized system designs including cross-stage caching and two-phase computation strategy[3]