Layer Duplication Tops Open LLM Leaderboard

🔑 Enhanced Key Takeaways

• •The layer duplication technique was first publicly shared by researcher dnhkng on Hacker News, revealing development on consumer-grade 2x RTX 4090 GPUs in a home setup.[4]
• •A variant model, Solshine/Qwen2.5-142B-Doubled72B-Math-Instruct, applies full-layer doubling by alternating layers from Qwen2.5-72B-Instruct and Qwen2.5-Math-72B, including MLP adjustments.[1]
• •Qwen2-72B base architecture features 80 transformer layers, hidden dimension of 12,288, 64 query heads with 8 KV heads using GQA, SwiGLU nonlinearity, RMSNorm, and RoPE embeddings.[2][3]

🛠️ Technical Deep Dive

• •Qwen2.5-72B employs a decoder-only Transformer with 80 layers, hidden size 12,288, Grouped Query Attention (64 query / 8 key-value heads, per-head dim 192), MLP inner dim 49,152.[2][3]
• •Training used phased sequence lengths: 4,096 tokens for 80% of steps, then 32,768 tokens for 20%; warmup 5% linear, cosine decay to zero LR, batch 1M tokens (~80 seqs of 32k).[2]
• •Model includes pre-LayerNorm RMSNorm for stability, QKV bias for length extrapolation, RoPE with tunable base (10k to 1M in later stages), total ~72B params (70B non-embedding).[2][3]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline