Layer Duplication Tops Open LLM Leaderboard

🔑 Enhanced Key Takeaways

• •The RYS (layer duplication) technique is orthogonal to fine-tuning and weight optimization, enabling stacking with other training methods like ORPO and instruction tuning to achieve even higher leaderboard scores[1].
• •Layer duplication works only for ~7-layer 'circuit blocks' discovered through orthogonal probing; single-layer duplication has no effect, suggesting pretraining carves discrete functional circuits that must be preserved whole[1][3].
• •As of early 2026, the top four Open LLM Leaderboard models are all descendants of the original RYS technique, with MaziyarPanahi's calme-3.2-instruct-78b achieving 52.08 points[1].

🛠️ Technical Deep Dive

• •Layer duplication mechanism: Duplicates ~7-layer functional blocks within the model architecture without modifying weights, providing additional iterations through the model's internal reasoning space[1].
• •Orthogonal probing approach: Two narrow, orthogonal probes were used to identify optimal layer configurations that generalized across diverse benchmarks[1].
• •Hardware efficiency: Developed on 2x RTX 4090 GPUs, demonstrating that significant architectural improvements can be achieved without enterprise-scale compute[1].
• •Stacking compatibility: The method combines with fine-tuning (calme-2.4-rys-78b), ORPO training (CalmeRys-78B-Orpo-v0.1), and instruction tuning (calme-3.1, calme-3.2) for cumulative performance gains[1].
• •Circuit discovery: Empirical testing revealed that too few layers (single-layer) produce no effect, while too many layers degrade performance, indicating pretraining creates discrete functional units of approximately 7 layers[3].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline