LLM Layer Duplication Hits 50% Depth Danger Zone
๐กMap your LLM's circuits: duplicate layers to boost weak models, avoid 50% death zone.
โก 30-Second TL;DR
What Changed
Universal danger zone at 50-56% depth kills reasoning when layers duplicated across architectures
Why It Matters
Reveals non-modular nature of LLMs, guiding layer editing for capability boosts in weak models. Highlights limits of 'think harder' via repetition, informing distillation and surgery techniques.
What To Do Next
Test layer duplication at 75% depth on your 7-9B coding model using MLX on Apple Silicon.
Key Points
- โขUniversal danger zone at 50-56% depth kills reasoning when layers duplicated across architectures
- โขDuplicating late layers (75-84%) boosts Hybrid 9B coding score from 4/10 to 7/10
- โขDouble/triple-stacking good circuits interferes; deletion causes total failure
- โขStrong models like 32B unchanged; technique helps weaker ones without retraining
๐ง Deep Insight
Background and context from public sources โ not the original article. 8 sources cited.
๐ Enhanced Key Takeaways
- โขRYS-XLarge, a 78B model created by duplicating 7 middle layers (45-51) in a 72B base via pointers without extra memory or training, topped the HuggingFace Open LLM Leaderboard and spawned the top four models as of early 2026.[1][5][4]
- โขHeatmap 'brain scans' identify optimal duplication ranges by evaluating all (i,j) pairs on probes like math and EQ, revealing model-specific circuit blocks of ~7 layers that must be preserved intact for gains.[5][4]
- โขSubsequent leaderboard toppers like calme-2.4-rys-78b and CalmeRys-78B-Orpo-v0.1 combined layer duplication with fine-tuning and ORPO training for even higher scores.[5]
๐ ๏ธ Technical Deep Dive
- โขDuplication implemented via layer pointers, repeating execution without increasing GPU memory usage or parameters beyond minor effective count rise (72B to 78B).[5][4]
- โขOptimal for RYS-XLarge: (i=45, j=52) in 80-layer stack, running layers 0-51 once then 45-79, effectively duplicating layers 45-51 seven times total.[5]
- โขEvaluation via pixel heatmaps: x-axis end point j, y-axis start i; each pixel scores full model on math/EQ probes, showing single-layer duplication ineffective while ~7-layer blocks excel.[5]
- โขWorks across Qwen, MiniMax, GLM architectures but requires per-model heatmap analysis for optimal (i,j) ranges.[1][4]
๐ฎ Future ImplicationsAI analysis grounded in cited sources
โณ Timeline
๐ Sources (8)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
- topaiproduct.com โ Rys Xlarge LLM Neuroanatomy How Copying 7 Layers with Zero Training Topped the Huggingface Leaderboard
- research.google โ Making Llms More Accurate by Using All of Their Layers
- blog.promptlayer.com โ Prompt Repetition Improves LLM Accuracy
- news.ycombinator.com โ Item
- dnhkng.github.io โ Rys
- lakera.ai โ Guide to Hallucinations in Large Language Models
- dev.to โ The Silent Evolution of Llms in 2026 2mc4
- youtube.com โ Watch
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.