Tiny Transformers Perfectly Add 10-Digit Numbers

🔑 Enhanced Key Takeaways

• •Dimitris Papailiopoulos prompted AI agents like Claude Code to discover transformers, achieving 6,080 parameters for 10-digit addition before human optimizations.[1]
• •A 777-parameter transformer demonstrates grokking, suddenly generalizing to unseen 10-digit additions after training on dynamically generated examples, ruling out memorization.[2]
• •A 456-parameter transformer solves the task, further reducing size while maintaining generalization on large held-out test sets.[2]

🛠️ Technical Deep Dive

• •Ziming Liu's 181-parameter model: 2 blocks of kernel size 3 convolution (hidden channels 2) followed by MLP; weights show symmetry per digit position and hierarchical scaling (1:10:100 ratios).[1]
• •Grokking model (~777 params): Trained on-the-fly generated examples with ~100k test cases; compresses 10^20 possibilities into algorithmic carry propagation, impossible via memorization (3.4e21 bits needed vs 2.5e4 in model).[2]
• •456-parameter transformer: Detailed in report.pdf on GitHub, achieves solution via optimized architecture search.[2]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline