Tiny Transformers Perfectly Add 10-Digit Numbers

#tiny-models#arithmetic#efficiencytiny-transformerstransformers

💡Ultra-tiny transformers nail 10-digit math—efficiency game-changer for edge AI!

⚡ 30-Second TL;DR

What Changed

<100 parameters for full model

Why It Matters

Demonstrates transformers can be ultra-efficient for narrow tasks, inspiring edge AI deployments.

What To Do Next

Replicate the tiny transformer from the Reddit link to test arithmetic efficiency.

Who should care:Researchers & Academics

Background and context from public sources — not the original article. 5 sources cited.

•Dimitris Papailiopoulos prompted AI agents like Claude Code to discover transformers, achieving 6,080 parameters for 10-digit addition before human optimizations.[1]
•A 777-parameter transformer demonstrates grokking, suddenly generalizing to unseen 10-digit additions after training on dynamically generated examples, ruling out memorization.[2]
•A 456-parameter transformer solves the task, further reducing size while maintaining generalization on large held-out test sets.[2]

📊 Competitor Analysis▸ Show

Model	Parameters	Accuracy	Notes
Claude Code (D. Papailiopoulos)	6,080	High	AI-discovered via prompting
Grokking Transformer	~777	100% on test	Generalizes post-grokking
yinglunz 456-param	456	Solves 10-digit	JAX implementation
Ziming Liu ConvNet	181	Learns perfectly	Transformer-like, conv+MLP

•Ziming Liu's 181-parameter model: 2 blocks of kernel size 3 convolution (hidden channels 2) followed by MLP; weights show symmetry per digit position and hierarchical scaling (1:10:100 ratios).[1]
•Grokking model (~777 params): Trained on-the-fly generated examples with ~100k test cases; compresses 10^20 possibilities into algorithmic carry propagation, impossible via memorization (3.4e21 bits needed vs 2.5e4 in model).[2]
•456-parameter transformer: Detailed in report.pdf on GitHub, achieves solution via optimized architecture search.[2]

Sub-100 parameter transformers will automate toy algorithmic tasks by 2027

Progressive reductions from 6k to 181 parameters show architecture search rapidly minimizes sizes for narrow tasks like addition.

Grokking in tiny models enables reliable generalization without large datasets

777-param model generalizes across unseen combinations via carry algorithm discovery, bypassing memorization limits.

2026-02

Dimitris Papailiopoulos tweets AI agents discovering 6,080-param transformers for 10-digit addition

2026-02

777-parameter grokking transformer paper released, showing generalization jump

2026-02

456-parameter transformer report published on GitHub

2026-02-24

Ziming Liu releases 181-parameter transformer-like convnet blogpost

Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.

Weekly AI Recap

Read this week's curated digest of top AI events →

Same topic

Explore #tiny-models

Same product