Primitive Layer in Small LLMs Evidenced
π‘Reproducible proof of primitive layers in small LLMs β probes internals easily
β‘ 30-Second TL;DR
What Changed
Consistent +0.245 activation gap in all 4 models: Qwen 2.5, Gemma 3, LLaMA 3.2, SmolLM2
Why It Matters
Reveals innate structure in small LLMs evolving with scale, potentially explaining emergent capabilities. Enables mechanistic interpretability probes without APIs.
What To Do Next
Reproduce the 18 experiments locally with Ollama from the GitHub repo.
Key Points
- β’Consistent +0.245 activation gap in all 4 models: Qwen 2.5, Gemma 3, LLaMA 3.2, SmolLM2
- β’11 pre-registered compositions predict Layer 1 concepts (e.g., WANT + GRIEF β longing)
- β’Gap largest in smallest models, narrows as scaffolding primitives strengthen at scale
- β’Fully reproducible locally via Ollama, code/data on GitHub
Weekly AI Recap
Read this week's curated digest of top AI events β
πRelated Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning β
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.