Prisma: Garage Model with Novel FFN Gate
💡Garage model beats transformers 25% on data efficiency—novel FFN gate worth testing
⚡ 30-Second TL;DR
What Changed
Attention and output weight sharing reduces parameters
Why It Matters
Offers compute-efficient alternative for garage projects, potentially inspiring parameter-efficient designs amid rising training costs. Community feedback could refine it into a viable open-source contender.
What To Do Next
Download Prisma from Hugging Face (y3i12/Prisma) and benchmark against baselines on ARC/PIQA.
Key Points
- •Attention and output weight sharing reduces parameters
- •Nested FFN gate: W2 @ (W1 @ x * silu(W3 @ x * silu(W4 @ x)))
- •Introduces Word-Relative Rotary Position Embedding
- •25% more data-efficient than standard transformers
- •Trained on 30B tokens (OpenWebText + FineWeb-Edu)
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.