SHADOW-250M: A 60 MB Long-Context LLM
💡See how a 60 MB quantized LLM uses disk-based context to retrieve information millions of tokens deep.
⚡ 30-Second TL;DR
What Changed
The 250M-parameter model is quantized to under 2 bits, requiring about 60 MB for deployment and 80 MB of RAM.
Why It Matters
SHADOW-250M demonstrates a possible tradeoff between model size, CPU inference speed, and extremely long-term retrieval. Its reported results are promising for local and embedded applications, but the small model size and limited reasoning over archived context mean it should not be treated as a replacement for larger general-purpose LLMs.
What To Do Next
Clone the SHADOW-250M repository, run its CPU inference and archive-retrieval demos, then benchmark latency, RAM, and retrieval accuracy on your own long-context dataset.
Key Points
- •The 250M-parameter model is quantized to under 2 bits, requiring about 60 MB for deployment and 80 MB of RAM.
- •The latest 2,048 tokens remain in fp16, while older context is compressed to roughly 320 bytes per token and stored on disk.
- •It was trained on 30B FineWeb tokens and achieved 3.15 nats per token, or 23.3 perplexity, on held-out English web text.
- •The fixed 512-bit vocabulary codes use 8.4 MB for 131K tokens and achieved 0.619 Spearman correlation on WordSim-353.
- •The open-source repository includes inference demos, fine-tuning weights, and reproducible before-and-after examples.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.