SHADOW-250M Fits an LLM in 60MB
💡See how a 250M model reaches 400 tok/s in 60MB with disk-backed long-context retrieval.
⚡ 30-Second TL;DR
What Changed
The 250M-parameter model was trained from scratch on 30B FineWeb tokens.
Why It Matters
SHADOW-250M demonstrates a practical direction for extremely small, CPU-first language-model deployments and long-term external memory. Its modest parameter count and stated limitations mean it is better suited to narrow retrieval and embedded use cases than broad reasoning or open-domain assistance.
What To Do Next
Clone the SHADOW-250M GitHub repository and benchmark its CPU throughput, RAM usage, and archive retrieval accuracy on your own edge-device workload.
Key Points
- •The 250M-parameter model was trained from scratch on 30B FineWeb tokens.
- •Sub-2-bit quantization reduces deployment size to 60MB and runtime RAM usage to about 80MB.
- •The model runs at approximately 400 tokens per second on a normal laptop CPU without a GPU or heavyweight framework.
- •Older context is compressed to one bit and stored on disk, enabling retrieval from archives of up to 100M tokens.
- •The held-out evaluation reports 3.15 nats per token and 23.3 perplexity on English web text.
🧠 Deep Insight
Background and context from public sources — not the original article. 13 sources cited.
🔑 Enhanced Key Takeaways
- •The model utilizes a fixed 512-bit code for its 131k-token vocabulary, requiring only 8.4 MB of storage and zero trained parameters for the embedding layer.
- •SHADOW-250M employs a hybrid memory architecture that keeps the most recent 2,048 tokens in fp16 precision while offloading older context to disk as 1-bit compressed data.
- •The project is split into two distinct repositories: the runnable inference engine hosted at QLNI/SHADOW-250M-Instruct and the master weights for fine-tuning at NODEMIND/SHADOW-250M.
- •The model's design prioritizes retrieval-augmented generation over complex reasoning, specifically enabling the querying of archives containing up to 100 million tokens.
- •The development represents a shift toward 'extreme' local AI, focusing on architectural optimization and data representation efficiency rather than scaling parameter counts.
📊 Competitor Analysis▸ Show
| Feature | SHADOW-250M | TinyLlama-1.1B | Phi-3-mini (3.8B) |
|---|---|---|---|
| Size | 60MB | ~600MB (Q4) | ~1.8GB (Q4) |
| RAM Usage | ~80MB | ~800MB | ~2GB |
| Context | 100M tokens (Disk) | 2k-4k tokens | 128k tokens |
| Hardware | Laptop CPU | Laptop CPU | Modern CPU/GPU |
🛠️ Technical Deep Dive
- Architecture: Hybrid memory system using fp16 for active context and 1-bit disk-based compression for long-term storage.
- Vocabulary: Fixed 512-bit code mapping for 131,072 tokens, occupying 8.4 MB total.
- Quantization: Sub-2-bit per parameter implementation optimized for CPU-only execution.
- Inference: Optimized C++ runtime bypassing standard heavyweight frameworks like PyTorch or TensorFlow.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
📎 Sources (13)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.