๐ผVentureBeatโขStalecollected in 9m
Researchers train foundation model from scratch for $1,500
๐กA $1,500 foundation model could disrupt the 'bigger is better' scaling trend in enterprise AI.
โก 30-Second TL;DR
What Changed
Replaces standard Transformers with Hierarchical Recurrent Models (HRM) for higher sample efficiency.
Why It Matters
This research challenges the 'brute-force' scaling dogma, potentially democratizing foundational model development for enterprises with limited budgets. It shifts the focus from massive data memorization to efficient, task-oriented reasoning architectures.
What To Do Next
Evaluate if your enterprise use case can benefit from instruction-tuned recurrent architectures instead of standard Transformer-based LLMs to reduce training costs.
Who should care:Researchers & Academics
Key Points
- โขReplaces standard Transformers with Hierarchical Recurrent Models (HRM) for higher sample efficiency.
- โขDecouples computation into slow-evolving strategic and fast-evolving execution layers.
- โขTrains exclusively on instruction-response pairs, bypassing the need for massive internet-scale raw text scraping.
- โขAchieved performance competitive with larger open models at a total training cost of approximately $1,500.
๐ง Deep Insight
Web-grounded analysis with 13 cited sources.
๐ Enhanced Key Takeaways
- โขHRM-Text is a 1.15-billion-parameter model, trained on approximately 40 billion tokens of structured data, which is significantly less (up to 1,000x) than the trillions of tokens typically used by other large language models.
- โขThe model was pretrained in about one day using 16 GPUs, costing approximately $1,000 to $1,500, and is openly released on GitHub and Hugging Face.
- โขHRM-Text operates with a compact 0.6 GiB footprint at int4 quantization, making it suitable for on-device and offline deployment on smartphones and edge devices.
- โขThe model's benchmark performance (e.g., 56.2% on MATH, 82.2% on DROP, 81.9% on ARC-Challenge, and 60.7% on MMLU) reflects its base architecture without post-training or fine-tuning, suggesting a higher potential ceiling after alignment.
- โขDespite its reasoning strengths, HRM-Text currently exhibits weakness in coding tasks due to its training data mix, though early fine-tuning indicates potential for improvement.
๐ Competitor Analysisโธ Show
| Feature/Metric | HRM-Text (Sapient) | Llama 3.2 3B (Meta) | Qwen 3.5 2B (Alibaba) | GPT-3.5 (OpenAI) |
|---|---|---|---|---|
| Architecture | Hierarchical Recurrent Model (HRM) | Transformer-based | Transformer-based | Transformer-based |
| Parameters | 1.15 Billion | 3 Billion | 2 Billion | Billions (proprietary) |
| Training Cost | ~$1,000 - $1,500 | Hundreds of millions (estimated) | Hundreds of millions (estimated) | Hundreds of millions (estimated) |
| Training Tokens | ~40 Billion (structured data) | ~9 Trillion | ~36 Trillion | Trillions (estimated) |
| Training Time | ~1 day (on 16 GPUs) | Weeks/Months (estimated) | Weeks/Months (estimated) | Weeks/Months (estimated) |
| MATH Benchmark | 56.2% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| DROP Benchmark | 82.2% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| ARC-Challenge | 81.9% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| MMLU Benchmark | 60.7% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| Deployment Footprint | 0.6 GiB (int4 quantization) | Larger (estimated) | Larger (estimated) | Cloud-dependent (estimated) |
| Open Source | Yes (GitHub, Hugging Face) | Yes (Llama series) | Yes (Qwen series) | No (proprietary) |
| Key Differentiator | Sample-efficient, latent-space reasoning, low cost | Large-scale, general-purpose LLM | Large-scale, general-purpose LLM | State-of-the-art general-purpose LLM |
๐ ๏ธ Technical Deep Dive
- HRM-Text employs a hierarchical latent recurrent architecture consisting of two interdependent modules: a High-level (H) module for slow, abstract planning, and a Low-level (L) module for fast, detailed computations.
- In a single forward pass, the model performs 2 H updates and 6 L updates, totaling 8 stack iterations, with the L-stack updating more frequently to create a multi-timescale reasoning structure.
- Cross-level information flows at every recurrent step, enabling the model to perform internal reasoning in its continuous latent state before producing outputs, reducing reliance on explicit intermediate reasoning tokens.
- The training objective uses a task-completion approach, where loss is computed exclusively on the answer portion of instruction-response pairs, rather than on every token in a raw text sequence.
- It utilizes PrefixLM masking, where prompt tokens attend bidirectionally, and response tokens attend causally, to match training-time forward at inference.
- To ensure stable training of its deep recurrent network, HRM-Text incorporates specific designs: MagicNorm for stability in forward and backward propagation, and warmup deep credit assignment.
- The model has approximately 1.15 billion parameters, a hidden size of 1536, 16 layers per H/L stack, 12 attention heads (Multi-Head Attention with head_dim 128), and a maximum sequence length of 4096.
- It uses a vocabulary of 65,536, Scaled (lecun_normal) embedding, RoPE (Rotary Positional Embedding) with theta 10000, SwiGLU activations, and Parameterless Pre-Normalization.
- Training was conducted using bfloat16 precision and an AdamATan2 optimizer.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Democratization of foundation model pretraining will accelerate.
The significantly reduced cost and data requirements of HRM-Text make it feasible for smaller organizations and researchers to train powerful reasoning models from scratch, lowering entry barriers to foundational AI research and development.
Development of specialized AI applications will become more efficient.
The ability to affordably pretrain models exclusively on instruction-response pairs allows for faster creation of domain-specific AI without the need for massive, general internet-scale raw text datasets.
On-device and edge AI for complex reasoning will see increased adoption.
HRM-Text's compact footprint of 0.6 GiB at int4 quantization enables advanced reasoning capabilities to be deployed on resource-constrained devices like smartphones and edge hardware, reducing cloud dependency.
โณ Timeline
2024
Sapient Intelligence founded by Guan Wang and William Chen.
2025-01
Sapient Intelligence raised a $22 million seed round.
2025-06
Sapient proposed the initial Hierarchical Reasoning Model (HRM) architecture (27 million parameters) in an arXiv paper.
2025-07
Sapient open-sourced the first-generation model, HRM-Symbolic, for symbolic reasoning tasks.
2026-04
Independent verification of HRM-Text's benchmark performance was conducted.
2026-05-18
Sapient Intelligence launched HRM-Text, making it fully open-source on GitHub and Hugging Face.
๐ Sources (13)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: VentureBeat โ