⚛️量子位•Stalecollected in 51m
High-Performance 1B HRM Model Trained for Only $1,500
💡Discover how a $1,500 1B model is disrupting reward modeling, backed by top AI researchers.
⚡ 30-Second TL;DR
What Changed
Model features a highly efficient 1B parameter architecture.
Why It Matters
This highlights a shift toward cost-effective, small-scale model training for specialized tasks like reward modeling. It challenges the assumption that state-of-the-art performance requires massive compute budgets.
What To Do Next
Evaluate your current RLHF pipeline to see if a smaller, specialized reward model can replace larger, more expensive alternatives.
Who should care:Researchers & Academics
Key Points
- •Model features a highly efficient 1B parameter architecture.
- •Total training cost was remarkably low at only $1,500.
- •Received strong endorsements from industry leaders including HuggingFace CEO and Yoshua Bengio's team.
🧠 Deep Insight
Web-grounded analysis with 8 cited sources.
🔑 Enhanced Key Takeaways
- •The model, specifically named HRM-Text-1B, is a 1.15-billion-parameter language model developed by the Singapore-based AI startup Sapient Intelligence, founded in 2024.
- •HRM-Text-1B was trained on 16 GPUs over 1.9 days, utilizing approximately 40 billion structured instruction-response tokens, which is significantly less data (100 to 900 times fewer training tokens) compared to traditional Transformer models that often require trillions of tokens.
- •The model employs a Hierarchical Recurrent Model (HRM) architecture, which decouples computation into slow-evolving strategic and fast-evolving execution layers, enabling effectively unbounded compute depth with a bounded parameter count.
- •HRM-Text-1B demonstrates competitive performance against open models in the 2B-to-7B parameter range on several benchmarks, scoring 56.2 on MATH, 82.2 on DROP, 84.5% on GSM8K, and 81.9% on ARC-C. However, it exhibits weaker performance on MMLU (60.7%), suggesting a factual knowledge gap due to its specialized training data.
- •The model is fully open-sourced and available on platforms like GitHub and Hugging Face, making it accessible for inspection, modification, and deployment.
🛠️ Technical Deep Dive
- Architecture: Hierarchical Recurrent Model (HRM) architecture.
- Core Components: Features a dual-timescale recurrent design with two Transformer modules: a high-level (H) slow-updating module for abstract planning and a low-level (L) fast-updating module for detailed computations.
- Computational Depth: The H and L modules iterate over the same input embeddings for H_cycles × (L_cycles + 1) steps, incorporating additive state injection (z_L + z_H), which provides effectively unbounded computational depth at a fixed parameter count.
- Training Data: Pre-trained from scratch on approximately 40 billion structured instruction-response tokens, a stark contrast to the trillions of raw tokens typically used for conventional Transformer models.
- Training Objective: Utilizes a PrefixLM mask during pre-training, where prompt tokens attend bidirectionally to each other, and response tokens attend causally. The mask is controlled by
token_type_ids. - Hardware & Duration: Trained on a cluster of 16 GPUs over a period of 1.9 days.
- Limitations: The model is predominantly English-only due to its training corpus and shows expectedly weak performance on coding tasks as it was not trained on code datasets, though it demonstrates promising adaptation potential with additional code data.
🔮 Future ImplicationsAI analysis grounded in cited sources
Foundational LLM training will become more democratized.
The remarkably low training cost of HRM-Text-1B ($1,500) significantly lowers the barrier to entry for foundational model pre-training, making it accessible to a broader range of organizations and research labs beyond highly resourced institutions.
The AI industry will see a shift towards more sample-efficient training paradigms.
HRM-Text-1B's ability to achieve competitive performance with significantly fewer training tokens (40 billion versus trillions for traditional models) challenges the prevailing brute-force scaling approach, suggesting that specialized architectures and structured data can lead to useful general performance more efficiently.
There will be increased research and adoption of hierarchical and recurrent architectures in large language models.
The demonstrated success of the HRM architecture in achieving high performance with low computational cost and parameter count could inspire further development and integration of similar recurrent designs that offer unbounded effective depth in future LLMs.
⏳ Timeline
2024
Sapient Intelligence founded.
2025-01
Sapient Intelligence raised a $22 million seed round.
2025-06
The Hierarchical Reasoning Model (HRM) architecture debuted in a paper, demonstrating competitive performance with a 27 million parameter model.
2026-05-20
The `sapientinc/HRM-Text-1B` model checkpoint was released on Hugging Face.
2026-06-10
Sapient Intelligence publicly released HRM-Text, a 1.15-billion-parameter language model.
📎 Sources (8)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: 量子位 ↗