來源VentureBeat•較早收集於 9m
研究人員以 1,500 美元從零訓練出基礎模型
#llm-efficiency#cost-reduction#model-architecturehrm-textsapienthrm-text
💡僅需 1,500 美元的基礎模型，可能顛覆企業 AI 領域「越大越好」的擴展趨勢。
⚡ 30 秒速覽
有什麼變化
以分層遞迴模型 (HRM) 取代標準 Transformer，顯著提升樣本效率。
為什麼重要
這項研究挑戰了「暴力擴展」的教條，可能使預算有限的企業也能開發基礎模型。它將重點從海量數據記憶轉向高效、以任務為導向的推理架構。
下一步行動
評估您的企業應用場景是否能改用指令微調的遞迴架構，而非標準的 Transformer 大型語言模型，以降低訓練成本。
誰應關注：Researchers & Academics
關鍵要點
- •以分層遞迴模型 (HRM) 取代標準 Transformer，顯著提升樣本效率。
- •將計算解耦為慢速演進的策略層與快速演進的執行層。
- •僅使用指令-回應對進行訓練，無需大規模抓取網際網路原始文本。
- •以約 1,500 美元的總訓練成本，實現了與大型開源模型相當的效能。
🧠 深度解析
背景與延伸：來自公開資料，非原文內容。引用 13 個來源。
🔑 增強重點摘要
- •HRM-Text is a 1.15-billion-parameter model, trained on approximately 40 billion tokens of structured data, which is significantly less (up to 1,000x) than the trillions of tokens typically used by other large language models.
- •The model was pretrained in about one day using 16 GPUs, costing approximately $1,000 to $1,500, and is openly released on GitHub and Hugging Face.
- •HRM-Text operates with a compact 0.6 GiB footprint at int4 quantization, making it suitable for on-device and offline deployment on smartphones and edge devices.
- •The model's benchmark performance (e.g., 56.2% on MATH, 82.2% on DROP, 81.9% on ARC-Challenge, and 60.7% on MMLU) reflects its base architecture without post-training or fine-tuning, suggesting a higher potential ceiling after alignment.
- •Despite its reasoning strengths, HRM-Text currently exhibits weakness in coding tasks due to its training data mix, though early fine-tuning indicates potential for improvement.
📊 競品分析▸ Show
| Feature/Metric | HRM-Text (Sapient) | Llama 3.2 3B (Meta) | Qwen 3.5 2B (Alibaba) | GPT-3.5 (OpenAI) |
|---|---|---|---|---|
| Architecture | Hierarchical Recurrent Model (HRM) | Transformer-based | Transformer-based | Transformer-based |
| Parameters | 1.15 Billion | 3 Billion | 2 Billion | Billions (proprietary) |
| Training Cost | ~$1,000 - $1,500 | Hundreds of millions (estimated) | Hundreds of millions (estimated) | Hundreds of millions (estimated) |
| Training Tokens | ~40 Billion (structured data) | ~9 Trillion | ~36 Trillion | Trillions (estimated) |
| Training Time | ~1 day (on 16 GPUs) | Weeks/Months (estimated) | Weeks/Months (estimated) | Weeks/Months (estimated) |
| MATH Benchmark | 56.2% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| DROP Benchmark | 82.2% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| ARC-Challenge | 81.9% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| MMLU Benchmark | 60.7% | Competitive, but higher cost | Competitive, but higher cost | Competitive, but higher cost |
| Deployment Footprint | 0.6 GiB (int4 quantization) | Larger (estimated) | Larger (estimated) | Cloud-dependent (estimated) |
| Open Source | Yes (GitHub, Hugging Face) | Yes (Llama series) | Yes (Qwen series) | No (proprietary) |
| Key Differentiator | Sample-efficient, latent-space reasoning, low cost | Large-scale, general-purpose LLM | Large-scale, general-purpose LLM | State-of-the-art general-purpose LLM |
🛠️ 技術深入
- HRM-Text employs a hierarchical latent recurrent architecture consisting of two interdependent modules: a High-level (H) module for slow, abstract planning, and a Low-level (L) module for fast, detailed computations.
- In a single forward pass, the model performs 2 H updates and 6 L updates, totaling 8 stack iterations, with the L-stack updating more frequently to create a multi-timescale reasoning structure.
- Cross-level information flows at every recurrent step, enabling the model to perform internal reasoning in its continuous latent state before producing outputs, reducing reliance on explicit intermediate reasoning tokens.
- The training objective uses a task-completion approach, where loss is computed exclusively on the answer portion of instruction-response pairs, rather than on every token in a raw text sequence.
- It utilizes PrefixLM masking, where prompt tokens attend bidirectionally, and response tokens attend causally, to match training-time forward at inference.
- To ensure stable training of its deep recurrent network, HRM-Text incorporates specific designs: MagicNorm for stability in forward and backward propagation, and warmup deep credit assignment.
- The model has approximately 1.15 billion parameters, a hidden size of 1536, 16 layers per H/L stack, 12 attention heads (Multi-Head Attention with head_dim 128), and a maximum sequence length of 4096.
- It uses a vocabulary of 65,536, Scaled (lecun_normal) embedding, RoPE (Rotary Positional Embedding) with theta 10000, SwiGLU activations, and Parameterless Pre-Normalization.
- Training was conducted using bfloat16 precision and an AdamATan2 optimizer.
🔮 前景展望基於引用來源的 AI 分析
Democratization of foundation model pretraining will accelerate.
The significantly reduced cost and data requirements of HRM-Text make it feasible for smaller organizations and researchers to train powerful reasoning models from scratch, lowering entry barriers to foundational AI research and development.
Development of specialized AI applications will become more efficient.
The ability to affordably pretrain models exclusively on instruction-response pairs allows for faster creation of domain-specific AI without the need for massive, general internet-scale raw text datasets.
On-device and edge AI for complex reasoning will see increased adoption.
HRM-Text's compact footprint of 0.6 GiB at int4 quantization enables advanced reasoning capabilities to be deployed on resource-constrained devices like smartphones and edge hardware, reducing cloud dependency.
⏳ 時間線
2024
Sapient Intelligence founded by Guan Wang and William Chen.
2025-01
Sapient Intelligence raised a $22 million seed round.
2025-06
Sapient proposed the initial Hierarchical Reasoning Model (HRM) architecture (27 million parameters) in an arXiv paper.
2025-07
Sapient open-sourced the first-generation model, HRM-Symbolic, for symbolic reasoning tasks.
2026-04
Independent verification of HRM-Text's benchmark performance was conducted.
2026-05-18
Sapient Intelligence launched HRM-Text, making it fully open-source on GitHub and Hugging Face.
📎 來源 (13)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： VentureBeat ↗
每週電子報
每週一封，可隨時退訂。