🤖Reddit r/MachineLearning•較早收集於 3h
MiniMax 發布支援 1M Context 的稀疏注意力架構
#attention-mechanism#long-context#model-optimizationminimax-sparse-attention-(msa)minimaxflash-sparse-attention
💡全新的稀疏注意力架構實現了 15 倍的解碼加速，並為長程代理提供了 1M 的上下文視窗。
⚡ 30-Second TL;DR
有什麼變化
原生支援 1M token 上下文視窗擴展
為什麼重要
此架構顯著降低了長上下文任務的運算成本，使持續性的代理執行在規模化應用中更具可行性。
下一步行動
針對您的長上下文代理工作流程，對比測試新的 MiniMax MSA 架構，以評估潛在的效能提升。
誰應關注：Researchers & Academics
關鍵要點
- •原生支援 1M token 上下文視窗擴展
- •執行速度比 Flash-Sparse-Attention 快 4 倍
- •預填充速度提升 9 倍，解碼速度提升 15 倍
- •首個結合頂尖程式編寫、1M 上下文與多模態的開源權重模型
🧠 深度解析
Web-grounded analysis with 14 cited sources.
🔑 增強重點摘要
- •MiniMax Sparse Attention (MSA) is integrated into the MiniMax M3 model, which is positioned as the first open-weight model to combine frontier coding capabilities, a 1M token context window, and native multimodality (supporting image and video input, and desktop operation).
- •The MSA architecture achieves a per-token compute cost that is 1/20th of MiniMax's previous-generation models at a 1M context depth, making ultra-long-context AI agent deployment economically viable.
- •MSA's efficiency stems from a 'KV outer gather Q' approach at the operator level, which processes Key-Value (KV) blocks sequentially, ensuring contiguous memory access and reading each block only once, unlike traditional methods that fetch blocks multiple times.
- •MiniMax M3, powered by MSA, demonstrates top-tier performance on coding and agentic benchmarks, surpassing models like GPT-5.5 and Gemini 3.1 Pro on SWE-Bench Pro and OmniDocBench, and approaching Opus 4.7.
- •The model's native multimodality means its entire data pipeline was rebuilt for multimodal training from step zero, scaling pre-training data to over 100 trillion, achieving deep alignment between textual and visual semantic spaces.
📊 競品分析▸ Show
| Feature/Benchmark | MiniMax M3 (with MSA) | GPT-5.5 | Gemini 3.1 Pro | Opus 4.7 | DeepSeek's Multi-head Latent Attention (MLA) / DSA |
|---|---|---|---|---|---|
| Context Window | 1M tokens (native) | Not specified (implied shorter than 1M for direct comparison) | Not specified (implied shorter than 1M for direct comparison) | Not specified (implied shorter than 1M for direct comparison) | Up to 64k context length for NSA (DeepSeek's Native Sparse Attention) |
| Execution Speed (vs. Flash-Sparse-Attention) | 4x faster | N/A | N/A | N/A | N/A |
| Prefilling Speedup | 9x (over previous generation) | N/A | N/A | N/A | 11x performance increase during decoding for NSA at 64k context |
| Decoding Speedup | 15x (over previous generation) | N/A | N/A | N/A | 11x performance increase during decoding for NSA at 64k context |
| Per-token Compute (at 1M context) | 1/20th of previous generation | N/A | N/A | N/A | N/A |
| Coding Capability (SWE-Bench Pro) | Surpasses GPT-5.5 and Gemini 3.1 Pro, approaches Opus 4.7 | Surpassed by MiniMax M3 | Surpassed by MiniMax M3 | Approached by MiniMax M3 | N/A (DeepSeek V3.2 DSA mentioned as having heavy kernel engineering) |
| Multimodal Benchmarks (OmniDocBench) | Scores above Gemini 3.1 Pro | N/A | Scored below MiniMax M3 | N/A | N/A |
| Browsing/Information Retrieval (BrowseComp) | Scores 83.5, surpassing Opus 4.7 (79.3) | N/A | N/A | Scores 79.3 (surpassed by MiniMax M3) | N/A |
| Attention Mechanism | MiniMax Sparse Attention (MSA): block-level selection on real, uncompressed Key-Values, two-stage process (index branch + sparse branch) | Full attention (standard quadratic complexity) | Full attention (standard quadratic complexity) | Full attention (standard quadratic complexity) | Multi-head Latent Attention (MLA) / Native Sparse Attention (NSA): compresses keys and values into a low-dimensional latent space, token-level top-k, sliding window, compression branch |
| Open-weight Status | First open-weight model with frontier coding, 1M context, and multimodality | Closed-source | Closed-source | Closed-source | DeepSeek V3.2 DSA is an open-source sparse attention design |
| Pricing | Token Plan price remains unchanged for M3's enhanced capabilities | N/A | N/A | N/A | N/A |
🛠️ 技術深入
- Sparse Attention Mechanism (MSA): MSA is a clean and extensible new sparse attention architecture that avoids the quadratic computational complexity of full attention by calculating attention scores only for selected segments.
- KV Cache Partitioning: The stored context, known as the Key-Value (KV) cache, is split into blocks. A preliminary filtering step determines which blocks are relevant to the current query.
- Memory Access Optimization: MSA adopts a "KV outer gather Q" approach. This means KV blocks are used as the outer loop to aggregate the queries that hit them. Each block is read only once, and memory access is contiguous, significantly improving arithmetic intensity.
- Two-Stage Process: MSA functions as a two-stage mechanism. A lightweight index branch first scans incoming tokens and selects relevant blocks of the KV cache. The main attention layer then performs the expensive attention computation only on these selected blocks.
- Uncompressed Key-Values: Unlike some other sparse attention methods (e.g., DeepSeek's MLA), MSA operates on real, uncompressed Key-Values, which MiniMax states helps avoid precision loss at long contexts.
- Hardware-Friendly Design: The architecture is designed to be hardware-friendly, allowing its theoretical gains to be fully realized in practice, leading to significant speedups.
- Integration with M3: MSA enables the MiniMax M3 model to support a 1M token context window with a guaranteed minimum of 512K tokens, crucial for long-range agent tasks, coding, and video understanding.
🔮 前景展望AI analysis grounded in cited sources
AI agents will become significantly more capable and autonomous.
The combination of 1M context, frontier coding, and native multimodality in an open-weight model provides a robust foundation for developing highly sophisticated AI agents that can handle complex, long-horizon tasks across various data types.
The economic viability of deploying ultra-long-context AI models will increase substantially.
MSA's drastic reduction in per-token compute cost (1/20th of previous models) and significant speedups in prefilling and decoding make it much more cost-effective to run AI models with million-token context windows in production environments.
Open-weight models will accelerate innovation in multimodal and long-context AI.
As the first open-weight model to combine these frontier capabilities, MiniMax M3 will likely foster broader research and development in areas previously dominated by closed-source models, leading to faster advancements and new applications.
⏳ 時間線
2021-12
MiniMax founded by former SenseTime researchers.
2022-10
MiniMax launched its first AI companion app, Glow.
2023-06
Glow relaunched as Talkie for international markets.
2024-03
Alibaba Group led a $600 million financing round, valuing MiniMax at $2.5 billion.
2024-03
MiniMax launched Hailuo AI, an AI text and music-generating LLM platform.
2026-01-09
MiniMax held its initial public offering on the Hong Kong Stock Exchange.
2026-06-01
MiniMax M3 model, featuring MiniMax Sparse Attention (MSA), officially released.
📎 來源 (14)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： Reddit r/MachineLearning ↗