๐คReddit r/MachineLearningโขStalecollected in 3h
MiniMax Releases 1M Context Sparse Attention Architecture
#attention-mechanism#long-context#model-optimizationminimax-sparse-attention-(msa)minimaxflash-sparse-attention
๐กNew sparse attention architecture achieves 15x decoding speedup and 1M context window for long-horizon agents.
โก 30-Second TL;DR
What Changed
Native 1M token context window scaling
Why It Matters
This architecture significantly lowers the compute cost for long-context tasks, making sustained agent execution more viable at scale.
What To Do Next
Benchmark your long-context agent workflows against the new MiniMax MSA architecture to evaluate potential performance gains.
Who should care:Researchers & Academics
Key Points
- โขNative 1M token context window scaling
- โข4x faster execution speed than Flash-Sparse-Attention
- โข9x prefilling and 15x decoding speedups
- โขFirst open-weight model with frontier coding, 1M context, and multimodality
๐ง Deep Insight
Web-grounded analysis with 14 cited sources.
๐ Enhanced Key Takeaways
- โขMiniMax Sparse Attention (MSA) is integrated into the MiniMax M3 model, which is positioned as the first open-weight model to combine frontier coding capabilities, a 1M token context window, and native multimodality (supporting image and video input, and desktop operation).
- โขThe MSA architecture achieves a per-token compute cost that is 1/20th of MiniMax's previous-generation models at a 1M context depth, making ultra-long-context AI agent deployment economically viable.
- โขMSA's efficiency stems from a 'KV outer gather Q' approach at the operator level, which processes Key-Value (KV) blocks sequentially, ensuring contiguous memory access and reading each block only once, unlike traditional methods that fetch blocks multiple times.
- โขMiniMax M3, powered by MSA, demonstrates top-tier performance on coding and agentic benchmarks, surpassing models like GPT-5.5 and Gemini 3.1 Pro on SWE-Bench Pro and OmniDocBench, and approaching Opus 4.7.
- โขThe model's native multimodality means its entire data pipeline was rebuilt for multimodal training from step zero, scaling pre-training data to over 100 trillion, achieving deep alignment between textual and visual semantic spaces.
๐ Competitor Analysisโธ Show
| Feature/Benchmark | MiniMax M3 (with MSA) | GPT-5.5 | Gemini 3.1 Pro | Opus 4.7 | DeepSeek's Multi-head Latent Attention (MLA) / DSA |
|---|---|---|---|---|---|
| Context Window | 1M tokens (native) | Not specified (implied shorter than 1M for direct comparison) | Not specified (implied shorter than 1M for direct comparison) | Not specified (implied shorter than 1M for direct comparison) | Up to 64k context length for NSA (DeepSeek's Native Sparse Attention) |
| Execution Speed (vs. Flash-Sparse-Attention) | 4x faster | N/A | N/A | N/A | N/A |
| Prefilling Speedup | 9x (over previous generation) | N/A | N/A | N/A | 11x performance increase during decoding for NSA at 64k context |
| Decoding Speedup | 15x (over previous generation) | N/A | N/A | N/A | 11x performance increase during decoding for NSA at 64k context |
| Per-token Compute (at 1M context) | 1/20th of previous generation | N/A | N/A | N/A | N/A |
| Coding Capability (SWE-Bench Pro) | Surpasses GPT-5.5 and Gemini 3.1 Pro, approaches Opus 4.7 | Surpassed by MiniMax M3 | Surpassed by MiniMax M3 | Approached by MiniMax M3 | N/A (DeepSeek V3.2 DSA mentioned as having heavy kernel engineering) |
| Multimodal Benchmarks (OmniDocBench) | Scores above Gemini 3.1 Pro | N/A | Scored below MiniMax M3 | N/A | N/A |
| Browsing/Information Retrieval (BrowseComp) | Scores 83.5, surpassing Opus 4.7 (79.3) | N/A | N/A | Scores 79.3 (surpassed by MiniMax M3) | N/A |
| Attention Mechanism | MiniMax Sparse Attention (MSA): block-level selection on real, uncompressed Key-Values, two-stage process (index branch + sparse branch) | Full attention (standard quadratic complexity) | Full attention (standard quadratic complexity) | Full attention (standard quadratic complexity) | Multi-head Latent Attention (MLA) / Native Sparse Attention (NSA): compresses keys and values into a low-dimensional latent space, token-level top-k, sliding window, compression branch |
| Open-weight Status | First open-weight model with frontier coding, 1M context, and multimodality | Closed-source | Closed-source | Closed-source | DeepSeek V3.2 DSA is an open-source sparse attention design |
| Pricing | Token Plan price remains unchanged for M3's enhanced capabilities | N/A | N/A | N/A | N/A |
๐ ๏ธ Technical Deep Dive
- Sparse Attention Mechanism (MSA): MSA is a clean and extensible new sparse attention architecture that avoids the quadratic computational complexity of full attention by calculating attention scores only for selected segments.
- KV Cache Partitioning: The stored context, known as the Key-Value (KV) cache, is split into blocks. A preliminary filtering step determines which blocks are relevant to the current query.
- Memory Access Optimization: MSA adopts a "KV outer gather Q" approach. This means KV blocks are used as the outer loop to aggregate the queries that hit them. Each block is read only once, and memory access is contiguous, significantly improving arithmetic intensity.
- Two-Stage Process: MSA functions as a two-stage mechanism. A lightweight index branch first scans incoming tokens and selects relevant blocks of the KV cache. The main attention layer then performs the expensive attention computation only on these selected blocks.
- Uncompressed Key-Values: Unlike some other sparse attention methods (e.g., DeepSeek's MLA), MSA operates on real, uncompressed Key-Values, which MiniMax states helps avoid precision loss at long contexts.
- Hardware-Friendly Design: The architecture is designed to be hardware-friendly, allowing its theoretical gains to be fully realized in practice, leading to significant speedups.
- Integration with M3: MSA enables the MiniMax M3 model to support a 1M token context window with a guaranteed minimum of 512K tokens, crucial for long-range agent tasks, coding, and video understanding.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
AI agents will become significantly more capable and autonomous.
The combination of 1M context, frontier coding, and native multimodality in an open-weight model provides a robust foundation for developing highly sophisticated AI agents that can handle complex, long-horizon tasks across various data types.
The economic viability of deploying ultra-long-context AI models will increase substantially.
MSA's drastic reduction in per-token compute cost (1/20th of previous models) and significant speedups in prefilling and decoding make it much more cost-effective to run AI models with million-token context windows in production environments.
Open-weight models will accelerate innovation in multimodal and long-context AI.
As the first open-weight model to combine these frontier capabilities, MiniMax M3 will likely foster broader research and development in areas previously dominated by closed-source models, leading to faster advancements and new applications.
โณ Timeline
2021-12
MiniMax founded by former SenseTime researchers.
2022-10
MiniMax launched its first AI companion app, Glow.
2023-06
Glow relaunched as Talkie for international markets.
2024-03
Alibaba Group led a $600 million financing round, valuing MiniMax at $2.5 billion.
2024-03
MiniMax launched Hailuo AI, an AI text and music-generating LLM platform.
2026-01-09
MiniMax held its initial public offering on the Hong Kong Stock Exchange.
2026-06-01
MiniMax M3 model, featuring MiniMax Sparse Attention (MSA), officially released.
๐ Sources (14)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning โ