🏠IT之家•Stalecollected in 2m
Tencent Hunyuan introduces Stem sparse attention algorithm
💡Reduce LLM inference costs and latency by 3.6x using Tencent's new open-source sparse attention algorithm.
⚡ 30-Second TL;DR
What Changed
Reduces first-token latency by 3.6x in 128K context scenarios.
Why It Matters
This research provides a practical pathway for developers to optimize long-context LLM inference efficiency without significant quality degradation.
What To Do Next
Clone the AngelSlimHPC repository and test the Stem+BSA operators on your long-context inference tasks to evaluate latency improvements.
Who should care:Researchers & Academics
Key Points
- •Reduces first-token latency by 3.6x in 128K context scenarios.
- •Achieves near-dense precision using only 25% of computational resources.
- •Innovations include Token Position Decay (TPD) and Output-Aware Metric (OAM).
- •Open-source HPC operator library available on GitHub for real-world deployment.
🧠 Deep Insight
Web-grounded analysis with 10 cited sources.
🔑 Enhanced Key Takeaways
- •The Stem algorithm functions as a 'plug-and-play' sparsity module, allowing seamless integration with existing training-free and training-based sparse attention models, such as DeepSeek-V3.2 and MiniCPM-4.1, without requiring retraining.
- •The Token Position Decay (TPD) strategy within Stem dynamically adjusts the sparse budget, prioritizing initial tokens crucial for recursive dependencies to prevent global distortion in causal information flow.
- •Stem's Output-Aware Metric (OAM) improves token selection by prioritizing high-impact tokens based on their approximate output magnitude, incorporating value information, which is a significant advancement over methods relying solely on attention scores.
- •Evaluations on benchmarks like LongBench and RULER demonstrate Stem's superior performance, achieving higher accuracy with reduced computational budgets compared to other training-free sparse attention methods.
- •Tencent Hunyuan's open-source HPC operator library (HPC-Ops), which includes optimized sparse attention kernels, is production-grade, tailored for NVIDIA H20 GPUs, and offers up to a 2.22x speedup for large language model (LLM) inference, with easy integration into frameworks like vLLM and SGLang.
🛠️ Technical Deep Dive
- Problem Addressed: The core challenge Stem addresses is the quadratic computational complexity of self-attention, particularly during the pre-filling phase of LLM inference, which creates a bottleneck for scaling models to long contexts.
- Causal Information Flow: Stem rethinks sparse attention by explicitly modeling causal information flow, recognizing that many existing sparse methods overlook the cumulative dependency of token information in causal architectures.
- Token Position Decay (TPD) Strategy:
- This strategy applies a position-dependent top-k selection within each layer of the attention mechanism.
- It dynamically adjusts the sparse budget, allocating a larger proportion of computational resources to tokens at the initial positions in a layer.
- The goal is to retain these initial tokens, which are critical for recursive dependencies, thereby preventing global information distortion across layers.
- Output-Aware Metric (OAM):
- OAM is designed to preserve information-rich tokens by prioritizing those with a high approximate output magnitude.
- Unlike previous Score-Aware Metrics (SAM) that only approximated attention scores, OAM simulates the actual output magnitude by incorporating value information.
- This ensures that tokens contributing significantly to the final output are retained, minimizing the accuracy loss between sparse and dense attention.
- Implementation: Stem is a training-free framework that integrates TPD and OAM. It leverages the Block Sparse Attention library for efficient sparse kernel execution. The associated HPC operator library (HPC-Ops) provides deeply optimized sparse attention kernels.
- Inference Phases: The algorithm primarily targets the pre-filling phase, where the entire input prompt is processed in parallel to compute Key-Value (KV) states, which is the most computationally intensive part of LLM inference for long contexts.
🔮 Future ImplicationsAI analysis grounded in cited sources
Operational costs for long-context LLMs will significantly decrease.
The 3.6x reduction in first-token latency and use of only 25% of the computational budget directly translates to lower infrastructure and energy costs for running large language models with extensive context windows.
Adoption of long-context LLMs will accelerate across various industries.
By making long-context processing more efficient and cost-effective, Stem removes a major barrier, enabling broader deployment of LLMs in applications requiring deep understanding of lengthy documents, complex code, or extended conversations.
Real-time AI applications will become more responsive and user-friendly.
The substantial reduction in first-token latency directly improves the perceived speed and interactivity of AI systems, enhancing user experience in applications like conversational AI, intelligent assistants, and real-time content generation.
⏳ Timeline
2016
Tencent AI Lab established, laying groundwork for AI research.
2023-09
Tencent debuted its proprietary universal large language model, Hunyuan.
2025-03
Tencent released Hunyuan T1, a reasoning language model based on the Transformer-Mamba architecture.
2026-03
The research paper 'Stem: Rethinking Causal Information Flow in Sparse Attention' was published on arXiv.
2026-04
Tencent updated its Hunyuan AI model (Hy3), featuring improvements in complex reasoning and coding capabilities.
2026-06
Tencent Hunyuan introduced the Stem sparse attention algorithm and released its HPC-Ops operator library.
📎 Sources (10)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: IT之家 ↗