⚡雷峰网•Stalecollected in 2h
ByteDance Seed Team's Four CVPR 2026 Breakthroughs in Efficiency
💡Learn how ByteDance is cutting AI inference costs by 10x using algorithmic optimizations instead of more GPUs.
⚡ 30-Second TL;DR
What Changed
TEMF enables single-step generation by learning bidirectional mapping, replacing multi-step iterative sampling.
Why It Matters
These techniques offer a viable path for enterprises to maintain high-performance AI models despite GPU supply constraints and rising energy costs. By optimizing at the algorithmic level, companies can significantly extend the lifespan of existing hardware infrastructure.
What To Do Next
Review your current inference pipeline for KV cache bottlenecks and evaluate if token-level importance routing can be integrated into your attention layers to save compute.
Who should care:Researchers & Academics
Key Points
- •TEMF enables single-step generation by learning bidirectional mapping, replacing multi-step iterative sampling.
- •Beyond Token Eviction introduces mixed-dimension KV cache compression to optimize memory usage without full retraining.
- •Mixture-of-Depths Attention dynamically routes tokens to reduce redundant computation in Transformer layers.
- •The research suite provides a comprehensive framework for reducing inference costs across generation, memory, and compute.
🧠 Deep Insight
Web-grounded analysis with 13 cited sources.
🔑 Enhanced Key Takeaways
- •Mixture-of-Depths Attention (MoDA) directly tackles the 'information dilution' problem in deep Large Language Models (LLMs) by enabling attention heads to dynamically access Key-Value (KV) pairs from all preceding layers, achieving 97.3% of FlashAttention-2's efficiency at 64K context length with only a 3.7% FLOPs computational overhead.
- •The 'Beyond Token Eviction' research introduces MixedDimKV, a novel KV cache compression method that allocates dimensions to tokens at a more granular level, significantly outperforming prior token eviction techniques. Its variant, MixedDimKV-H, which incorporates head-level importance, achieves performance comparable to full attention on LongBench while using only 6.25% of the KV cache and maintaining 100% accuracy at 50K context length in Needle-in-a-Haystack tests with just 0.26% cache usage.
- •ByteDance's efforts in single-step generation extend to both video and code. Their 'Seaweed-APT' model, utilizing Diffusion Adversarial Post-Training, can generate 2-second, 1280×720, 24fps videos in real-time with a single forward pass. Additionally, 'Seed Diffusion Preview' for code generation achieves speeds of 2,146 tokens per second on Nvidia H20 GPUs by employing a discrete-state diffusion approach for parallel token reconstruction.
- •The ByteDance Seed team's research scope is broad, encompassing not only efficiency breakthroughs but also foundational work in Artificial General Intelligence (AGI) through its 'Seed Edge' initiative, multimodal interaction, world models, AI for science, and robotics, indicating a comprehensive long-term strategy in advanced AI development.
🛠️ Technical Deep Dive
- Mixture-of-Depths Attention (MoDA):
- Addresses signal degradation and 'information dilution' in deep LLMs where informative features from shallow layers are lost through repeated residual updates.
- Each attention head dynamically attends to both sequence Key-Value (KV) pairs from the current layer and depth KV pairs from all preceding layers.
- Unifies sequence attention and depth attention under a single softmax function.
- Features a hardware-aware implementation that resolves non-contiguous memory-access patterns, achieving 97.3% of FlashAttention-2's efficiency at a 64K context length.
- Introduces a negligible 3.7% FLOPs computational overhead.
- Improves average perplexity by 0.2 and average performance by 2.11% across 10 downstream tasks for 1.5B-parameter models.
- Combines with post-normalization for superior performance compared to pre-normalization.
- Utilizes a custom Triton kernel for a 1458x speedup over naive PyTorch implementations, employing flash-compatible depth-KV layout, chunk-aware depth packing, and group-aware indexing for memory optimization.
- Beyond Token Eviction (MixedDimKV):
- Proposes MixedDimKV, a KV cache compression method that allocates dimensions to tokens at a more granular level, moving beyond coarse token eviction (zero or full dimension).
- MixedDimKV-H is an enhanced version that integrates head-level importance information.
- Achieves comparable performance to full attention on LongBench with only 6.25% of the KV cache.
- Demonstrates 100% accuracy at a 50K context length in Needle-in-a-Haystack tests while using only 0.26% of the cache.
- Inspired by dimensionality reduction techniques for KV cache compression.
- Single-step generation (TEMF - related work):
- Seaweed-APT: Employs Diffusion Adversarial Post-Training (APT) against real data, following diffusion pre-training, for one-step video generation.
- Capable of generating 2-second, 1280×720, 24fps videos in real-time and 1024px images in a single forward evaluation step.
- Seed Diffusion Preview: An experimental AI model for code generation designed for parallel token generation.
- Achieves inference speeds of 2,146 tokens per second on Nvidia H20 GPUs.
- Uses a 'discrete-state diffusion' approach, reconstructing code from a noisy, placeholder-filled state.
- Employs a two-stage training process: mask-based training followed by edit-based training with insertions and deletions to enhance quality.
- Optimizes generation order based on code structure and dependencies and uses on-policy learning with a verifier model to minimize generation steps while maintaining output quality.
🔮 Future ImplicationsAI analysis grounded in cited sources
LLM inference costs will significantly decrease, making advanced AI more accessible for a wider range of applications.
ByteDance's breakthroughs directly target reducing inference costs and hardware reliance through single-step generation, efficient KV cache compression, and dynamic attention, which are critical bottlenecks for deploying large models.
Future LLMs will be able to process and understand much longer contexts more effectively without significant performance degradation.
MoDA directly addresses information dilution in deep models, and MixedDimKV drastically reduces the KV cache memory footprint for long contexts, enabling more robust long-context understanding and generation.
ByteDance will strengthen its competitive position in the global AI industry, particularly in consumer-facing AI applications.
These innovations provide a significant competitive edge by enabling the deployment of powerful AI models more cheaply and efficiently, which is crucial for scaling consumer products like Doubao and TikTok, where ByteDance already has a strong presence.
⏳ Timeline
2012
ByteDance founded by Zhang Yiming.
2014
Toutiao, an AI-powered news aggregation app, launched.
2016
ByteDance AI Lab established, led by Wei-Ying Ma.
2017
TikTok (Douyin internationally) launched, leveraging AI for personalized video recommendations.
2023
ByteDance launched its first LLM, Skylark (later rebranded to Doubao).
2025-01
ByteDance launched Seed Edge, an AGI research initiative, and Doubao 1.5 Pro.
2026-03
Mixture-of-Depths Attention (MoDA) paper released on arXiv.
2026-03
Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression paper released on arXiv.
📎 Sources (13)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: 雷峰网 ↗