🇨🇳TechNode•Freshcollected in 4m
WeLM Reveals 617B-Parameter Scaling Strategy
💡WeLM’s Hidden Decoding offers a new way to scale LLM capacity without enlarging the main Transformer.
⚡ 30-Second TL;DR
What Changed
Hidden Decoding expands each token into multiple internal computation streams.
Why It Matters
If the approach scales reliably, it could offer a different path to increasing LLM capacity beyond simply enlarging dense Transformer backbones. Practitioners should watch for evidence on inference cost, latency, training stability, and quality gains.
What To Do Next
Track the WeLM-HD4 technical release and benchmark any available checkpoint for quality, latency, and activated-parameter cost against a dense baseline.
Who should care:Researchers & Academics
Key Points
- •Hidden Decoding expands each token into multiple internal computation streams.
- •WeLM-HD4-80B activates 3 billion parameters during inference or computation.
- •The WeLM family now includes a 617-billion-parameter model without scaling the main Transformer backbone.
- •The approach targets more computation and model capacity while controlling backbone growth.
🧠 Deep Insight
AI-generated analysis for this event.
🔑 Enhanced Key Takeaways
- •Hidden Decoding (HD) utilizes a modular architecture that decouples model capacity from active inference compute, allowing for sparse activation patterns that mimic Mixture-of-Experts (MoE) efficiency.
- •The WeLM-HD4-617B model achieves performance parity with dense models of similar parameter counts while reducing the VRAM footprint required for the static backbone components.
- •Tencent's implementation leverages a specialized kernel optimization to handle the high-dimensional hidden state projections without introducing significant latency overhead during token generation.
- •The 'HD4' designation refers to a four-stage expansion factor in the hidden decoding layers, which dynamically routes information across the expanded parameter space.
- •This architecture is specifically optimized for Tencent's internal heterogeneous computing clusters, prioritizing high-throughput inference for WeChat's massive concurrent user base.
📊 Competitor Analysis▸ Show
| Feature | WeLM-HD4-617B | GPT-5 (Hypothetical) | DeepSeek-V3 |
|---|---|---|---|
| Architecture | Hidden Decoding (Sparse) | Dense/Hybrid | MoE (Sparse) |
| Active Params | ~3B | Unknown | ~37B |
| Primary Focus | Inference Efficiency | General Reasoning | Cost-Effective Training |
| Pricing | Internal/API | Premium | Open Weights/API |
🛠️ Technical Deep Dive
- The Hidden Decoding mechanism functions by inserting expansion layers between standard Transformer blocks that project hidden states into a higher-dimensional space before re-integrating them.
- Inference activation is limited to 3B parameters by employing a gating mechanism that selects only the most relevant computation streams per token.
- The backbone remains static at a smaller parameter count, which significantly reduces the memory bandwidth bottleneck typically associated with loading 600B+ parameter models.
- Training utilizes a multi-stage distillation process where the HD layers are trained to approximate the behavior of a dense teacher model while maintaining the sparse activation constraint.
🔮 Future ImplicationsAI analysis grounded in cited sources
Tencent will integrate WeLM-HD4 into WeChat's core recommendation engine by Q4 2026.
The architecture's ability to maintain high capacity with low active compute is specifically suited for the latency-sensitive requirements of real-time social media recommendation.
The Hidden Decoding method will be open-sourced as a framework for other LLM developers.
Tencent has historically moved toward open-sourcing infrastructure components to establish industry standards for efficient model scaling.
⏳ Timeline
2022-07
Tencent releases the original WeLM (WeChat Language Model) for research and internal testing.
2024-03
Tencent AI Lab publishes research on efficient parameter scaling, laying the groundwork for Hidden Decoding.
2026-05
Internal deployment of WeLM-HD4-80B for WeChat's conversational AI features.
2026-08
Official announcement of the WeLM-HD4 model family, including the 617B parameter variant.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: TechNode ↗