๐ฆReddit r/LocalLLaMAโขFreshcollected in 60m
Ling-3.0 Releases Six Base Checkpoints
๐กSix MIT-licensed base checkpoints let builders choose their entry point for continued training.
โก 30-Second TL;DR
What Changed
The release includes six separate official repositories
Why It Matters
The staged release gives model builders more control over where to enter the training pipeline instead of forcing a single base checkpoint. It is particularly useful for researchers comparing continued-pretraining starting points, though downstream quality and quantization behavior remain unestablished.
What To Do Next
Download the three Ling-3.0 stages for one size and run the same fine-tuning recipe to compare downstream validation loss and task performance.
Who should care:Developers & AI Engineers
Key Points
- โขThe release includes six separate official repositories
- โขTwo sizes are available: tiny and flash
- โขEach size has pretrained, mid-trained, and WSM-merged checkpoints
- โขAll checkpoints are base models with no post-training
- โขRepositories were public, ungated, and declared MIT-licensed when checked
๐ง Deep Insight
Background and context from public sources โ not the original article. 15 sources cited.
๐ Enhanced Key Takeaways
- โขAntLing is an artificial intelligence initiative from Ant Group, specifically developed by its InclusionAI lab.
- โขThe Ling-3.0 models feature a native hybrid-linear attention architecture, which involves alternating Kimi Delta Attention (KDA) and Multi-Head Latent Attention (MLA) layers.
- โขThe 'tiny' model has 7.9 billion total parameters with approximately 1.3 billion parameters activated per token, while the 'flash' model boasts 124 billion total parameters with about 5.1 billion activated parameters per token.
- โขLing-3.0 models are specifically engineered for AI agents, coding, mathematics, instruction following, tool use, and long-context applications, supporting a native context window of up to 256K tokens, with the 'flash' model scalable to 1 million tokens.
- โขThe 'WSM-merged' checkpoints refer to the Warmup-Stable-Merge (WSM) learning rate schedule, a decay-free approach that utilizes checkpoint merging as an alternative to traditional learning rate annealing, aiming for superior performance and continuous training in LLM pre-training.
๐ Competitor Analysisโธ Show
| Model | Total Parameters | Active Parameters (per token) | Architecture | Context Window | Key Benchmarks / Features |
|---|---|---|---|---|---|
| AntLing Ling-3.0-flash | 124B | 5.1B | MoE, Hybrid-Linear Attention (5:1 KDA:MLA) | 256K (scales to 1M) | Matches/surpasses models 2-3x its size |
| AntLing Ling-3.0-tiny | 7.9B | 1.3B | MoE, Hybrid-Linear Attention (3:1 KDA:MLA) | 256K | Artificial Analysis Intelligence Index: 25, Agentic Index: 16 |
| Kimi K3 | 2.8T | N/A | MoE | 1M | GPQA Diamond: 93.5% |
| DeepSeek V4 Pro | N/A | N/A | MoE | 1M | SWE-Bench: 80.6% |
| Google Gemma 4 31B | 31B | N/A | Dense | N/A | MMLU: 85.2%, GPQA Diamond: 84.3%, SWE-Bench: 80.0% |
| Qwen3-Coder-480B | 480B | 35B | MoE | 256K (extendable to 1M) | SWE-bench Verified: ~70% |
Note: Pricing for open-source models typically refers to the cost of self-hosting and inference, rather than a direct purchase price. Ling-3.0-flash was initially offered with promotional API pricing before its weights were open-sourced.
๐ ๏ธ Technical Deep Dive
- Model Architecture: Ling-3.0 models employ a native hybrid-linear attention architecture. The 'flash' model uses a 5:1 alternating stack of Kimi Delta Attention (KDA) and Multi-Head Latent Attention (MLA) layers, meaning five KDA layers for every one MLA layer in a block. The 'tiny' model uses a 3:1 KDA-MLA architecture.
- Mixture-of-Experts (MoE): Ling-3.0-flash features 124 billion total parameters with an expert activation ratio compressed to 1/64, meaning approximately 5.1 billion parameters are active per token. Ling-3.0-tiny has 7.9 billion total parameters, activating about 1.3 billion parameters per token, utilizing a sparse MoE FFN with 128 routed experts where only 8 routed experts and 1 shared expert are activated per token.
- Kimi Delta Attention (KDA): This is an upgraded form of Lightning Attention, incorporating fine-grained diagonal gating in its Delta Rule state updates. This enhancement allows the model to retain critical information more precisely when processing extensive documents and codebases, improving long-range memory.
- Context Window: Both Ling-3.0-tiny and Ling-3.0-flash natively support a 256K token context window. The 'flash' model is designed to seamlessly scale up to 1 million tokens.
- Deployment Flexibility: Ling-3.0-tiny provides weights in BF16, FP8, and INT4 formats, optimizing for a wide range of hardware and deployment settings, including local and resource-constrained environments like Apple Silicon MacBooks.
- Warmup-Stable-Merge (WSM): This is a decay-free learning rate schedule used in pre-training. WSM replaces traditional learning rate annealing by strategically merging checkpoints, establishing a theoretical connection between learning rate decay and model averaging. This method aims to eliminate the explicit decay phase, enabling continuous training and achieving performance improvements over conventional Warmup-Stable-Decay (WSD) approaches.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Ling-3.0's hybrid-linear attention architecture will become a benchmark for efficient long-context processing.
Its novel design, combining KDA and MLA with optimized MoE, directly addresses the challenges of long-context efficiency and computational cost, likely influencing future LLM architectural innovations.
The MIT license will significantly boost the adoption and community-driven development of Ling-3.0.
The highly permissive nature of the MIT license encourages widespread commercial and research use without stringent restrictions, fostering a larger ecosystem of derivative works and applications.
The Warmup-Stable-Merge (WSM) training methodology could gain broader acceptance as a standard for LLM pre-training.
WSM's demonstrated ability to achieve superior performance and enable continuous training by eliminating traditional learning rate decay offers a more stable and efficient training paradigm that could be adopted by other developers.
โณ Timeline
2025-08-20
InclusionAI releases Ling Lite 1.5 model.
2025-09-09
InclusionAI releases Ling Mini 2.0 model.
2025-10-13
Ant Group releases Ling-1T, a 1-trillion parameter MoE model with a new LPO post-training method.
2026-07-23
Ant Group's InclusionAI lab initially releases Ling-3.0-flash, available via API.
2026-07-27
Ant Group officially announces Ling-3.0-Flash, detailing its hybrid-linear attention and MoE architecture.
2026-08-17
Details emerge for Ling 3.0 Tiny, highlighting its architecture and focus on AI agents.
๐ Sources (15)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.