Deploy Qwen3.8-2.4T-A95B on SageMaker
💡See the exact AWS stack for serving a 2.4T open-weight model with reasoning and tool use.
⚡ 30-Second TL;DR
What Changed
The deployment targets the 2.4-trillion-parameter Qwen3.8-2.4T-A95B model.
Why It Matters
This lowers the barrier to serving an extremely large open-weight model through managed AWS infrastructure. Teams can evaluate advanced reasoning and tool-use workloads without building the entire serving stack from scratch.
What To Do Next
Follow the AWS guide on a small HyperPod cluster and benchmark NVFP4 throughput against your current vLLM deployment.
Key Points
- •The deployment targets the 2.4-trillion-parameter Qwen3.8-2.4T-A95B model.
- •SageMaker HyperPod and vLLM are used for cluster-based serving.
- •The workflow includes NVFP4 quantization.
- •The endpoint supports reasoning, tool calling, and native MTP speculative decoding.
🧠 Deep Insight
Background and context from public sources — not the original article. 4 sources cited.
🔑 Enhanced Key Takeaways
- •Authored by Dmitry Soldatkin, Andrew Smith, and Vinay Arora, this deployment guide is part of an AWS HyperPod series benchmarking trillion-parameter open models, following a previous guide for Moonshot AI's Kimi K3.
- •Qwen3.8-2.4T-A95B was released on August 12, 2026, representing the first time Alibaba made a flagship 'Max-class' frontier model available with open weights.
- •The model activates only 95 billion parameters per token via a fine-grained MoE architecture consisting of 512 routed experts (10 active per token) and 1 shared expert.
- •Across 92 layers, the architecture utilizes a hybrid linear-attention pattern repeating three Gated DeltaNet-plus-MoE layers for every one Gated Attention-plus-MoE layer.
- •The AWS deployment blueprint specifically targets Amazon SageMaker HyperPod on ml.p6-b300 instances powered by 8× NVIDIA B300 Blackwell Ultra GPUs.
📊 Competitor Analysis▸ Show
| Model / Architecture | Developer | Total Parameters | Active Parameters | Context Window | Native Thinking Mode |
|---|---|---|---|---|---|
| Qwen3.8-2.4T-A95B | Alibaba Cloud / Qwen | 2.4 Trillion | 95 Billion | 262K (extensible to ~1M) | Mandatory ( |
| Kimi K3 | Moonshot AI | Trillion-class MoE | Proprietary / Disclosed separately | Long-context | Yes |
🛠️ Technical Deep Dive
- Parameter Scale: 2.4 trillion total parameters with 95 billion active parameters per token.
- MoE Configuration: 512 routed experts total with 10 routed experts activated per token, alongside 1 shared expert.
- Layer Layout: 92 layers utilizing a hybrid linear/full attention layout structured as
3 × (DeltaNet → MoE) → 1 × (Attention → MoE)to optimize long-sequence throughput. - Context Window: 262,144 native tokens, extensible up to ~1 million tokens.
- Inference Stack: vLLM serving framework running on AWS
ml.p6-b300instances (8× NVIDIA B300 Blackwell Ultra GPUs) with NVFP4 quantization and native Multi-Token Prediction (MTP) speculative decoding. - Reasoning Mechanism: Non-optional mandatory native thinking mode generating structured chain-of-thought tokens within
<think>blocks.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
📎 Sources (4)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: AWS Machine Learning Blog ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.