Mamba-3: Faster SSM for Inference

Post LinkedIn

🤝Read original on Together AI Blog

#ssm#inference#state-space-modelmamba-3mamba-3together-aimamba-2

💡Open-source SSM beats Transformers on decode – transform your inference pipeline!

⚡ 30-Second TL;DR

What Changed

SSM built specifically for inference workloads

Why It Matters

Mamba-3 challenges Transformer dominance in inference, enabling cost-effective deployments. Open-source access accelerates adoption by developers and researchers.

What To Do Next

Clone Mamba-3 repo from Together AI and benchmark decode speed on your GPU.

Who should care:Developers & AI Engineers

Key Points

•SSM built specifically for inference workloads
•Faster decoding than Transformers
•Stronger performance than Mamba-2
•Open-source from launch day

🧠 Deep Insight

Background and context from public sources — not the original article. 7 sources cited.

🔑 Enhanced Key Takeaways

•Mamba-3 introduces a more expressive recurrence formula from SSM discretization, complex-valued state updates for richer dynamics, and a MIMO variant that enhances accuracy without increasing decode latency.[1][2]
•Mamba-3 SISO achieves 100% accuracy on parity state-tracking tasks where Mamba-2 scores near-random (0.90%), recovering rotational dynamics lost in prior real-valued simplifications.[3]
•At 1.5B scale, Mamba-3 MIMO improves downstream language modeling accuracy by +2.2 points over Transformers, +1.9 over Mamba-2, and +1.8 over Gated DeltaNet.[2]

📊 Competitor Analysis▸ Show

Feature	Mamba-3 SISO (1.5B)	Mamba-3 MIMO (1.5B)	Mamba-2 (1.5B)	Gated DeltaNet (1.5B)	Llama-3.2-1B (Transformer)
Prefill+Decode Latency	Fastest across seq lengths	Comparable to Mamba-2	Slower	Slower	Slower
Downstream Accuracy	+0.6 pts over GDN	+1.8 pts over GDN (+2.2 over Transformer)	Lower	Baseline	Lower
State Size Efficiency	Half of Mamba-2 for equiv perplexity	Half of Mamba-2 for equiv perplexity	128	N/A	N/A
Pricing	Open-source (free)	Open-source (free)	Open-source	Open-source	Open-source

🛠️ Technical Deep Dive

•Core improvements: (1) Expressive recurrence from SSM discretization, (2) Complex-valued state update rule to recover rotational dynamics, (3) MIMO formulation for multi-input multi-output processing without decode overhead.[1][2]
•SISO variant matches Mamba-2 architecture shapes (dimensions, state size) but outperforms on quality; MIMO boosts accuracy by >1 pt at 1B scale despite longer training.[1]
•Advances Pareto frontier: Mamba-3 (state size 64) matches Mamba-2 (state size 128) perplexity, halving latency; fixes memory-bound decoding and state-tracking failures like parity/arithmetic.[2][3]

🔮 Future ImplicationsAI analysis grounded in cited sources

Mamba-3 shifts SSM design priority to inference over training

Unlike Mamba-2's training focus, Mamba-3's inference-first upgrades like complex states and MIMO enable real-time deployment advantages on hardware like H100 without idle cores.[1][3]

Open-source Mamba-3 accelerates hybrid architectures

Superior latency and state-tracking vs Transformers positions it for integration into MoE or speculative decoding systems, as seen in emerging Nemotron hybrids.[1][4]

SSM inference reaches Transformer parity at 1.5B scale

Mamba-3's fastest prefill+decode and +2.2 pt accuracy gain over Llama-3.2-1B validate linear-time models for production-scale language tasks.[1][2]

⏳ Timeline

2023-12

Mamba-1 released as selective SSM architecture for language modeling.

2024-09

Mamba-2 launched, optimizing for training speed and linear attention.

2026-03

Mamba-3 announced by Together AI, focusing on inference efficiency with SISO/MIMO variants.

📎 Sources (7)

Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.

🤝Read original article on Together AI Blog

📰

Weekly AI Recap

Read this week's curated digest of top AI events →

👉Related Updates

Same topic

Explore #ssm

Same product