Holotron-12B High Throughput Agent

🔑 Enhanced Key Takeaways

• •Holotron-12B likely refers to NVIDIA's Nemotron 3 Super 120B-A12B, a model with 120B total parameters but only 12B active parameters via its Mixture-of-Experts design[1][2].
• •Features a hybrid Mamba-Attention MoE architecture, extending the design from Nemotron-3 Nano for enhanced agentic reasoning capabilities[2].
• •Pre-trained on a 25-trillion-token corpus using NVFP4, with data mixtures including code (14%), synthetic crawl (22.4%), and math (6.4%)[2].
• •Supports a 1M-token context window and demonstrates superior benchmark accuracy over models like Ling-flash-Base-2.0 and GLM-4.5-Air-Base[1][2].

🛠️ Technical Deep Dive

• •Total parameters: 120B, active parameters: 12B in MoE configuration[1][2].
• •Architecture: Hybrid Mamba-Attention Mixture-of-Experts (MoE), scaling up from Nemotron-3 Nano[2].
• •Variants: Nemotron 3 Super 120B-A12B FP8 (post-trained and FP8 quantized), BF16 (post-trained), and Base BF16[2].
• •Pre-training: 25-trillion-token corpus across phases with mixtures like crawl-high (6.5%), syn-crawl-high (22.4%), math (6.4%), wiki (0.6%), code (14%); used Warmup-Stable-Decay hyperparameters and NVFP4[2].
• •Context window: 1M tokens; optimized for long-context extension and agentic reasoning[1][2].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline