Dante-2B Phase 1: Bilingual Italian-English LLM Done

🔑 Enhanced Key Takeaways

• •Dante-2B is developed by the Italian research collective 'Dante-AI', which focuses on democratizing high-performance LLMs for Romance languages to counter the English-centric bias in foundational models.
• •The training pipeline utilized a unique data-curation strategy involving synthetic data generation to balance the Italian corpus, specifically targeting regional dialects and formal literary Italian that are often underrepresented in common web crawls.
• •The project is open-source under the Apache 2.0 license, with the team explicitly aiming to provide a lightweight alternative for edge-computing applications in the Italian public sector and educational institutions.

🛠️ Technical Deep Dive

• •Tokenizer: Custom 64K BPE vocabulary trained on a balanced corpus of Italian literature, legal documents, and technical manuals to minimize sub-word fragmentation for Italian-specific morphology.
• •Hardware Utilization: Achieved 28% Model Flops Utilization (MFU) by leveraging FP8 precision on NVIDIA H200s, utilizing DeepSpeed ZeRO-2 for memory-efficient sharding of optimizer states.
• •Architecture Details: 28 layers, d_model=2560, 20 attention heads for queries, 4 heads for keys/values (GQA), SwiGLU activation function, and RoPE (Rotary Positional Embeddings) with a base frequency of 10,000.
• •Data Composition: 100B tokens total; 42B Italian (web, books, legal), 36B English (refined web/academic), 22B code (GitHub-derived, filtered for Italian-commented snippets).

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline