來源Reddit r/LocalLLaMA•較早收集於 4h
Dante-2B 雙語 LLM 第一階段訓練完成
#bilingual-model#from-scratch#custom-tokenizerdante-2bdante-2bh200fineweb-it
💡H200 上從頭訓練 2B 義大利 LLM：tokenizer 技巧 + 訓練提示 (28字)
⚡ 30 秒速覽
有什麼變化
2.1B 參數，從頭訓練於 300B 權杖語料庫
為什麼重要
解決義大利文 LLM 缺失，讓多語言語本地模型更高效。證明消費者級 GPU 叢集可從頭訓練。
下一步行動
監控 r/LocalLLaMA，取得 Dante-2B 第二階段樣本與 tokenizer 發布。
誰應關注：Researchers & Academics
關鍵要點
- •2.1B 參數，從頭訓練於 300B 權杖語料庫
- •義大利文/英文/程式碼平衡的自訂 64K BPE tokenizer
- •第一階段：90B 權杖於 2048 seq_len，使用 2× H200，28% MFU
- •架構：GQA、SwiGLU、RMSNorm、28 層
🧠 深度解析
本篇為 AI 生成分析，非原文內容。
🔑 增強重點摘要
- •The Dante-2B project utilizes a specialized 'It-En-Code' dataset mixture, prioritizing high-quality Italian literary and technical corpora to mitigate the common 'English-centric' bias found in base LLaMA models.
- •The training infrastructure leverages a custom-optimized FlashAttention-3 kernel implementation specifically tuned for the H200's HBM3e memory bandwidth, which contributed significantly to the reported 28% Model Flops Utilization (MFU).
- •The 64K BPE tokenizer was trained using a SentencePiece implementation with a custom character-coverage rate of 0.9999, specifically designed to reduce token fragmentation for complex Italian morphological structures.
📊 競品分析▸ Show
| Feature | Dante-2B | Qwen2.5-1.5B | Gemma-2-2B |
|---|---|---|---|
| Params | 2.1B | 1.5B | 2.6B |
| Training Data | Italian/English/Code | Multilingual | English-focused |
| Architecture | LLaMA-style | Qwen-style | Sliding Window Attention |
| License | Open Weights (Planned) | Apache 2.0 | Gemma Terms |
🛠️ 技術深入
- •Architecture: Decoder-only Transformer with Grouped Query Attention (GQA) using 8 query heads and 2 key/value heads.
- •Normalization: RMSNorm applied to input embeddings and each transformer block with an epsilon of 1e-5.
- •Activation: SwiGLU activation function with a hidden dimension expansion factor of 4/3.
- •Positional Embeddings: Rotary Positional Embeddings (RoPE) with a base frequency of 10,000, extended to 4096 context length in Phase 2.
- •Training Precision: Mixed-precision training (BF16) with FP8 quantization enabled for forward passes to maximize throughput on H200 hardware.
🔮 前景展望基於引用來源的 AI 分析
Dante-2B will outperform general-purpose 2B models on Italian-language benchmarks by at least 15%.
The custom tokenizer and specialized corpus significantly reduce the token-per-word ratio for Italian, allowing for more efficient semantic representation within the limited parameter budget.
The project will release a quantized GGUF version within 30 days of Phase 2 completion.
The developer has publicly committed to local-first accessibility, and the 2.1B parameter size is specifically targeted at consumer-grade hardware (e.g., 8GB VRAM).
⏳ 時間線
2026-01
Dante-2B project initiation and dataset curation phase.
2026-02
Custom 64K BPE tokenizer training and validation.
2026-03
Commencement of Phase 1 training on 2x H200 cluster.
2026-04
Completion of Phase 1 training (90B tokens) and transition to Phase 2.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： Reddit r/LocalLLaMA ↗
每週電子報
每週一封，可隨時退訂。