Tag: #tokenizer5 results
quicktok is a new C++ based BPE tokenizer that achieves 4–11× faster performance than tiktoken while remaining byte-identical. It supports major models including Llama-3, Qwen2.5/3, and various GPT-based encodings.
Meituan released and fully open-sourced its native multimodal large model LongCat-Next and core component dNaViT visual tokenizer on March 27. It unifies image, speech, and text into discrete tokens, breaking language-centric architectures. The model uses pure Next Token Prediction for native vision and speech handling.
A developer shares insights from training a 216M parameter SLM from scratch, highlighting the critical importance of tokenizer quality over architecture tweaks. The project details the challenges of GGUF export and the impact of dataset composition on conversational performance.
Western LLMs like Claude/GPT charge more tokens for Chinese due to BPE tokenizers trained on English-heavy data, splitting hanzi inefficiently. Chinese models Qwen/DeepSeek optimize for hanzi/words, making Chinese cheaper. Claude 4.7 upgrade inflated English tokens only, sparing Chinese users.
New zero-allocation, header-only C++ tokenizer for Qwen models achieves 1009 MB/s on Ryzen 5 3600, nearly 20x faster than OpenAI Tiktoken's 50 MB/s. Designed for HPC with no dependencies, it's an educational optimization project. Source code available on GitHub.