Tag: #llm-architecture16 results
Kimi unveils a groundbreaking new architecture that impressed Elon Musk. The innovation rotates the attention mechanism by 90 degrees. It was authored by a 17-year-old high school student, propelling them to fame.
Hierarchos is an experimental 232M-parameter language model that utilizes a hybrid non-Transformer architecture to achieve instruction coherence. It combines RWKV recurrence, hierarchical manager/worker loops, and differentiable slot-based long-term memory to prove efficiency in small-scale models.
This research proposes a novel architecture that handles massive context by treating semantic compression as a diffusion-like noise function. The model processes source documents in multiple passes at varying compression levels to refine an integration state without exceeding the context window.
This study provides a comprehensive evaluation of eight state-of-the-art Diffusion Language Models (DLMs) across various benchmarks. It explores the trade-offs between generation quality and computational efficiency, highlighting how inference-time design choices impact performance.
This research explores using visual graph mind maps as internal reasoning scaffolds for LLMs. It finds that visual graph guidance significantly outperforms flattened text representations in multi-hop reasoning tasks.
Rumors circulate about a new Mistral model, 'Le Gros Chaton', allegedly featuring 1B context and real-time self-improvement. The community is questioning its open-source status.