Tag: #memory-efficiency7 results
Spectral Compact Training (SCT) achieves 172x memory reduction (1,245 GB to 7.24 GB) for 70B-class LLaMA-3 style models. Full training steps run on Steam Deck CPU in 6.28 seconds with exact gradient matching dense methods. Open-source code released under Apache 2.0.
Gefen is a new drop-in replacement for the AdamW optimizer designed to significantly reduce memory overhead during LLM training. It aims to make large-scale training more accessible by optimizing memory usage by up to 8x.
Apple proposes Stochastic KV Routing to enable adaptive depth-wise cache sharing in transformer models, reducing KV cache memory for high-throughput serving. It optimizes the depth dimension, orthogonal to temporal compression methods. Prior research indicates full per-layer caches are redundant.
Google unveiled a research breakthrough that significantly reduces memory requirements for AI development. This triggered a sharp sell-off in storage chip stocks, with SK Hynix and Samsung dropping at least 6% in Seoul. US firms like Micron, Western Digital, and SanDisk fell over 5%.
A community implementation demonstrates VibeVoice 1.5B running locally on an iPhone with about 2.2 GB of memory and speeds up to 1.28× real time. The model has been uploaded to the audio.cpp Hugging Face repository, with an xcframework and code branch planned after audio.cpp 0.6.
DeepSpeed releases a PyTorch-identical backward API for efficient training of multimodal and multi-component models, including non-scalar backward calls. It also introduces low-precision model optimizations to enhance memory efficiency during training.
Reddit post hypes TurboQuant as lossless tech making memory obsolete for local LLMs. Poster sarcastically urges selling all memory hardware. Aims to perpetuate hype narrative to lower memory prices.