Tag: #distillation27 results
Structured distillation compresses personalized AI agent conversation histories into compact retrieval structures, achieving 11x token reduction from 371 to 38 tokens per exchange. Evaluated on 14k exchanges, it preserves 96% of verbatim recall and exceeds baselines in cross-layer search. Open-source implementation released.
PACED optimizes LLM distillation by targeting the zone of proximal development with Beta-weighted pass rates, avoiding compute waste on mastered or unreachable problems. It proves theoretical SNR optimality and minimax-robustness of the weighting. Empirical results show gains in forward KL, self-distillation, and two-stage schedules on reasoning benchmarks.
Jeff Dean predicts that in the future, each developer will manage 50 intelligent agents, making writing requirements the core skill. He revealed that model distillation was the key breakthrough for Google's Flash model.
Post-'distillation event,' Anthropic extends aggressively into finance. Company launched four updates in 48 hours, foreshadowing major industry shifts.
Anthropic accuses DeepSeek, Moonshot, MiniMax of distilling Claude via 2.4k fake accounts and 16M interactions. Article exposes hypocrisy, as distillation is industry standard and Anthropic faced copyright suits. Chinese models like Qwen 3.5 top HF leaderboard.
Anthropic uncovered three Chinese AI firms—DeepSeek, Moonshot AI, MiniMax—using distillation to extract capabilities from its Claude chatbot. It calls this industrial-scale IP theft. OpenAI issued similar accusations last month.
This discussion explores the feasibility of applying Exponential Moving Average (EMA) to LoRA adapters to act as a self-teacher. The goal is to generate soft labels for trainable adapters, similar to on-policy self-distillation techniques.
Savant Commander 48B is a custom Qwen3-based 4x12B MOE merging distills from Claude, Gemini, OpenAI, DeepSeek, and more with hand-coded routing. Users control activation via prompts and test differences easily. GGUF versions include regular and Heretic uncensored, available on Hugging Face.
Community creator released a fully uncensored 9B Qwen model by merging uncensored tensors from HauhauCS with Claude Opus reasoning distillation from Jackrong. It eliminates refusals, boosts creativity for roleplay and image prompts, with thinking disabled via custom chat template. GGUF version optimized for RTX 3060 in LM Studio.