Open-Sourced 20GB MedQA 84% LLM

🔑 Enhanced Key Takeaways

• •The model utilizes a novel 'Chain-of-Thought Distillation' technique specifically optimized for clinical reasoning, which differentiates it from standard fine-tuning approaches used in previous medical LLMs.
• •Empirisch Tech has integrated a proprietary 'Privacy-Preserving Inference Layer' that ensures zero-data leakage during local execution, specifically targeting compliance with HIPAA and GDPR requirements for on-premise healthcare deployments.
• •The 1.6x speed improvement is achieved through a custom kernel optimization for the DeepSeek-R1 architecture that reduces KV-cache memory overhead during long-context medical consultations.

🛠️ Technical Deep Dive

• •Architecture: Based on DeepSeek-R1-32B (Mixture-of-Experts/Dense hybrid).
• •Quantization: 4-bit GPTQ using AutoGPTQ framework with ActOrder enabled for improved perplexity.
• •Fine-tuning: QLoRA (Quantized Low-Rank Adaptation) applied to all linear layers with a rank (r) of 64 and alpha of 128.
• •Inference Optimization: Custom CUDA kernels for speculative decoding, enabling the reported 36.86 tok/s throughput on L40 hardware.
• •Dataset: Trained on a curated subset of PubMed, MIMIC-IV, and synthetic clinical reasoning chains generated by GPT-4o.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline