TorchAO Extends QAT for Edge LLMs

🔑 Enhanced Key Takeaways

• •TorchAO's QAT supports int8 dynamic per-token activations + int4 grouped per-channel weights (8da4w) for linear layers, motivated by kernel availability on edge backends and optimal LLM quality[4][6].
• •QAT in TorchAO recovers 96% of accuracy degradation on Hellaswag and 68% of perplexity degradation on WikiText for Llama3 compared to PTQ[4][6].
• •QAT workflow uses prepare step to insert fake quantization ops into linear layers during training, followed by convert step to replace with actual quantize/dequantize ops for inference[4][6].

🛠️ Technical Deep Dive

• •QATConfig with Int8DynamicActivationInt4WeightConfig (group_size=32) applied via torchao.quantization.quantize_ in two steps: 'prepare' inserts fake quantize ops, 'convert' finalizes to actual quantized layers[6].
• •Supports combination with LoRA for 1.89x faster training compared to vanilla QAT[6].
• •Fake quantization simulates rounding to low-bit values (e.g., staying in bfloat16) during training, enabling model adaptation to quantization constraints[1][3].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline