Tinylora: LoRA Works with Just 13 Parameters

🔑 Enhanced Key Takeaways

• •Tinylora utilizes a novel 'sparse-projection' initialization strategy that prevents gradient vanishing during the training of ultra-low-rank adapters, a common failure mode in traditional LoRA when rank is reduced to near-zero.
• •The methodology relies on a specific weight-tying mechanism across the adapter matrices, which allows the 13-parameter constraint to remain effective by forcing the model to learn a singular, highly compressed directional update rather than a full rank-decomposition.
• •Empirical analysis indicates that while Tinylora excels at stylistic and formatting shifts, it exhibits a 'catastrophic forgetting' threshold significantly lower than standard LoRA when applied to multi-task instruction following, limiting its use to single-behavior injection.

🛠️ Technical Deep Dive

• •Adapter Architecture: Employs a rank-1 decomposition where the projection matrices A and B are constrained to a single scalar weight per layer, effectively reducing the parameter count to the number of target layers.
• •Initialization: Uses a zero-centered Gaussian initialization with a scaling factor inversely proportional to the square root of the hidden dimension, specifically tuned for the 13-parameter constraint.
• •Optimization: Utilizes a modified AdamW optimizer with a significantly higher learning rate (1e-2) compared to standard LoRA, necessitated by the extremely low parameter count to ensure sufficient signal propagation.
• •Targeting: The implementation targets the Query (Q) and Value (V) projection matrices in the attention blocks, and the gate projection in the MLP blocks, totaling 26 parameters when both are active.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline