Feature Superposition Explains Emergent Misalignment

🔑 Enhanced Key Takeaways

• •The research utilizes a novel 'feature-interference metric' to quantify how specific fine-tuning datasets activate overlapping neurons in the residual stream, providing a causal link between superposition and model drift.
• •The study identifies that 'polysemantic neurons'—neurons representing multiple unrelated concepts—are the primary vectors for emergent misalignment, as they act as bridges between benign and harmful feature clusters.
• •The geometry-aware filtering method specifically targets the 'interference-prone' subset of SAE features, demonstrating that removing only 5% of these high-interference features yields the majority of the 34.5% misalignment reduction.

🛠️ Technical Deep Dive

• •Architecture: Utilizes Sparse Autoencoders (SAEs) with a L1-regularization penalty applied to the hidden layer activations to force sparsity and disentangle features.
• •Interference Metric: Defined as the cosine similarity between the activation vectors of harmful features and the gradient updates during fine-tuning, normalized by the feature's sparsity coefficient.
• •Filtering Mechanism: Employs a 'Geometric Projection Filter' that masks the activation of features whose vectors lie within a threshold distance (determined by the manifold curvature) of the harmful feature cluster in the latent space.
• •Evaluation Framework: Uses the 'Harmful Feature Probing' (HFP) benchmark, which measures the activation magnitude of known harmful latent features when the model is prompted with benign, domain-specific inputs.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline