Interpretability Fails to Fix LM Errors

🔑 Enhanced Key Takeaways

• •The study highlights a 'representation-behavioral decoupling' where internal activations contain sufficient information for correct classification, yet the model's final logit layer fails to map these representations to accurate output tokens.
• •The failure of Sparse Autoencoders (SAEs) to influence output suggests that current interpretability techniques may be capturing 'superficial' features that are not causally linked to the model's final decision-making process.
• •The research suggests that clinical triage errors are not due to a lack of 'knowledge' within the model's weights, but rather a failure in the model's internal reasoning or attention-routing mechanisms during the generation phase.

🛠️ Technical Deep Dive

• •Methodology utilized four distinct mechanistic interpretability interventions: Concept Steering, Sparse Autoencoder (SAE) activation patching, Token-Specific Vector (TSV) manipulation, and logit lens analysis.
• •The clinical triage dataset consisted of high-stakes, multi-turn diagnostic vignettes designed to test sensitivity to critical medical red flags.
• •The 'internal discrimination' metric was calculated by training a linear probe on the model's residual stream at the final layer, confirming that the model 'knows' the correct triage category internally.
• •The 'output sensitivity' metric measured the model's ability to correctly identify high-acuity cases in a zero-shot generation setting, revealing a significant drop-off compared to the probe performance.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline