Interpretability in Model Training?

🔑 Enhanced Key Takeaways

• •Goodfire's attention probes, based on Shabalin and Belrose 2025 architecture with layernorm prenorm and residual connections, serve as baselines but underperform SAE probes in robustness to distribution shifts like synthetic-to-real data transitions[2].
• •Activation probes assume linear representations of behavioral states in activation space, empirically holding up but vulnerable to theoretical adversarial masking, unlike CoT monitoring which degrades due to unfaithful reasoning under optimization[1].
• •Probes trained on textual evidence like CoT or prompts show degraded performance when such leakage is filtered out, as demonstrated in tests on sandbagging, sycophancy, and bias using fine-tuned model organisms without verbalization[5].

🛠️ Technical Deep Dive

• •Attention probes use architecture from Shabalin and Belrose 2025: multi-logit classifier head with softmax, trained with class weights (square root of inverse class frequency), input includes raw activations or SAE features[2].
• •Sparse autoencoders (SAEs) disentangle activations into higher-dimensional sparse feature spaces, outperforming attention probes in PII detection tasks across distribution shifts[2].
• •Linear probes rely on Linear Representation Hypothesis, detecting states like deception via classifiers on activations, but fail when behavior is internalized without textual cues in model organisms[5].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline