Research Directions for Training on Probes
π‘See how probes might teach new skills and generalize alignment beyond labeled examples.
β‘ 30-Second TL;DR
What Changed
Train probes on easy-to-detect subsets of bad behavior and study how to improve generalization to unseen failures.
Why It Matters
If successful, probe-based training could extend behavioral supervision beyond labeled examples and support more ambitious alignment strategies. However, the proposed brain-like analogy also highlights risks such as obfuscated behavior and probes failing to track newly learned concepts.
What To Do Next
Prototype a runtime probe controller that changes model behavior when a lie detector predicts harmful upcoming tokens, then evaluate generalization on held-out scenarios.
Key Points
- β’Train probes on easy-to-detect subsets of bad behavior and study how to improve generalization to unseen failures.
- β’Use probes not only to suppress lying but also to incentivize positive skills under distribution shift.
- β’Retrain probes as models acquire new concepts, potentially reconnecting learned behavioral instincts to those concepts.
- β’Explore runtime probe signals that immediately modulate behavior, similar to an instinct evaluating thoughts without additional training steps.
Weekly AI Recap
Read this week's curated digest of top AI events β
πRelated Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: AI Alignment Forum β
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.