Recurrence Rewrites the Transformer Global Workspace
π‘Recurrence may preserve a workspaceβbut radically change how you can inspect or steer it.
β‘ 30-Second TL;DR
What Changed
Ouro-2.6B reconstructs workspace content in every loop, preventing linear transport across loop boundaries.
Why It Matters
The findings suggest that interpretability and activation steering techniques designed for untied transformers may fail on recurrent architectures. Practitioners building latent-reasoning or weight-sharing models should treat recurrence boundaries and local causal windows as first-class design constraints.
What To Do Next
When probing a recurrent model, implement virtual unrolling and test activation reads, writes, and ablations across both individual iterations and sliding windows of two or more recurrences.
Key Points
- β’Ouro-2.6B reconstructs workspace content in every loop, preventing linear transport across loop boundaries.
- β’For Ouro, writes and ablations must cover every remaining loop to have an effect.
- β’Huginn-0125 carries content through all 16 recurrences, but reads, writes, and ablations operate within a roughly two-iteration sliding window.
- β’Verbalisation of newly injected content follows explicit per-iteration supervision, while steering existing content does not.
Weekly AI Recap
Read this week's curated digest of top AI events β
πRelated Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: ArXiv AI β
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.