COCONUT Latent Reasoning Debunked by Controls

🔑 Enhanced Key Takeaways

• •COCONUT enables breadth-first search in latent space by encoding multiple alternative reasoning paths in continuous thoughts, outperforming CoT on tasks requiring backtracking like ProsQA[1][2].
• •Without multi-stage curriculum training, COCONUT models fail to learn effective latent reasoning and perform no better than baselines without CoT[2][3][5].
• •COCONUT exhibits training instability and compute inefficiency, with pretraining costs growing exponentially with thinking tokens and adaptation challenges to hidden state inputs[1][5].

🛠️ Technical Deep Dive

• •COCONUT uses the last hidden state of a pretrained GPT-2 as 'continuous thought', fed directly back as input embedding without token decoding, applied iteratively[1][3].
• •Multi-stage curriculum progressively increases continuous thoughts (k=1 to higher), mixing prior stage data (0.3 probability) to prevent forgetting, with cross-entropy loss on remaining tokens[2][3][4].
• •At inference, increasing thoughts per step (up to 2-6 on ProsQA) improves accuracy via search tree exploration, prioritizing promising paths and pruning others based on implicit probabilities[2][4].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline