Refined Counterfactual Prisoner's Dilemma

🔑 Enhanced Key Takeaways

• •Scott Garrabrant's critique of expected utility theory identifies a hidden assumption in Von Neumann-Morgenstern axiomatization: the implicit requirement that agents update beliefs branch-by-branch, which formally encodes the independence axiom and creates vulnerabilities to counterfactual reasoning attacks[1].
• •Ergodicity economics, developed by Ole Peters, offers an alternative framework that derives utility functions from the dynamics of stochastic processes rather than postulating ad hoc utility functions, producing logarithmic utility for multiplicative dynamics and linear utility for additive dynamics[1].
• •The independence axiom violation debate extends beyond decision theory into AI alignment: recent work argues that expected utility theory is both unnecessary and insufficient for rational agency, and AI systems can be designed with locally coherent preferences not representable as utility functions[6].
• •Updatelessness versus updatefulness presents a genuine trade-off in decision theory: agents can strategically choose updatelessness in some decision problems to gain coherence benefits while remaining updateful in others to capture value-of-information gains[2].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline