๐ArXiv AIโขStalecollected in 17h
OmniToM: Benchmarking Theory of Mind in LLMs
๐กDiscover why current LLMs fail at social reasoning and how to benchmark their ability to track complex mental states.
โก 30-Second TL;DR
What Changed
Introduces a two-stage evaluation process: Belief Extraction and Belief Labeling.
Why It Matters
This research shifts the focus of ToM evaluation from simple output accuracy to the underlying reasoning process. It provides a framework for developers to diagnose why models fail to understand complex social dynamics.
What To Do Next
Incorporate the OmniToM evaluation pipeline into your model testing suite to identify weaknesses in social reasoning and belief tracking.
Who should care:Researchers & Academics
Key Points
- โขIntroduces a two-stage evaluation process: Belief Extraction and Belief Labeling.
- โขUtilizes a seven-dimensional schema to analyze knowledge, intentions, and false beliefs.
- โขIdentifies a significant 'belief-tracking bottleneck' in current LLMs during zero-shot evaluation.
- โขBuilt on 895 stories with 22,343 labeled belief propositions for robust testing.
๐ง Deep Insight
AI-generated analysis for this event.
๐ Enhanced Key Takeaways
- โขA critical distinction is emerging between 'literal theory of mind' (predicting others' behavior) and 'functional theory of mind' (adapting to others in context), with many current benchmarks primarily measuring the former and struggling with the latter. [1, 3]
- โขSome researchers contend that existing Theory of Mind (ToM) benchmarks for LLMs are flawed because they are often adapted from human psychological tests and may not accurately assess an LLM's ability to adapt to new partners or may be solvable without explicit human-like reasoning. [1, 3, 6, 8, 17]
- โขThe performance of LLMs on ToM tasks is significantly influenced by factors such as model scale and prompt design, with larger models like GPT-4 demonstrating near-human or even adult-level capabilities on certain higher-order ToM inferences. [2, 4, 5]
- โขThe field is seeing the development of multimodal ToM benchmarks, such as MOMENTS, which utilize realistic, narrative-rich scenarios from short films to evaluate LLMs' ability to interpret and reason across visual, acoustic, and textual inputs. [9, 17]
๐ Competitor Analysisโธ Show
| Benchmark Name | Key Features |
|---|---|
| ToMBench [1, 6, 15] | Systematic, automated, and bilingual benchmark with 2,860 testing samples, covering 8 tasks and 31 abilities. Focuses on a broad taxonomy of social and pragmatic ToM tasks (e.g., faux-pas detection, persuasion, hidden emotions, desires). |
| MOMENTS (Multimodal Mental States) [1, 9, 17] | Comprehensive multimodal benchmark with over 2,300 multiple-choice questions derived from short films. Designed to assess ToM capabilities through realistic, narrative-rich scenarios, emphasizing multimodal integration. |
| MoToMQA (Multi-Order Theory of Mind Q&A) [2, 5] | A handwritten test suite that compares LLM performance to adult human benchmarks on higher-order ToM tasks, specifically evaluating recursive mental state reasoning (e.g., up to 6th-order inferences). |
| 'Theory of Mind Benchmarks are Broken' (Conceptual) [1, 3] | A position paper arguing that most current ToM benchmarks are inadequate as they primarily measure 'literal theory of mind' (predicting behavior) rather than 'functional theory of mind' (adapting to partners), advocating for interactive evaluation components. |
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Future LLM development will necessitate interactive and user-centered ToM benchmarks to accurately assess social intelligence.
Current benchmarks are criticized for measuring only 'literal ToM' and not how LLMs adapt in dynamic social interactions, indicating a need for more interactive and user-centric evaluation methods. [1, 3, 6, 8]
The advancement of socially intelligent AI agents will increasingly rely on multimodal Theory of Mind capabilities.
Humans utilize more than just language to express mental states, and emerging benchmarks are designed to evaluate LLMs' ability to interpret and reason across visual, acoustic, and textual inputs for a more comprehensive ToM. [9, 12, 17]
โณ Timeline
1978
Premack and Woodruff introduce the concept of Theory of Mind in research on chimpanzee social intelligence.
1983
Wimmer and Perner develop the false-belief task, a foundational method for evaluating Theory of Mind in humans.
2023-03
ChatGPT-3.5-turbo achieves 20% accuracy on false-belief tasks, performing below the average of 3-year-old children.
2023-06
ChatGPT-4 demonstrates a significant improvement, solving 75% of false-belief tasks, comparable to the performance of 6-year-old children.
2024-02
ToMBench, a systematic and bilingual Theory of Mind benchmark for LLMs, is introduced.
2024-12
A position paper is published arguing that many existing Theory of Mind benchmarks for LLMs are 'broken' and advocates for evaluating 'functional theory of mind'.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: ArXiv AI โ