StoryTR: ToM for Narrative Video Retrieval

🔑 Enhanced Key Takeaways

• •StoryTR utilizes a novel 'Theory of Mind' (ToM) annotation schema that explicitly maps character mental states—such as false beliefs and hidden intentions—to specific temporal segments in short-form video content.
• •The Agentic Data Pipeline employed for the 7B Shorts-Moment model utilizes a multi-agent framework where one agent acts as a 'Narrative Critic' to filter out low-quality ToM chains before training, significantly reducing noise in the synthetic dataset.
• •The benchmark specifically addresses the 'temporal grounding gap' in current multimodal LLMs, where models often identify the correct narrative event but fail to precisely localize the start and end points due to a lack of causal understanding of character actions.

🛠️ Technical Deep Dive

• •Model Architecture: Employs a lightweight temporal-adapter layer integrated into a 7B parameter backbone, specifically optimized for long-range dependency modeling in short-form video.
• •ToM Chain Generation: Uses a recursive prompting strategy where the Agentic Pipeline generates a 'Mental State Map' (MSM) for each character before performing temporal localization.
• •Training Objective: Implements a dual-loss function combining standard IoU regression with a 'Causal Consistency Loss' that penalizes the model if the predicted moment contradicts the inferred character intent.
• •Data Pipeline: The pipeline leverages a frozen, high-capacity teacher model to distill reasoning chains into the 7B student model, focusing on high-entropy narrative segments.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline