VLMs Excel on MCQs but Fail Open Long Video Reasoning

🔑 Enhanced Key Takeaways

• •Research indicates that the 'multiple-choice bias' in video benchmarks is largely driven by the model's ability to leverage language-only priors or superficial visual cues rather than temporal understanding, a phenomenon often termed 'shortcut learning'.
• •Recent studies suggest that current VLM architectures struggle with long-context video because they rely on frame-sampling strategies (e.g., uniform or sparse sampling) that discard critical temporal transitions necessary for multi-step reasoning.
• •The industry is shifting toward 'Video-Language-Action' (VLA) models and embodied AI benchmarks to move beyond static MCQ evaluation, aiming to force models to demonstrate reasoning through interactive or generative tasks rather than selection.

🛠️ Technical Deep Dive

• •Current VLM architectures for long video typically utilize a Vision Encoder (e.g., CLIP-ViT) combined with a Large Language Model (LLM) via a projection layer (Q-Former or MLP).
• •Long-video processing often employs 'token compression' or 'temporal pooling' to fit high-frame-count inputs into the LLM's context window, which inherently loses fine-grained temporal resolution.
• •The 'MCQ bias' is exacerbated by the use of contrastive loss functions during pre-training, which prioritize distinguishing between provided options rather than generating open-ended, temporally grounded descriptions.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline