GISTBench: LLM User Understanding Benchmark

🔑 Enhanced Key Takeaways

• •GISTBench addresses the 'long-context forgetting' phenomenon in recommendation systems, where LLMs struggle to maintain accurate user interest profiles over extended interaction sequences.
• •The benchmark specifically targets the 'attribution gap' in multi-modal recommendation, where models fail to distinguish between passive consumption (e.g., scrolling) and active engagement (e.g., liking/sharing) when generating interest summaries.
• •The synthetic dataset utilizes a proprietary 'User-Interest-Graph' (UIG) generation framework, allowing researchers to control the ground-truth interest distribution to measure model bias against niche vs. mainstream interests.

🛠️ Technical Deep Dive

• •Interest Groundedness (IG) is calculated as the harmonic mean of 'Interest Precision' (ratio of extracted interests present in the ground-truth interaction history) and 'Interest Recall' (coverage of the ground-truth interest set).
• •Interest Specificity (IS) utilizes a hierarchical taxonomy mapping (e.g., 'Sports' -> 'Basketball' -> 'NBA') to penalize models that provide overly generic interest summaries.
• •The evaluation pipeline employs a 'Chain-of-Thought' (CoT) prompting strategy to force models to cite specific interaction timestamps before finalizing interest labels, reducing hallucination rates.
• •The dataset includes 'noise injection' scenarios where irrelevant interaction history (e.g., accidental clicks) is mixed with meaningful signals to test model robustness.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline