CreativityBench: LLM Creative Tool Benchmark

🔑 Enhanced Key Takeaways

• •CreativityBench utilizes a hierarchical taxonomy of physical objects, mapping specific sub-components to their functional affordances to test cross-domain reasoning.
• •The benchmark specifically targets the 'functional fixedness' bias in LLMs, where models struggle to decouple an object's primary utility from its potential secondary uses.
• •Evaluation metrics include a novel 'repurposing success rate' that penalizes models for hallucinating non-existent physical mechanisms or impossible material interactions.

🛠️ Technical Deep Dive

• •Knowledge Base Structure: Employs a graph-based representation where nodes are physical entities and edges represent affordance relationships (e.g., 'can-be-used-as', 'has-part').
• •Task Generation: Utilizes a constrained generation pipeline that forces models to propose solutions within a defined set of physical laws and material properties.
• •Evaluation Framework: Implements a multi-stage verification process involving both automated semantic similarity checks against ground-truth affordances and a secondary LLM-based 'physical feasibility' judge.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline