This paper introduces a theoretical framework that reimagines AI benchmarking as a multilayer, adaptive network connecting evaluation metrics, model components, and stakeholder priorities through weighted interactions. It embeds human tradeoffs using conjoint-derived utilities and a human-in-the-loop update rule, allowing benchmarks to evolve dynamically while maintaining stability. The approach generalizes traditional leaderboards and promotes context-aware, human-aligned evaluations.
Key Points
- 1.Multilayer network linking metrics, models, and stakeholders
- 2.Human-in-loop updates with conjoint utilities
- 3.Generalizes leaderboards for accountable AI evaluation
Impact Analysis
This framework could transform AI evaluation by incorporating diverse stakeholder needs, leading to more robust and fair benchmarks. It enables dynamic adaptation to real-world contexts, potentially accelerating progress in human-aligned AI systems while enhancing interpretability and accountability.
Technical Details
The formulation uses weighted interactions and an update rule to embed human tradeoffs into benchmark structures. It preserves stability and interpretability during evolution, providing tools to analyze benchmark properties. Classical leaderboards emerge as a special case.