
Fixing Rater Bias in AI Evals with IRT
This paper integrates psychometric rater models into AI evaluations to correct systematic errors from human raters. It employs Item Response Theory, particularly the multi-faceted Rasch model, to disentangle true output quality from rater effects like severity and centrality. Applied to OpenAI's summarization dataset, it delivers adjusted quality scores and rater diagnostics for more reliable AI assessments.






