Tag: #psychometrics7 results
This research proposes a new paradigm for evaluating AI models that have surpassed human capability by using relative measurement instead of static benchmarks. The framework utilizes model-generated challenges to create an adversarial psychometric rating system that scales alongside agent capabilities.
Researchers developed a psychometric framework to assess generative AI cognitive profiles against human norms, revealing top performance in verbal and working memory but failures in perceptual reasoning. The AIQ Benchmark tracks evolution across six generations and two model families, showing asymmetric gains with language bias over visual tasks. This indicates scaling alone may not achieve balanced AGI.
Researchers developed the GenAI-RTS, a 20-item psychometrically validated instrument to categorize how undergraduate students rely on generative AI. The scale identifies four distinct reliance types: Strategic, Instrumental, Dependent, and Dialogic, providing a framework for future AI literacy interventions.
This position paper argues that item-level benchmark data is essential for a rigorous science of AI evaluation, addressing systemic validity failures in current paradigms. It draws from psychometrics and computer science to advocate for granular diagnostics. OpenEval is introduced as a repository to support evidence-centered AI evaluation.
Researchers used AI Psychometrics and TAM to evaluate psychological reasoning in GPT-3.5, GPT-4, LLaMA-2, and LLaMA-3. All models met validity criteria, with GPT-4 and LLaMA-3 showing superior performance.
This paper integrates psychometric rater models into AI evaluations to correct systematic errors from human raters. It employs Item Response Theory, particularly the multi-faceted Rasch model, to disentangle true output quality from rater effects like severity and centrality. Applied to OpenAI's summarization dataset, it delivers adjusted quality scores and rater diagnostics for more reliable AI assessments.
Researchers used GPT-4.1-nano to predict difficulty of data visualization literacy test items via text, images, or both. Multimodal approach achieved lowest MAE of 0.224, outperforming text-only (0.338) and vision-only (0.282). Held-out test showed MSE of 0.108, proving LLMs' psychometric potential.