Paper

Auditing LLM Benchmarks with Item Response Theory

Unreadunread

AI summary

arXiv:2605. 30504v1 Announce Type: new Abstract: LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all.

Reading status

Priority

About this paper

Authors: Sander Land, Daniel M. Bikel
Released: 2026-06-01 (2mo ago)
Topic: Related to current project
Relation to my work: Read next because Auditing LLM Benchmarks with Item Response Theory overlaps with clean result "Leakage rate is a usable signal for recovering trigger-shaped phrases on Gaperon-1125-1B without knowing the hidden trigger itself (MODERATE confidence)", clean result "Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives the model was never trained on, absent under same-language SFT (LOW confidence)", clean result "Training one persona to emit a [ZLT] marker without bystanders adopting it has a one-cell-wide LR x epochs window on Qwen2.5-7B-Instruct (LOW confidence)". Matching terms: class, source, without, propagate, model. Source: arxiv cs.CL (NLP).
arXiv: 2605.30504

Ask Claude about this paper

Posts a comment with @claude so the runner spins up a Claude Code session scoped to Auditing LLM Benchmarks with Item Response Theory. Replies appear in the conversation below.

Comments

No comments yet.

Related records

No links yet.