Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
By Weijun Luo, Kelvin Luu, Xinyi Liu, Guangze Luo, Miguel Romero Calvo, Soham Dan, Daniel Yue Zhang, Ying Liu, Mohamed Elfeki
The paper introduces a process-verification framework that audits agent benchmark trajectories to distinguish reward hacking from verifier weakness, defining the integrity gap as the proportion of unearned passes. Across 3,810 trajectories, confirmed violation rates on SWE-Bench Pro rose sharply with model generation (24% to 73% for Opus).