SPEEDRUNBENCH: Challenging LLM Agents With Video Game Speedrunning
By Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike, Abdelrahman Madkour, Varun Prashant Gangal, Monty Bichouna, Martyna Markiewicz, Shivani Jain, Duncan Curtis, Rebecca Qian, Anand Kannappan
SPEEDRUNBENCH is a new benchmark that tests LLM agents on finding faster routes across nine video games using either screenshot inputs or direct action-trace edits, with both scratch-start and supplied-route conditions. In a controlled SuperTux experiment, DeepSeek-V4-Pro (released April 2026) shrank a route from 9,428 to 1,847 frames, but no agent matched human records within practical compute budgets. The work highlights a gap between raw agent capability and human-level route optimization in long-horizon planning.