Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 chang...
By @karpathy
Building on yesterday's Reddit discussion of autoresearch, Karpathy's major post: autoresearch agent autonomously found ~20 improvements to nanochat over 2 days, reducing 'Time to GPT-2' by 11%. Improvements included fixing attention scaling, regularization, attention bandwidth, AdamW betas, weight decay, and initialization. He predicts all frontier labs will adopt agent-driven optimization and envisions multi-agent collaboration for research at scale.