Social Twitter Jan 31
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is...
By @karpathy
95 score
AI Analysis
Karpathy announces nanochat can train GPT-2 grade LLM for ~$73 in 3 hours on single 8xH100 node - a 600X cost reduction from OpenAI's original $43K in 2019. Details Flash Attention 3, Muon optimizer, and other optimizations.
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node).
GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100.
Originally in 2019, GPT-2 was trained by OpenA
training-efficiencycost-reductionopen-source-mloptimization-techniquesscaling-laws
Source details
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node).
GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100.
Originally in 2019, GPT-2 was trained by OpenA
training-efficiencycost-reductionopen-source-mloptimization-techniquesscaling-laws