News MarkTechPost May 14
Nous Research Releases Token Superposition Training to Speed Up LLM Pre-Training by Up to 2.5x Across 270M to 10B Parameter Models
By Asif Razzaq
40 score
AI Analysis
Nous Research released Token Superposition Training (TST), a method that speeds up LLM pre-training by up to 2.5x without modifying model architecture, optimizer, or data. At the 10B-parameter MoE scale, TST achieved lower final training loss while using only 4,768 B200-GPU-hours versus 12,311 for the baseline.
Pre-training large language models is expensive enough that even modest efficiency improvements can translate into meaningful cost and time savings. Nous Research is releasing Token Superposition Training (TST), a method that substantially reduces pre-training wall-clock time at fixed compute without touching the model architecture, optimizer, tokenizer, parallelism strategy, or training data.
At the 10B-A1B mixture-of-experts scale, TST reaches a lower final training loss than a matched-FLO
training efficiencyopen source researchLLM infrastructure