Researchers pinpoint why larger language models pick up skills that small ones miss
By Jonathan Kemper
A new study using models from 4 million to 4 billion parameters explains why small models fail at rare tasks: frequent tasks overwrite learned skills. The researchers suggest increasing how often a target task appears in training data may be as effective as scaling up model size.