Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
By Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan
This Anthropic interpretability work scales sparse autoencoders to extract up to 34 million interpretable, multilingual, multimodal features from the production model Claude 3 Sonnet, demonstrating dictionary learning beyond small transformers and steering capabilities. Note Claude 3 Sonnet is an older model, so this is analysis of an existing production system rather than a current release.