Morning Edition · Tuesday, June 23, 2026Published at 6:45 AM EDT · New York

Study Finds Transformers' 'Massive Activations' Are Robust, Not a Removable Artifact

A controlled residual-stream experiment argues that the giant outlier activations trained transformers develop are an architectural feature, with implications for quantization and interpretability.

A new preprint examines massive activations, the small number of hidden dimensions whose magnitude sits far above the median and which concentrate on the sequence-start token. The question the paper sets up is whether these outliers are a r…