Detect and handle massive activations

[spikymoth]

I've been trying to figure out why the winsorization added in #52 actually helps, and I think I've worked it out: The winsorization indirectly mitigates the influence of ablation on dimensions affected by so-called massive activations.

Massive activations are a phenomenon where specific tokens, sometimes in a position dependent way, trigger massive activations in specific dimensions: https://github.com/locuslab/massive-activations

These activations aren't context sensitive (they have similar values regardless), tend to crop up suddenly in early to middle layers, and tend to persist through several layers before eventually being attenuated.

Due to their magnitude they can bias the mean and end up dominating the ablation direction. More importantly however, ablating them can be catastrophic for the model's performance!

Winsorization mitigates this by setting the magnitude of any massive activations to something more normal. This reduces their impact on the mean direction, but also reduces the impact of ablation on those "special" dimensions.

However, I think a better and more direct way to deal with them is to simply zero out those dimensions in the directions used for ablation (across all layers), so they are left completely untouched.

The more difficult part is actually detecting the affected dimensions. There's no real way to infer what tokens might trigger massive activations, so the only way to find them is to examine the hidden state for each token across different prompts.

Detection is heuristic: If a dimension of the hidden state (for a particular token in a particular layer) has abs_value > 100 and abs_value > 1000 * abs_layer_median then it's considered a massive activation. Since a characteristic of massive activations is that they tend to persist for several layers, the detection can be made more robust by checking that large activations are also seen in the next ~4 layers, e.g. with a slightly less strict criterion like abs_value > 200 * abs_layer_median to account for attenuation.

I've implemented this locally and it seems to help a lot: For Gemma-3-12b-it, dimension 2339 exhibits a lot of massive activations, and simply zeroing it out in the directions leads to similar improvements in refusals / KL divergence as what I've observed with winsorization (though I don't have enough data to confidently say that it's a complete replacement). The full detection identifies a further 3 massive activation dimensions for this model. Qwen3-4B-Instruct-2507 has two at dimensions 4 and 396, which trigger for a token from its chat template.

Detection can be integrated into get_residuals using per-layer hooks, since we already have to set output_hidden_states=True anyway. But a small dedicated detection with a prompt containing the most common trigger tokens might also be a good idea (since regular prompts may not contain all of the most common triggers). I've currently implemented a combination of both, and I think it's reasonably clean.

Would you be interested in a PR?

[p-e-w]

Interesting! IIRC, the article introducing projected abliteration blames the problem on specific activation functions, but it appears that such outliers are common across different model architectures.

Which immediately raises the question: Why is winsorization so effective specifically for the Gemma 3 series, if massive activations can be found elsewhere as well?

Some more discussion points:

However, I think a better and more direct way to deal with them is to simply zero out those dimensions in the directions used for ablation (across all layers), so they are left completely untouched.

It's not immediately clear to me whether this is indeed better from a theoretical perspective. If we assume that refusals are controlled by a linear subspace, then zeroing out high-activation dimensions rotates that subspace. I get that those activations distort the result, but they still contain information, and pretending that they didn't happen also distorts the result. In a way, winsorization (or other approaches that dampen, rather than eliminate, high activations) seems more principled as it at least preserves the information that those coefficients were larger than all others. Your proposal inverts that relationship, and turns the largest-magnitude coefficients into the smallest ones.

Since a characteristic of massive activations is that they tend to persist for several layers, the detection can be made more robust by checking that large activations are also seen in the next ~4 layers, e.g. with a slightly less strict criterion like abs_value > 200 * abs_layer_median to account for attenuation.

If our goal is to reduce distortion from activation outliers, I don't see why it matters whether those outliers are "massive activations" in the sense proposed by the linked paper. Any high-magnitude coefficients will force cosine alignment in the affected layer, regardless of whether the phenomenon persists over the next few layers for the same dimension. So I'm not sure if that heuristic makes sense for what we are trying to accomplish.

But a small dedicated detection with a prompt containing the most common trigger tokens might also be a good idea

Aren't those trigger tokens completely model-dependent? The paper certainly appears to suggest so. Why would we expect any commonality here?

The way I see it, the basic dilemma is this: Massive activations distort the refusal direction, but removing them also distorts the refusal direction from what it presumably "really" is.

So here's an alternative idea: What about completely discarding the residuals from prompts that exhibit massive activations? From the paper, it appears that massive activations don't occur at each token position. We can assume that prompts that don't give rise to massive activations at the last position are a better representation of the "true" refusal direction. The original abliteration paper suggested that as few as 30-40 prompts can be sufficient to calculate the direction, so we have a lot of wiggle room here. We could discard the 50% of residuals with the largest-magnitude activations before computing the means.

I'm definitely interested in a PR exploring this, and I'm happy to ship experimental features if they are gated behind a flag. That being said, I don't think this should hold back #52 in any way.

[p-e-w]

What about completely discarding the residuals from prompts that exhibit massive activations?

The more I think about this, the more I believe that this is actually a very promising approach that we should probably be doing (in some form) for all models by default.

If the goal is to determine the "typical" refusal direction, outliers should be discarded before computing the means. This could be done more generally than by activation magnitude, e.g. using clustering algorithms. We detect the largest cluster, compute its mean, and discard everything else. This should be more robust than what we are doing right now.

PaCMAP shows that outliers are present even in layers that already clearly distinguish between harmful and harmless prompts, and those can't be doing us any good.

[spikymoth]

However, I think a better and more direct way to deal with them is to simply zero out those dimensions in the directions used for ablation (across all layers), so they are left completely untouched.

It's not immediately clear to me whether this is indeed better from a theoretical perspective. If we assume that refusals are controlled by a linear subspace, then zeroing out high-activation dimensions rotates that subspace. I get that those activations distort the result, but they still contain information, and pretending that they didn't happen also distorts the result. In a way, winsorization (or other approaches that dampen, rather than eliminate, high activations) seems more principled as it at least preserves the information that those coefficients were larger than all others. Your proposal inverts that relationship, and turns the largest-magnitude coefficients into the smallest ones.

Well, I think the point is that these massive activations in some sense "anchor" the model. They have a context-agnostic (and very high) value, but they tend to have more of a syntactic or low-semantic meaning, appearing for e.g. the <bos> (always at position 0), the first \n, or function words like and or of.

So unless we specifically detect them on refusal marker tokens (which wouldn't match their usual form, at least for single words like "sorry"), I wouldn't expect them to be informative about the refusal direction.

The reason for zeroing them out in the refusal direction is simply to keep ablation from impacting the characteristic values produced in these dimensions, to keep the model from losing its anchor points and (potentially) becoming incoherent.

Aren't those trigger tokens completely model-dependent? The paper certainly appears to suggest so. Why would we expect any commonality here?

While they're model-dependent, some tokens like <bos>, . and \n are observed in many models. Larger models tend add more low-semantic ones like function words. And while the research doesn't explore this (being focused on base models), I think instruction-trained models might add tokens corresponding to their templates, to keep them focused and able to follow instructions. That would explain what I observed with Qwen3-4B-Instruct-2507.

So here's an alternative idea: What about completely discarding the residuals from prompts that exhibit massive activations? From the paper, it appears that massive activations don't occur at each token position. We can assume that prompts that don't give rise to massive activations at the last position are a better representation of the "true" refusal direction.

This could be worth a try, assuming we're talking about the residuals of the 1st (and only) token generated by the model in response to each prompt. If we look at every token in each prompt, I think we're likely to discard everything. I'll think about how to integrate this.

I'm definitely interested in a PR exploring this, and I'm happy to ship experimental features if they are gated behind a flag. That being said, I don't think this should hold back #52 in any way.

Understood.

[spikymoth]

This could be worth a try, assuming we're talking about the residuals of the 1st (and only) token generated by the model in response to each prompt. If we look at every token in each prompt, I think we're likely to discard everything. I'll think about how to integrate this.

Er, I'm realizing that I was a bit confused about what the residuals correspond to. We're talking about the residuals after processing the prompt, which are used to generate the singular token. So we'd be discarding residuals where a massive activation shows up for the last token in the prompt.

[spikymoth]

The plot thickens:

• For Qwen3-4B-Instruct-2507, the final token never contains a massive activation
• For gemma-3-12b-it, the final token always contains a massive activation

The problem with gemma-3-12b-it is that dimension 2339 appears to be essentially broken, with massive values for almost every token in early layers - for example across 512 prompts in the "harmless" set, for 28160/28672 tokens dimension 2339 in layer 16 met the criteria for a massive activation!

With those characteristics it might not even properly qualify as a massive activation - but I can certainly see it skewing the direction. In my experiments with this dimension zeroed out, refusals were able to drop just fine and kl divergence was lower at a similar number of refusals, like when enabling winsorization.

So filtering out residuals with massive activations might still make sense, but we'll have to deal with anomalous cases like this - perhaps by filtering out massive activations that trigger too frequently.

[p-e-w]

It's strange that Gemma 3 12B still performs so well with such an extreme artifact. The loss of dynamic range alone from one dimension dwarfing the others should cause issues even without abliteration.

Gemma 3 is by far the most important model for testing because unlike all others I have tried, it has a recurring, clearly recognizable artifact when abliterated: Corrupted contractions (like it' or they' m) in the responses. This occurs with every single abliteration I've seen. Eliminating this highly visible problem is the litmus test for whether we are understanding what is really going on.

[spikymoth]

What might be happening, although I'd need to confirm this, is that the huge values in the "broken" dimension are mostly cancelling out between the good and bad directions. The pre-normalization and winsorization might be further limiting the effect (pre-normalizing by not assigning high norm dimensions extra weight, and winsorization by clamping the activation magnitudes). Anecdotally, in my low-KL (<0.05) abliterations based on #52 I haven't seen problems with contractions.

I added some logic to exclude dimensions that meet the massive activation criteria for over 10% of all tokens (a very generous limit) and that "fixed" the bad residual purge for Gemma 3 12B. Expanding the number of good prompts to 5000, I saw 3 prompts with a massive activation in the residuals at the end of prompt processing. So that's much more reasonable, although it also probably means that removing them won't make much of a difference.

To handle this properly I think I'll need to experiment a bit more with the criteria. (abs_value > 100) & (abs_value > 1000 * abs_median) is nice and simple, but it completely ignores the dispersion within the layer. Adding a criterion based on the MAD (median absolute deviation) might make sense, but I'd have to test it carefully.

There's also still a question of how much general massive activation detection actually helps. Clearly removing Gemma 3 12B's dimension 2339 makes a big difference, but that neuron doesn't really behave like "normal" massive activations so removing it is more like a side effect.

[spikymoth]

Still experimenting with different ways of detecting them; might be that simple is best. I also found A Refined Analysis of Massive Activations in LLMs (associated repository) which gives some additional details and some nice plots including for Gemma-3-12b-pt.