Summary
Mechanistic interpretability reverse-engineers a model’s internal mechanisms—its neurons, features, and circuits—unlike general XAI-style attribution that only explains which inputs mattered for an output (such as heatmaps). A central challenge in this field is polysemanticity, where a single neuron responds to several seemingly unrelated concepts. This is commonly explained by superposition, in which a model represents more sparse features than it has dimensions at the cost of interference that requires nonlinear filtering (Anthropic, Toy Models of Superposition, 2022-09-14). Researchers either untangle layer activations using sparse autoencoders (SAEs) or debate newer working hypotheses—such as Nick Cammarata’s essay (“Neural networks are clean”, Goodfire), where the author argues that much apparent polysemanticity is actually “recursive generalization” and that neurons can be studied directly.
What is mechanistic interpretability, and how does it differ from general explainable AI (XAI)?
One-line answer: While XAI-style attribution explains which inputs mattered for an output (for example, via heatmaps), mechanistic interpretability reverse-engineers the model’s internal mechanisms—its neurons, features, and circuits.
- XAI-style attribution vs. mechanistic interpretability: Attribution methods highlight which parts of the input contributed most to a given output. Mechanistic interpretability looks inside the network to reverse-engineer how individual neurons, features, and connected circuits actually operate.
- For a primer on how inputs are broken into tokens and context windows before entering a model, see our tokens and context guide.

What is polysemanticity, and why does a single neuron hold multiple concepts?
One-line answer: Polysemanticity occurs when a single neuron responds to several seemingly unrelated things, commonly explained by superposition—where a model represents more sparse features than it has dimensions at the cost of interference.
- Polysemantic vs. monosemantic: A monosemantic neuron or feature corresponds to one clear concept, whereas a polysemantic neuron responds to multiple seemingly unrelated concepts.
- Superposition and Toy Models of Superposition (Anthropic, 2022-09-14):
- Definition: Superposition happens when models “represent more features than they have dimensions.” When features are sparse, superposition enables compression “at the cost of ‘interference’ that requires nonlinear filtering.”
- Setup and caveat: Studied using “small ReLU networks trained on synthetic data with sparse input features,” the paper demonstrates that “Both monosemantic and polysemantic neurons can form,” while the authors explicitly caution that “it’s very unclear what to generalize to real networks.”
- Geometry example: When squeezing 5 features into 2 dimensions, dense features result in only the top 2 being stored, whereas sparse features allow extra features to be stored with interference. Geometrically, for example, 3 features arranged in 2D form a triangle.

actual output (hand-built illustration, not a trained model)
import numpy as np
# 3 features squeezed into 2 dimensions (triangle layout, as in Toy Models of Superposition)
ang = np.deg2rad([90, 210, 330])
W = np.stack([np.cos(ang), np.sin(ang)]) # shape (2 dims, 3 features)
print("W^T W (feature overlap):")
print(np.round(W.T @ W, 2))
x = np.array([1.0, 0.0, 0.0]) # only feature 0 is active
h = W @ x # 2-dim hidden vector
print("readout W^T h :", np.round(W.T @ h, 2))
print("after ReLU(.. -0.4):", np.round(np.maximum(W.T @ h - 0.4, 0), 2))
W^T W (feature overlap):
[[ 1. -0.5 -0.5]
[-0.5 1. -0.5]
[-0.5 -0.5 1. ]]
readout W^T h : [ 1. -0.5 -0.5]
after ReLU(.. -0.4): [0.6 0. 0. ]
In this Python 3 + numpy 2.5.3 illustration, 3 features share 2 dimensions. When only feature 0 is active (x = [1.0, 0.0, 0.0]), reading it back via W^T h leaks -0.5 into the other features (interference). Applying a ReLU with a negative bias (-0.4) filters out that interference and leaves [0.6, 0., 0.], illustrating the paper’s “nonlinear filtering” idea in miniature.
How do researchers extract entangled features from a neural network?
One-line answer: Researchers either train sparse autoencoders (SAEs) to decompose layer activations into a larger set of sparse features, or debate working hypotheses suggesting that neurons can be studied directly.
- Sparse Autoencoders (SAEs): An SAE is a second model trained to rewrite a layer’s activations as a larger set of sparse features, aiming for one feature ≈ one concept via dictionary learning (Anthropic’s 2023 Towards Monosemanticity is a well-known example).
- Recent debate and working hypothesis (Hacker News thread and Neural networks are clean): In an essay explicitly labelled “A working hypothesis”, Nick Cammarata (Goodfire) puts forward several claims:
- The essay author argues that canonical polysemantic evidence was misread, and that much apparent polysemanticity is actually “recursive generalization” in which a neuron sprouts related offshoots (for example, an
"iy"neuron →"diy"→"do it yourself"). - The essay author argues that the famous InceptionV1 “cat faces and car fronts” neuron is, in his reading, a V-shape neuron.
- He argues that neurons can be studied directly instead of first untangling them with a decomposition like an SAE.
- Citing a preliminary audit of Qwen3-32B, the essay author claims that “roughly 90% of neuron poles have one main job,” while noting the caveat that this number depends on thresholds.
- The essay author argues that canonical polysemantic evidence was misread, and that much apparent polysemanticity is actually “recursive generalization” in which a neuron sprouts related offshoots (for example, an
- Pushback and status: In the HN discussion, a commenter wrote they “would not rush to say that superposition is completely wrong.” This remains one researcher’s working hypothesis rather than settled consensus, and the field continues to debate it.
Why does mechanistic interpretability matter to developers?
One-line answer: It informs debugging model behaviour, safety research, and model editing/steering, though all of these remain research-stage with no guarantees in production.
- Debugging model behaviour: Investigating why a model produces specific outputs by tracing internal features and circuits.
- Safety research: Examining internal mechanisms to identify unintended or unsafe behaviours.
- Model editing and steering: Attempting to adjust internal activations to guide how a model responds.
- Research-stage caveat: All of these applications remain research-stage and come with no guarantees.
- For another perspective on how internal model representations and weights are compressed in practice, see our quantization post.
What are the most frequently asked questions about mechanistic interpretability?
Q1. What is the main difference between general XAI and mechanistic interpretability?
XAI-style attribution explains which inputs mattered for an output (such as via heatmaps), whereas mechanistic interpretability reverse-engineers the model’s internal neurons, features, and circuits.
Q2. Why does superposition cause interference?
Because the model represents more features than it has dimensions. Sparse features allow compression at the cost of interference leaking into other features, which requires nonlinear filtering—such as a ReLU with a negative bias—to filter out (Anthropic, Toy Models of Superposition, 2022).
Q3. Is the claim that "Neural networks are clean" settled scientific consensus?
No. The essay by Nick Cammarata at Goodfire is explicitly labelled by the author as “A working hypothesis.” As noted in the HN discussion—where a commenter wrote they “would not rush to say that superposition is completely wrong”—the field still actively debates it.