Mechanistic interpretability is a research program within AI safety and machine learning that attempts to reverse-engineer the trained weights of a Neural network into a human-understandable account of the algorithms it implements, rather than merely approximating its behavior from the outside as most Explainable AI methods do. The approach treats a trained model similarly to how a reverse engineer might treat compiled binary code without source: by probing individual neurons, layers, and combinations of weights, researchers try to recover the discrete features and circuits a network uses to perform a task, with the eventual goal of being able to predict, verify, or edit a model's behavior with the same confidence as reading its source code.
Origins
Early interpretability work in computer vision, including feature visualization research associated with the online journal Distill, showed that individual neurons or small groups of neurons in image classifiers such as Convolutional neural network models could correspond to recognizable concepts like curves, textures, or object parts. Chris Olah, a co-founder of Anthropic and a leading figure in the field, helped popularize this visual, circuit-based style of analysis before extending it to language models. The rise of the Transformer (architecture) architecture and Large language model systems shifted the field's focus toward understanding attention heads, residual streams, and the internal representations that let a model perform tasks such as In-context learning.
Key findings
Interpretability researchers have identified several recurring phenomena. Superposition describes the finding that neural networks often represent more distinct features than they have neurons, packing multiple concepts into overlapping directions in activation space, which makes individual neurons difficult to interpret in isolation. Sparse autoencoders, an application of the broader Autoencoder technique, have been used to decompose these overlapping representations into a much larger set of more monosemantic, individually interpretable features; Anthropic's 2024 work identifying millions of such features in a production-scale Claude model, and subsequent work locating and editing specific behaviorally relevant features, was among the most widely discussed results in the field. Other lines of work have identified specific circuits responsible for narrow capabilities, such as an "induction head" mechanism that lets transformer models complete repeated patterns, and have used interpretability tools to detect signs of deception or goal misgeneralization inside a model rather than relying solely on its stated output.
Motivation and stakes
Proponents argue that mechanistic interpretability is one of the few research directions that could give humans genuine insight into whether a highly capable model is pursuing the goals it appears to pursue, rather than relying on behavioral testing alone, a concern closely tied to debates about AI alignment and Reward hacking. Because interpretability aims to open the black box directly rather than explain it after the fact, some researchers consider it a prerequisite for confidently deploying much more capable future systems, including any approaching Artificial general intelligence.
Limitations
The field remains far from a complete account of any frontier model; current techniques scale poorly to the size of production models, sparse-autoencoder features do not cleanly cover all of a network's behavior, and there is no agreed method for verifying that an interpretability-based explanation is complete rather than merely plausible. Researchers generally describe mechanistic interpretability as an early-stage science, comparable in ambition, though not yet in maturity, to neuroscience's attempt to map biological brains.