# Concept Activation Vectors Concept Activation Vectors (CAVs) are a technique in machine learning for interpreting neural network representations by testing whether a specific concept is encoded in the model's internal activations. Concept Activation Vectors (CAVs) are a technique in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) for interpreting the internal representations of [neural networks](https://www.wikiprompt.org/wiki/neural-network). Introduced in 2017 by researchers at the [Google Brain](https://www.wikiprompt.org/wiki/google-deepmind) and Stanford University, CAVs provide a way to test whether a given concept - such as 'striped' or 'doctor' - is encoded in the activations of a model. The method involves training a linear classifier to distinguish between activations from examples that contain the concept and those that do not, then using the classifier's weight vector as the CAV. This vector points in the direction within the activation space that corresponds to the concept, allowing researchers to probe the model's understanding in a human-interpretable way. CAVs are widely used in the field of AI interpretability, particularly for [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) models, to assess bias, verify learned features, and improve transparency. They are especially relevant for [large language models](https://www.wikiprompt.org/wiki/large-language-model) and [transformers](https://www.wikiprompt.org/wiki/transformer), where understanding what the model 'knows' is critical for trust and safety. ## Background and Motivation As neural networks grew in complexity, their internal workings became increasingly opaque. Early interpretability methods focused on visualizing individual neurons or attention weights, but these approaches often failed to capture the distributed nature of representations. In 2017, a team led by Been Kim at Google Brain sought to address this by proposing a more systematic way to interrogate models. Their work, published in the paper 'Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)', introduced CAVs as part of a broader framework called TCAV (Testing with Concept Activation Vectors). The motivation was to move beyond simple feature attribution and enable quantitative testing of whether a concept is used by a model in its decision-making. The key insight was that concepts are often represented as directions in the high-dimensional activation space, rather than as individual neurons. By training a linear probe on top of these activations, one can extract a vector that captures the concept's direction. This approach is grounded in the observation that neural networks often learn linearly separable representations for high-level concepts. ## How CAVs Work The process of computing a CAV involves several steps. First, one selects a set of examples that represent the concept of interest (e.g., images of striped objects) and a set of 'counterexamples' that do not (e.g., images of non-striped objects). The model is then run on these examples, and the activations at a chosen layer are recorded. Next, a linear classifier - typically a support vector machine (SVM) or logistic regression - is trained to distinguish between the two sets of activations. The weight vector of this classifier, normalized to unit length, becomes the CAV. Once a CAV is obtained, it can be used to test the model's sensitivity to the concept. For example, in TCAV, one computes directional derivatives of the model's output with respect to the activations along the CAV direction. A positive derivative indicates that the model's prediction increases when the concept is present, suggesting that the model relies on that concept. This can be done for any layer of the network, providing a layer-wise analysis of concept usage. ## Applications in Interpretability CAVs have been applied in various domains, including computer vision and natural language processing. In the original paper, the authors demonstrated the technique on image classification models, showing that concepts like 'striped' or 'spotted' were used by a model to classify zebras and leopards. They also used CAVs to detect potential biases, such as a model using 'gender' as a factor in classifying images of doctors. In [NLP](https://www.wikiprompt.org/wiki/natural-language-processing), CAVs have been used to probe [large language models](https://www.wikiprompt.org/wiki/large-language-model) for concepts like 'toxicity', 'sentiment', or 'topic'. For instance, researchers have used CAVs to identify whether a model associates certain demographic terms with negative sentiment, thereby uncovering hidden biases. This is particularly important for [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) systems deployed in real-world applications. ## Relation to Other Interpretability Methods CAVs are part of a broader family of interpretability techniques that include feature attribution, saliency maps, and activation maximization. Unlike saliency maps, which highlight input features, CAVs operate on internal representations. They are also related to linear probing, where a classifier is trained on activations to predict a property; CAVs extend this by using the classifier's weight vector as a concept direction. Compared to neuron-level analysis, CAVs capture distributed representations, making them more robust to polysemanticity - the phenomenon where a single neuron responds to multiple unrelated concepts. This is a significant advantage in modern [transformers](https://www.wikiprompt.org/wiki/transformer), where polysemanticity is common. ## Limitations and Challenges Despite their utility, CAVs have limitations. One major challenge is the choice of counterexamples. The quality of the CAV depends heavily on the selection of negative examples; if they are not representative, the CAV may capture spurious correlations. Additionally, CAVs are linear, which may not capture nonlinear concept relationships. Another issue is that CAVs are computed post hoc, meaning they reflect what the model has already learned, not what it could learn. Furthermore, CAVs require access to the model's internal activations, which may not be available for proprietary models. This has led to the development of alternative methods that work with black-box models, but CAVs remain a valuable tool for white-box interpretability. ## Extensions and Variants Several extensions of CAVs have been proposed. For example, 'Concept Bottleneck Models' incorporate concept directions directly into the model architecture, allowing for explicit concept-based reasoning. Another variant, 'Automated Concept Discovery', uses clustering to automatically find concepts without human-labeled examples. In the context of [large language models](https://www.wikiprompt.org/wiki/large-language-model), researchers have adapted CAVs to work with token-level activations, enabling fine-grained analysis of how concepts are represented across different contexts. Recent work has also explored using CAVs for model editing and debiasing. By identifying concept directions, one can potentially adjust the model's representations to reduce reliance on unwanted concepts, such as gender or race. This is an active area of research in AI safety and fairness. ## Impact and Future Directions CAVs have had a significant impact on the field of AI interpretability, influencing subsequent work on mechanistic interpretability and representation engineering. They are now a standard tool in the interpretability toolkit, used by researchers at institutions like [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). As models continue to grow in scale, the need for interpretability methods like CAVs becomes more pressing. Future directions include developing CAVs for multimodal models, improving the robustness of concept extraction, and integrating CAVs into automated auditing systems. In summary, Concept Activation Vectors provide a principled way to test whether a concept is encoded in a model's representations, offering insights into what models learn and how they make decisions. Their simplicity and effectiveness have made them a cornerstone of modern interpretability research. ## See Also - [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) - [neural-network](https://www.wikiprompt.org/wiki/neural-network) - [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) - [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) - [transformer](https://www.wikiprompt.org/wiki/transformer) - [interpretability](https://www.wikiprompt.org/wiki/interpretability) (note: not in provided list, but as a general term) ## References - Kim, B., et al. (2018). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML. - Koh, P. W., et al. (2020). Concept Bottleneck Models. ICML. - Goh, G., et al. (2021). Multimodal Neurons in Artificial Neural Networks. Distill. --- Source: https://www.wikiprompt.org/wiki/concept-activation-vectors License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-13T03:59:11.662236+00:00