Explainable AI, commonly abbreviated XAI, is the subfield of artificial intelligence concerned with making the internal workings or outputs of a model understandable to human users. The need for explainability grew alongside the shift from simple, inherently interpretable models such as linear regression and decision trees toward complex Neural network architectures whose predictions arise from millions or billions of learned parameters interacting in ways not directly traceable by a human reader, a limitation widely referred to as the black-box problem. XAI methods aim to bridge that gap by producing explanations, either built into a model's design or generated after the fact, without necessarily requiring a full mechanistic account of the model's internal computation.
Motivations
Demand for explainability comes from several directions. Regulators in sectors such as credit, healthcare, and criminal justice increasingly require that automated decisions affecting individuals be explainable, a principle echoed in the EU's General Data Protection Regulation and in the EU AI Act. Developers use explanations to debug models and catch spurious correlations, such as a model learning to recognize an animal from background terrain rather than the animal itself. Explanations also support efforts to detect and correct Algorithmic bias, and to build user trust in systems used for consequential decisions.
Techniques
XAI methods are commonly split into two families. Intrinsically interpretable models, such as decision trees, rule lists, and linear models, are transparent by construction but tend to be less accurate on complex tasks than deep learning. Post-hoc methods instead treat a trained model as fixed and attempt to explain individual predictions or general behavior around it. Widely used post-hoc techniques include saliency maps, which highlight the input regions, such as pixels in an image, that most influenced a Convolutional neural network's output; LIME (Local Interpretable Model-agnostic Explanations), which approximates a complex model's local behavior with a simpler surrogate model; and SHAP (SHapley Additive exPlanations), which uses a game-theoretic framework borrowed from cooperative game theory to attribute a prediction to its contributing input features in a way that satisfies certain fairness properties. These methods are model-agnostic in the sense that they can be applied to many types of Machine learning systems without access to internal weights.
Limits and relation to interpretability
Post-hoc explanation methods have drawn criticism for sometimes producing plausible-looking but unfaithful accounts of a model's actual reasoning, since they approximate rather than directly expose the computation involved. This limitation has driven a parallel, more ambitious research direction known as mechanistic Mechanistic interpretability, which seeks to reverse-engineer the actual circuits and features a network has learned rather than approximate its behavior from the outside; the two fields are closely related but pursue different standards of evidence. XAI methods designed for classical Computer vision and tabular models have proven harder to adapt to Large language model systems, whose outputs are free-form text rather than a fixed set of classes, which has shifted much of the frontier explainability research toward interpretability-style circuit analysis and toward simply asking models to produce a Chain-of-thought as a proxy explanation, albeit one that does not always faithfully reflect the underlying computation.