# Adversarial machine learning Adversarial machine learning studies attacks on machine learning algorithms and defenses against them, exploiting vulnerabilities where models assume training and test data share the same distribution. Adversarial machine learning is the study of attacks on machine learning algorithms and of the defenses against such attacks. It examines how malicious actors can intentionally manipulate data or model behavior to cause incorrect predictions, and how to make models more robust against these manipulations. The field has grown from early work on spam filtering to a broad discipline spanning computer vision, malware detection, and autonomous systems. Machine learning techniques are mostly designed to work on specific problem sets, under the assumption that the training and test data are generated from the same statistical distribution (IID). However, this assumption is often violated in practical high-stake applications, where users may intentionally supply fabricated data that violates the statistical assumption. Adversarial machine learning addresses this gap by studying both the attacks that exploit such violations and the defenses that mitigate them. The most common attacks in adversarial machine learning include evasion attacks, data poisoning attacks, Byzantine attacks, and model extraction. Evasion attacks craft inputs that are misclassified at test time, while data poisoning attacks corrupt the training data itself. Byzantine attacks target distributed learning systems by compromising some participants, and model extraction attempts to steal a model's functionality through queries. ## History At the MIT Spam Conference in January 2004, John Graham-Cumming showed that a machine-learning spam filter could be used to defeat another machine-learning spam filter by automatically learning which words to add to a spam email to get the email classified as not spam. In 2004, Nilesh Dalvi and others noted that linear classifiers used in spam filters could be defeated by simple evasion attacks as spammers inserted "good words" into their spam emails. Around 2007, some spammers added random noise to fuzz words within "image spam" to defeat OCR-based filters. In 2006, Marco Barreno and others published "Can Machine Learning Be Secure?", outlining a broad taxonomy of attacks. As late as 2013, many researchers continued to hope that non-linear classifiers (such as support vector machines and [neural networks](https://www.wikiprompt.org/wiki/neural-network)) might be robust to adversaries, until Battista Biggio and others demonstrated the first gradient-based attacks on such machine-learning models in 2012-2013. In 2012, deep neural networks began to dominate [computer-vision](https://www.wikiprompt.org/wiki/computer-vision) problems. Starting in 2014, [Christian Szegedy](https://www.wikiprompt.org/wiki/christian-szegedy) and others demonstrated that deep neural networks could be fooled by adversaries, again using a gradient-based attack to craft adversarial perturbations. These [adversarial examples](https://www.wikiprompt.org/wiki/adversarial-example) are inputs with small, often imperceptible changes that cause misclassification. Further work showed that adversarial attacks are harder to produce in uncontrolled environments, due to the different environmental constraints that cancel out the effect of noise. For example, any small rotation or slight illumination on an adversarial image can destroy the adversariality. Researchers such as [Google Brain](https://www.wikiprompt.org/wiki/google-brain)'s [Nick Frosst](https://www.wikiprompt.org/wiki/nick-frosst) point out that it is much easier to make self-driving cars miss stop signs by physically removing the sign itself, rather than creating adversarial examples. Frosst also believes that the adversarial machine learning community incorrectly assumes models trained on a certain data distribution will also perform well on a completely different data distribution. He suggests that a new approach to machine learning should be explored, and is currently working on a unique neural network that has characteristics more similar to human perception than state-of-the-art approaches. While adversarial machine learning continues to be heavily rooted in academia, large tech companies such as [Google](https://www.wikiprompt.org/wiki/google), [Microsoft](https://www.wikiprompt.org/wiki/microsoft), and IBM have begun curating documentation and open source code bases to allow others to concretely assess the [robustness](https://www.wikiprompt.org/wiki/robustness) of machine learning models and minimize the risk of adversarial attacks. ## Attack Taxonomy Adversarial attacks can be categorized along several dimensions: the attacker's knowledge (white-box vs. black-box), the goal (misclassification, targeted misclassification, or model theft), and the stage of the machine learning pipeline at which the attack occurs (training or inference). Evasion attacks occur at inference time, where an attacker modifies input data to cause a model to make a wrong prediction. Data poisoning attacks occur at training time, where an attacker injects malicious samples into the training set to corrupt the model's behavior. Byzantine attacks target distributed or [federated learning](https://www.wikiprompt.org/wiki/federated-learning) systems by compromising one or more participants, sending malicious updates to the central model. Model extraction attacks aim to replicate a model's functionality by querying it and observing outputs, potentially stealing intellectual property or identifying vulnerabilities. ## Evasion Attacks in Practice Evasion attacks have been demonstrated in numerous real-world scenarios. In spam filtering, spam messages are obfuscated through the misspelling of "bad" words or the insertion of "good" words. In computer security, attackers obfuscate malware code within network packets or modify the characteristics of a network flow to mislead intrusion detection. In biometric recognition, fake biometric traits may be exploited to impersonate a legitimate user, or to compromise users' template galleries that adapt to updated traits over time. Researchers showed that by changing only one pixel it was possible to fool [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) algorithms. Others 3D-printed a toy turtle with a texture engineered to make Google's object detection AI classify it as a rifle regardless of the angle from which the turtle was viewed. Creating the turtle required only low-cost commercially available 3D printing technology. A machine-tweaked image of a dog was shown to look like a cat to both computers and humans. A 2019 study reported that humans can guess how machines will classify adversarial images. Researchers discovered methods for perturbing the appearance of a stop sign such that an autonomous vehicle classified it as a merge or speed limit sign. In a notable demonstration, McAfee attacked [Tesla's former Mobileye system](https://www.wikiprompt.org/wiki/tesla-autopilot), fooling it into driving 50 mph over the speed limit, simply by adding a two-inch strip of black tape to a speed limit sign. Adversarial patterns on glasses or clothing designed to deceive facial-recognition systems or license-plate readers have led to a niche industry of "stealth streetwear". An adversarial attack on a neural network can allow an attacker to inject algorithms into the target system. Researchers can also create adversarial audio inputs to disguise commands to intelligent assistants in benign-seeming audio; a parallel literature explores human perception of such stimuli. ## Data Poisoning and Generative Attacks Data poisoning attacks aim to corrupt the training data. A notable example is the Nightshade filter, released in 2023 by researchers at the University of Chicago. It was created for use by visual artists to put on their artwork to corrupt the data set of text-to-image models, which usually scrape their data from the internet without the consent of the image creator. This attack is a form of [data-poisoning](https://www.wikiprompt.org/wiki/data-poisoning) that can degrade the performance of generative models. In the context of malware detection, researchers have proposed methods for adversarial malware generation that automatically craft binaries to evade learning-based detectors while preserving malicious functionality. Optimization-based attacks such as GAMMA use genetic algorithms to inject benign content (for example, padding or new PE sections) into Windows executables, framing evasion as a constrained optimization problem that balances misclassification success with the size of the injected payload and showing transferability to commercial antivirus products. Complementary work uses [generative adversarial networks (GANs)](https://www.wikiprompt.org/wiki/generative-adversarial-network) to learn feature-space perturbations that cause malware to be classified as benign; Mal-LSGAN, for instance, replaces the standard GAN loss with a least-squares objective and modified activation functions to improve training stability and produce adversarial malware examples that substantially reduce true positive rates across multiple detectors. ## Adversarial Attacks on Emerging Applications Emerging adversarial attacks are also highly effective in human-action recognition systems, which are employed widely for real-world applications such as surveillance, autonomous vehicles, and in-house monitoring. In these attacks, attackers introduce noise into the representation of the human features to fool the recognition systems. These attacks do not necessarily require access to the recognition systems themselves, and they can be imperceptible by the human users of such systems. Clustering algorithms are used in security applications. Malware and computer virus analysis aims to identify malware families, and to generate specific detection signatures. Adversarial attacks on clustering can cause the grouping of dissimilar items or the separation of similar ones, undermining the security analysis. ## Defenses Defenses against adversarial attacks fall into several categories. Adversarial training involves augmenting the training set with adversarial examples to make the model more robust. Input preprocessing techniques, such as denoising or feature squeezing, aim to remove perturbations before classification. Detection methods attempt to identify adversarial inputs by analyzing model confidence or internal activations. Model hardening approaches, such as defensive distillation, make it harder for attackers to craft effective perturbations. However, no universal defense exists, and attackers continually adapt to new defenses, leading to an ongoing arms race. ## See Also - [adversarial-example](https://www.wikiprompt.org/wiki/adversarial-example) - [data-poisoning](https://www.wikiprompt.org/wiki/data-poisoning) - [robustness](https://www.wikiprompt.org/wiki/robustness) - [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) ## External links - [Wikipedia: Adversarial machine learning](https://en.wikipedia.org/wiki/Adversarial_machine_learning) --- Source: https://www.wikiprompt.org/wiki/adversarial-machine-learning License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-11-09T12:23:19.494033+00:00