Model stealing is a class of attacks against deployed machine learning systems in which an adversary attempts to recover the internal parameters, architecture, or functional behavior of a proprietary model by submitting carefully crafted queries and observing the outputs. The goal is to create a substitute model that closely mimics the original, thereby circumventing the intellectual property protections and computational investments of the model's owner. This threat has grown in significance with the proliferation of commercial Artificial intelligence services, particularly Large language model APIs offered by companies like OpenAI, Anthropic, and Google DeepMind, where access is gated behind paid interfaces but the underlying weights remain secret.
The attack exploits the fact that a model's input-output behavior is a rich source of information about its internal decision boundaries. By collecting a sufficient number of labeled examples - where labels are the model's predictions - an attacker can train a surrogate model that approximates the original with high fidelity. The feasibility of this approach depends on the model's complexity, the query budget available, and the degree of output detail exposed by the API. Early demonstrations focused on simple classifiers, but the technique has been extended to deep neural networks and generative models, raising concerns about the economic viability of selling model access without adequate safeguards.
Historical Context
The concept of model stealing emerged in the mid-2010s alongside the commercialization of machine learning as a service. In 2016, researchers at University of Toronto and cornell-university published foundational work showing that logistic regression and decision tree models could be extracted with high accuracy using only prediction queries. This work, led by Florian Tramèr and colleagues, demonstrated that even models with hidden architectures could be reverse-engineered through careful query selection. The attack was framed as an intellectual property threat, analogous to software piracy but applied to learned systems.
Subsequent research expanded the scope. In 2017, a team from Carnegie Mellon University showed that neural network classifiers trained on image datasets could be stolen with query budgets in the tens of thousands. By 2019, studies had extended the technique to Transformer (architecture)-based language models, where the high-dimensional output space made extraction more challenging but still feasible. The rise of API-based Generative AI services in the early 2020s intensified interest, as these models represent billions of dollars in training costs and are accessed remotely by millions of users.
Attack Vectors
Model stealing attacks operate through several distinct vectors, each exploiting different aspects of the model's interface. The most straightforward approach is functional extraction, where the attacker queries the model with a diverse set of inputs and records the outputs, then trains a surrogate model on this dataset. The success of this method hinges on the diversity and coverage of the query set; if the attacker can sample the input space sufficiently, the surrogate can approximate the original's decision boundaries.
A second vector is parameter extraction, which aims to recover the exact numerical values of the model's weights. This is significantly harder than functional extraction and typically requires additional information, such as the model's architecture and activation functions. In some cases, side-channel attacks - exploiting timing or memory usage patterns - have been used to infer architectural details. For example, a 2020 study demonstrated that the depth and width of a neural network could be deduced from the latency of API responses, enabling a more targeted parameter recovery.
A third vector exploits the confidence scores returned by many APIs. When a model outputs a probability distribution over classes, the attacker gains richer information than a single hard label. This allows for more efficient extraction, as the surrogate can be trained using soft labels that encode the model's internal certainty. Some APIs have mitigated this by returning only top-k predictions or by adding noise to the outputs, but these defenses are not always effective.
Mathematical Foundations
The theoretical basis for model stealing lies in the concept of active learning and query complexity. An attacker seeks to minimize the number of queries needed to achieve a given fidelity level. For linear models, the query complexity is proportional to the number of parameters, as each query provides one linear constraint on the weight vector. For deep networks, the relationship is more complex, but researchers have shown that the sample complexity of functional extraction scales with the model's VC dimension or Rademacher complexity.
A key result from the 2016 work is that equation-solving attacks can recover exact parameters for models with piecewise-linear activation functions, such as Neural network with ReLU units. By crafting inputs that lie on decision boundaries, the attacker can derive linear equations that constrain the weights. This approach requires precise control over inputs and access to continuous outputs, which is often unavailable in commercial APIs that discretize predictions.
More recent theoretical work has examined the information-theoretic limits of extraction. For a model with N parameters, an attacker needs at least O(N) queries to recover the function exactly, but functional approximation may require far fewer. This asymmetry - where a surrogate can achieve high accuracy on the training distribution without matching the original's behavior on out-of-distribution inputs - is a central challenge for defenders.
Defensive Strategies
Defenders have developed a range of countermeasures, broadly categorized into detection, rate limiting, and output perturbation. Detection methods aim to identify malicious query patterns, such as repeated queries from a single IP address or queries that probe decision boundaries. Rate limiting restricts the number of queries a user can make within a time window, increasing the cost of extraction. Output perturbation adds noise to predictions, degrading the quality of the surrogate but also reducing the utility for legitimate users.
A more sophisticated defense is watermarking, where the model is trained to embed a hidden pattern in its outputs that can be used to prove ownership. If an attacker steals the model and deploys it, the watermark can be detected in the surrogate's behavior. This approach has been explored for Deep learning models, but it is not foolproof, as attackers may attempt to remove the watermark through fine-tuning or distillation.
Another line of defense involves differential privacy, which bounds the influence of any single query on the model's outputs. By adding calibrated noise to the predictions, the model owner can guarantee that an attacker cannot extract precise information about the training data or parameters. However, this comes at the cost of reduced accuracy, making it less attractive for high-stakes applications.
Case Studies and Real-World Incidents
Several high-profile incidents have highlighted the practical threat of model stealing. In 2021, researchers from BAIR (Berkeley AI Research) demonstrated that they could extract a functional copy of a commercial image classifier using only 10,000 queries, achieving 98% agreement on a test set. The attack targeted a model hosted on Amazon Web Services and required no prior knowledge of the architecture.
In the language domain, a 2023 study showed that a Large language model could be distilled into a smaller surrogate with comparable performance on common benchmarks, using only API outputs. This raised concerns about the economic model of selling access to proprietary models, as competitors could potentially replicate functionality at a fraction of the training cost. The study noted that the surrogate performed poorly on rare or adversarial inputs, but this was not sufficient to deter the attack.
A notable legal case involved a startup that was accused of using model stealing to replicate a competitor's recommendation algorithm. The case was settled out of court, but it underscored the need for clear legal frameworks around model intellectual property. As of 2024, no major jurisdiction has established specific statutes for model theft, leaving companies to rely on trade secret protections and terms of service agreements.
Ethical and Legal Implications
The ethics of model stealing are contested. Proponents argue that models trained on publicly available data should not be subject to exclusive ownership, and that extraction enables research and competition. Critics counter that the massive computational resources required for training - often involving thousands of GPU (in AI)-hours on specialized hardware like AWS Trainium or Google Cloud - constitute a significant investment that deserves protection.
From a legal perspective, model stealing intersects with trade secret law, copyright, and contract law. In the United States, the Defend Trade Secrets Act of 2016 provides a federal cause of action for misappropriation, but applying it to machine learning models is complicated by the difficulty of proving that specific parameters constitute a trade secret. Some companies have resorted to patenting model architectures, but this is not universally applicable and can be costly.
The debate is further complicated by the open-source movement, which advocates for transparency in AI development. Many researchers argue that model weights should be publicly released to enable scrutiny and reproducibility, which would render stealing moot. However, commercial pressures and concerns about misuse have led to a trend of closed models, particularly among large tech firms.
Future Directions
As models become larger and more capable, the threat of model stealing is likely to evolve. One emerging area is generative model extraction, where attackers aim to replicate the output distribution of models like Stable Diffusion or GPT-4. This is more challenging because the output space is high-dimensional and continuous, but recent work has shown that distillation techniques can produce surrogates that generate similar samples.
Another direction is adversarial query generation, where attackers use optimization to find inputs that maximize the information gained per query. This could reduce the query budget needed for extraction by orders of magnitude. Defenders are responding with more sophisticated anomaly detection and adaptive rate limiting, but the arms race is ongoing.
Research is also exploring the use of hardware-based protections, such as trusted execution environments that prevent direct access to model weights. However, these approaches are not yet practical for large-scale deployment, and they do not address functional extraction, which only requires input-output access. As of 2025, model stealing remains an open problem, with no definitive solution on the horizon.
See Also
- Model Pruning
- Knowledge distillation
- adversarial-machine-learning
- intellectual-property
References
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016). Stealing machine learning models via prediction APIs. USENIX Security Symposium.
Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., & Papernot, N. (2020). High accuracy and high fidelity extraction of neural networks. USENIX Security Symposium.
Orekondy, T., Schiele, B., & Fritz, M. (2019). Knockoff nets: Stealing functionality of black-box models. CVPR.