Algorithmic bias refers to systematic and unfair discrimination produced by an algorithmic system, most often a Machine learning model whose predictions or outputs disadvantage particular groups defined by race, gender, age, or other protected characteristics. Because such systems learn statistical patterns from Training data, they tend to reproduce, and can amplify, biases already present in that data or introduced by how a problem is framed, labeled, or evaluated. Algorithmic bias is a central concern within AI ethics and has motivated both technical mitigation research and legal scrutiny of deployed systems.
Sources of bias
Researchers commonly distinguish several sources. Historical bias exists in the data itself, for example when past hiring decisions reflected discrimination that a model then learns to replicate. Representation bias arises when certain groups are under-sampled in a dataset, degrading model accuracy for them; this was central to the 2018 "Gender Shades" audit by Joy Buolamwini and Timnit Gebru, which found that commercial face-recognition systems performed markedly worse on darker-skinned women than on lighter-skinned men. Measurement bias occurs when the proxy a model is trained to predict, such as arrest rates as a stand-in for crime, is itself a biased measurement of the underlying concept. Aggregation bias arises from applying one model uniformly across groups for which the underlying relationships differ.
High-profile cases
Several documented cases shaped the public and academic discussion. The COMPAS recidivism-prediction tool used in U.S. courts was the subject of a 2016 ProPublica investigation that found the system flagged Black defendants as future risks at nearly twice the rate of white defendants relative to actual reoffense rates, though researchers subsequently debated which statistical definition of fairness the finding relied on. Amazon scrapped an internal AI recruiting tool in 2018 after discovering it penalized resumes containing the word "women's," a pattern learned from a decade of male-dominated hiring data. Face-recognition audits, including the U.S. National Institute of Standards and Technology's 2019 study of over 100 systems, repeatedly found higher error rates for women and people with darker skin, contributing to several U.S. cities banning police use of the technology and to some vendors pausing sales to law enforcement.
Mitigation and measurement
Technical responses include auditing models against fairness metrics such as demographic parity or equalized odds, rebalancing or augmenting training data, and applying post-processing adjustments to model outputs; a large research literature has also shown that different fairness definitions can be mathematically incompatible with one another, meaning that satisfying one metric can require violating another. Tools for Explainable AI are frequently used alongside fairness audits to help identify which input features drive disparate outcomes. Bias concerns have also extended into the era of Large language model systems, where researchers have documented stereotyped associations in generated text and images, including in Text-to-image generation systems.
Governance response
Algorithmic bias has become a recurring justification for regulation, appearing explicitly in the risk-tiering of the EU AI Act and in earlier U.S. proposals such as the Algorithmic Accountability Act. It remains one of the most litigated and most empirically documented harms in the broader field of AI governance.