Co-training is a semi-supervised learning algorithm introduced by Avrim Blum and Tom Mitchell in 1998. It addresses the challenge of limited labeled data by leveraging unlabeled examples when features can be naturally split into two conditionally independent views. The method trains two separate classifiers on these views, and each classifier's high-confidence predictions on unlabeled data are used to augment the other's training set, iteratively improving both models.
The approach is grounded in the assumption that each view is sufficient for learning on its own and that the views are conditionally independent given the class label. In practice, this holds for domains like web page classification, where the text on a page and the anchor text of hyperlinks pointing to it provide distinct but complementary signals. Co-training has influenced later semi-supervised techniques and remains a foundational concept in Machine learning research.
Algorithm and Mechanism
Co-training begins with two labeled datasets, each containing features from a single view (e.g., view A and view B). Two base classifiers, often Naive Bayes or Decision Trees models, are trained independently on these labeled sets. At each iteration, each classifier selects a small number of unlabeled examples for which it has the highest confidence (typically measured by predicted probability) and assigns them pseudo-labels. These newly labeled examples are then added to the other classifier's training set, and both classifiers are retrained.
The process repeats for a fixed number of iterations or until no high-confidence predictions remain. The key is that each classifier provides the other with 'new' labeled data that it would not have encountered alone, effectively expanding the training pool without human annotation. Theoretical analysis by Blum and Mitchell showed that if the views are conditionally independent and each is sufficient, co-training can boost accuracy even with very few initial labels.
Applications and Domains
Co-training has been applied across various fields where multi-view data naturally exists. In natural language processing, it has been used for tasks like Named-entity recognition and Sentiment Analysis, where one view might be word-level features and another syntactic or positional features. In computer vision, co-training can combine color histograms with texture descriptors for object classification. In bioinformatics, it has been used to classify gene expression data using different measurement platforms as views.
A classic example is web page classification: the text content of a page and the anchor text of inbound links form two views. Early work by Blum and Mitchell demonstrated that co-training could classify academic course homepages with high accuracy using only a few labeled pages and many unlabeled ones. This application highlighted the method's practical value in settings where labeling is expensive but unlabeled data is abundant.
Extensions and Variants
Several extensions have been proposed to relax co-training's strict assumptions. tri-training uses three classifiers and majority voting to assign pseudo-labels, reducing the need for view splitting. co-regularization adds a penalty term to encourage agreement between classifiers on unlabeled data. democratic-co-learning generalizes the idea to multiple learners with different algorithms rather than different feature views.
Another variant, self-training, is a simpler single-view approach where one classifier labels its own confident predictions, but it is more prone to error propagation. Co-training's multi-view design mitigates this by having each classifier correct the other's mistakes, provided the independence assumption holds. Researchers have also explored deep learning adaptations, where neural networks with different architectures or input transformations serve as the two views.
Theoretical Foundations and Limitations
The theoretical guarantee of co-training relies on the conditional independence of views given the label. In practice, this assumption is often violated, leading to degraded performance. When views are correlated, the classifiers may reinforce each other's errors, causing confirmation bias. Empirical studies have shown that co-training can still work well when the independence assumption is approximately satisfied, but it may underperform simple self-training on datasets with highly correlated features.
Another limitation is the need for a natural feature split, which is not always available. In such cases, practitioners may use random feature splits, but this often weakens the theoretical justification. The method also requires careful tuning of confidence thresholds and the number of examples added per iteration, as overly aggressive labeling can introduce noise. Despite these challenges, co-training remains a benchmark in semi-supervised learning literature and a building block for more robust algorithms.
Relationship to Modern Semi-Supervised Learning
With the rise of Deep learning, co-training has inspired modern approaches like consistency-regularization and pseudo-labeling used in state-of-the-art models. Techniques such as mixmatch and fixmatch combine multiple views through data augmentation and enforce prediction consistency, echoing co-training's core idea of leveraging agreement between different perspectives. The concept also aligns with multi-view-learning in Representation learning, where models learn from multiple modalities or feature sets.
In the era of Large language models, co-training principles appear in reinforcement-learning-from-human-feedback (Reinforcement Learning from AI Feedback (RLAIF)) pipelines, where separate models for reward prediction and policy generation iteratively refine each other. While modern methods often scale to massive unlabeled corpora, co-training's foundational insight - that diverse views can provide supervisory signals - continues to inform algorithm design in both academic and industrial settings.