Kernel methods are a well-established tool for analyzing the relationship between input data and the corresponding output of a function. In typical machine learning, these functions produce a scalar output, but recent developments have extended kernel methods to functions with vector-valued output. This extension is driven by interest in simultaneously solving related problems, where kernels that capture relationships between tasks allow algorithms to borrow strength from each other. Algorithms of this type include multi-task learning (also called multi-output learning or vector-valued learning), transfer learning, and co-kriging. Multi-label classification can be interpreted as mapping inputs to binary coding vectors with length equal to the number of classes. In Gaussian processes, kernels are called covariance functions, and multiple-output functions correspond to considering multiple processes.
The supervised learning problem in this context is to learn a function \( f \) that predicts vector-valued outputs \( \mathbf{y_i} \) given inputs \( \mathbf{x_i} \), for \( i = 1, \ldots, N \). The inputs belong to an input space \( \mathcal{X} \), often \( \mathbb{R}^p \), and the outputs belong to \( \mathbb{R}^D \). In general, each component of the output vector could have different input data with different cardinality and even different input spaces; geostatistics literature calls this case heterotopic, while isotopic indicates that each component shares the same set of inputs. For simplicity, the notation often assumes the number and sample space of the data for each output are the same.
History
The history of learning vector-valued functions is closely linked to transfer learning, which involves storing knowledge gained while solving one problem and applying it to a different but related problem. The fundamental motivation for transfer learning was discussed at a NIPS-95 workshop on "Learning to Learn," which focused on the need for lifelong machine learning methods that retain and reuse previously learned knowledge. Research on transfer learning has attracted much attention since 1995 under various names, including learning to learn, lifelong learning, knowledge transfer, inductive transfer, multitask learning, knowledge consolidation, context-sensitive learning, knowledge-based inductive bias, metalearning, and incremental/cumulative learning. Interest in learning vector-valued functions was particularly sparked by multitask learning, a framework that tries to learn multiple, possibly different tasks simultaneously.
Much of the initial research in multitask learning in the machine learning community was algorithmic in nature, applied to methods such as neural networks, decision trees, and k-nearest neighbors in the 1990s. The use of probabilistic models and Gaussian processes was pioneered and largely developed in the context of geostatistics, where prediction over vector-valued output data is known as cokriging. Geostatistical approaches to multivariate modeling are mostly formulated around the linear model of coregionalization (LMC), a generative approach for developing valid covariance functions used for multivariate regression and in statistics for computer emulation of expensive multivariate computer codes. The regularization and kernel theory literature for vector-valued functions followed in the 2000s. While the Bayesian and regularization perspectives were developed independently, they are in fact closely related.
Regularization Perspective
From the regularization perspective, the problem is to learn a function \( f_* \) belonging to a reproducing kernel Hilbert space of vector-valued functions (\( \mathcal{H} \)). This is similar to the scalar case of Tikhonov regularization, with extra care in notation. It is possible, though non-trivial, to show that a representer theorem also holds for Tikhonov regularization in the vector-valued setting. The matrix-valued kernel \( \mathbf{K} \) can also be defined by a scalar kernel \( R \) on the space \( \mathcal{X} \times \{1, \ldots, D\} \). This perspective connects to the broader field of machine learning, where regularization techniques are fundamental to preventing overfitting and improving generalization.
Multi-Task Learning and Related Paradigms
Multi-task learning is a core application of kernel methods for vector output. The goal is to learn multiple tasks simultaneously, leveraging shared information to improve performance on each task. This approach is related to transfer learning, where knowledge from one task is reused for another, and to co-kriging, a geostatistical method for multivariate spatial prediction. In multi-label classification, the vector output encodes class membership, and kernels that capture correlations between labels can improve classification accuracy. These methods have been applied in various domains, including artificial intelligence and deep learning, where vector-valued predictions are common in tasks such as multi-task neural networks.
Connection to Gaussian Processes
In Gaussian processes, kernels serve as covariance functions, and multiple-output functions correspond to considering multiple correlated processes. This Bayesian perspective is closely related to the regularization perspective, as both aim to model the relationship between inputs and outputs while capturing dependencies between output dimensions. The linear model of coregionalization is a standard approach for constructing valid covariance functions in this context. This connection has influenced modern approaches in neural networks and transformers, where multi-output predictions are often handled through shared representations.
Applications and Extensions
Kernel methods for vector output have been applied in geostatistics, computer emulation, and multi-task learning. They are also relevant to modern generative AI systems, where models often produce vector-valued outputs such as embeddings or multi-dimensional predictions. Research continues on developing efficient algorithms and kernels that scale to large datasets, building on foundational work from institutions like MIT CSAIL and Stanford AI Lab. The field remains an active area of study, with connections to curriculum learning and other advanced training techniques.