The group method of data handling (GMDH) is an inductive modeling approach that automatically constructs polynomial models from data. It was developed by Alexey Ivakhnenko in the Soviet Union in 1968 as a way to model complex systems without requiring prior knowledge of the underlying physical processes. GMDH is often described as a self-organizing method because it iteratively builds models by selecting the most relevant input variables and combining them through simple polynomial functions, typically quadratic, to minimize prediction error on validation data.
GMDH belongs to the broader family of machine learning techniques and is considered an early form of deep learning due to its layered, feed-forward structure. Unlike conventional neural networks that rely on backpropagation and gradient-based optimization, GMDH uses a heuristic selection process based on external criteria, such as the regularity criterion, to determine which nodes to retain at each layer. This makes it particularly effective for small-sample problems and for modeling nonlinear relationships in noisy environments.
Historical Development
The method was introduced by Alexey Ivakhnenko in 1968, building on earlier work in cybernetics and self-organizing systems. Ivakhnenko's research at the Institute of Cybernetics in Kyiv, Ukraine, aimed to create models that could automatically discover the structure of complex systems from data, without human intervention. The approach gained popularity in the 1970s and 1980s, especially in the Soviet Union and Eastern Europe, for applications in economics, ecology, and engineering.
GMDH was one of the first methods to use a layered architecture similar to modern deep learning models, but it differed in that it did not rely on gradient descent. Instead, it used a combinatorial selection process, which made it computationally intensive but also robust against overfitting when properly regularized. The method influenced later developments in neural networks and symbolic regression, although it remained relatively obscure in the West until the 1990s.
Algorithm and Architecture
The GMDH algorithm operates in a layer-by-layer fashion. At each layer, it generates candidate models by pairing input variables (or outputs from the previous layer) and fitting a polynomial, typically a quadratic of the form: y = a + bx_i + cx_j + dx_i^2 + ex_j^2 + fx_ix_j. The coefficients are estimated using least squares on a training subset of the data.
After generating all possible candidate models for a layer, the algorithm evaluates each candidate on a validation subset using an external criterion, such as the mean squared error or the regularity criterion. Only the best-performing candidates are retained and passed to the next layer. This process continues until the validation error stops decreasing, at which point the algorithm selects the best model from the final layer. The result is a polynomial network that can be expressed as a set of equations, making it interpretable compared to many black-box machine learning models.
The architecture is similar to a feedforward neural network, but with a key difference: the structure is not fixed in advance; it is determined by the data. This self-organizing property is the hallmark of GMDH and distinguishes it from traditional neural network training methods.
Applications and Use Cases
GMDH has been applied in a wide range of fields. In engineering, it has been used for process identification, fault detection, and control system design. In economics and finance, it has been employed for forecasting stock prices, exchange rates, and macroeconomic indicators. In environmental science, GMDH models have been used to predict air quality, water quality, and weather patterns.
One notable application is in the field of artificial intelligence and machine learning for time series prediction. GMDH's ability to automatically select relevant lagged variables makes it suitable for modeling dynamic systems. It has also been used in bioinformatics for gene expression analysis and in medicine for diagnostic support.
Despite the rise of more powerful methods like deep learning and large language models, GMDH remains useful in scenarios where data is scarce, interpretability is important, or the underlying system is nonlinear and poorly understood. Its polynomial form allows for easy integration with optimization and control algorithms.
Comparison with Other Methods
GMDH is often compared to neural networks, particularly in terms of training and interpretability. While neural networks use backpropagation and gradient descent, GMDH uses a heuristic search over polynomial combinations. This makes GMDH less prone to local minima but more sensitive to the choice of external criteria and the partitioning of data into training and validation sets.
Compared to deep learning methods, GMDH typically requires fewer parameters and less computational resources, but it may not scale as well to very high-dimensional problems. It is also less flexible in terms of the types of functions it can represent, as it is limited to polynomial combinations. However, its transparency and ability to work with small datasets make it a valuable tool in many practical applications.
GMDH is related to other inductive modeling techniques such as symbolic regression and genetic programming, but it differs in its deterministic, layer-wise selection process. It also shares conceptual similarities with residual networks in that each layer refines the prediction, but the selection mechanism is distinct.
Limitations and Extensions
One of the main limitations of GMDH is its computational complexity, especially when the number of input variables is large, as the number of candidate pairs grows quadratically. Additionally, the method can overfit if the external criterion is not properly chosen or if the validation set is not representative. Various extensions have been proposed to address these issues, including the use of different polynomial degrees, regularization techniques, and hybrid approaches that combine GMDH with other machine learning methods.
Another limitation is that GMDH assumes that the relationship between inputs and outputs can be approximated by polynomials, which may not hold for all systems. In such cases, other methods like deep learning or Transformer (architecture)-based models may be more appropriate. Nevertheless, GMDH remains an important historical and practical contribution to the field of artificial intelligence.