Feature Engineering

Feature engineering is the preprocessing step of selecting, transforming, and extracting raw data into features to improve machine learning model performance. It is applied across fields including physics and is essential for predictive accuracy.

Feature engineering is a preprocessing step in supervised machine learning and statistical modeling that transforms raw data into a more effective set of inputs. Each input comprises several attributes, known as features. By providing models with relevant information, feature engineering significantly enhances their predictive accuracy and decision-making capability. The principles extend beyond machine learning to scientific fields such as physics, where dimensionless numbers like the Reynolds number in fluid dynamics, the Nusselt number in heat transfer, and the Archimedes number in sedimentation are constructed, and analytical solutions for strength of materials are developed as first approximations.

Clustering Applications

Feature engineering is widely used for clustering feature-objects or sample-objects in datasets. Matrix decomposition-based methods, especially those with non-negativity constraints on feature coefficients, are extensively applied. These include Non-Negative Matrix Factorization (NMF), Non-Negative Matrix-Tri Factorization (NMTF), and Non-Negative Tensor Decomposition/Factorization (NTF/NTD). The non-negativity constraints yield part-based representations, and different factor matrices exhibit natural clustering properties. Extensions include orthogonality-constrained factorization for hard clustering and manifold learning to address inherent algorithm issues.

Other approaches leverage common hidden structures across multiple inter-related datasets to obtain consensus clustering. Multi-view Classification based on Consensus Matrix Decomposition (MCMD) mines a common clustering scheme across datasets, outputs scale-variant and scale-invariant class labels, is robust to missing information, can detect shape- and scale-based outliers, and handles high-dimensional data effectively. Coupled matrix and tensor decompositions are popular in multi-view feature engineering.

Predictive Modelling

In machine learning and statistical modeling, feature engineering involves selecting, creating, transforming, and extracting features. Key components include feature creation from existing data, transforming and imputing missing or invalid features, reducing dimensionality through methods like Principal Components Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA), and selecting relevant features based on importance scores and correlation matrices.

Features vary in significance; even relatively insignificant features may contribute to a model. Feature selection can reduce the number of features to prevent overfitting. Feature explosion occurs when the number of identified features is too large for effective model estimation, often caused by feature templates or feature combinations that cannot be represented by a linear system. It can be limited via regularization, kernel methods, and feature selection.

Feature Templates

Feature templates are abstract specifications of features used to automatically populate the set of features from every instance in the training and test set. They allow automatic generation of large numbers of specific features, reducing manual coding effort.

Automation

Automation of feature engineering has been a research topic since the 1990s, with commercial machine learning software incorporating it available since 2016. Academic literature splits into two main types: Multi-relational Decision Tree Learning (MRDTL) and Deep Feature Synthesis.

Multi-relational Decision Tree Learning (MRDTL)

MRDTL extends traditional decision tree methods to relational databases, handling complex data relationships across tables. It uses selection graphs as decision nodes, refined systematically until a termination criterion is reached. Most implementations are based on relational databases, leading to redundant operations that can be reduced using techniques like tuple id propagation.

Open-source Implementations

Several open-source libraries automate feature engineering on relational data and time series:

featuretools: Python library for transforming time series and relational data into feature matrices.
MCMD: Open-source algorithm for joint clustering of multiple datasets.
OneBM (One-Button Machine): Combines feature transformations and selection on relational data, reducing data exploration time.
getML community: C/C++ tool with Python interface, at least 60 times faster than tsflex, tsfresh, tsfel, featuretools, or kats.
tsfresh: Python library for time series feature extraction, evaluating feature quality via hypothesis testing.
tsflex: Python library for time series features, faster and more memory efficient than tsfresh, seglearn, or tsfel.
seglearn: Extension for multivariate, sequential time series data to scikit-learn.
tsfel: Python package for time series feature extraction.
kats: Python toolkit for time series analysis.

Deep Feature Synthesis

The deep feature synthesis (DFS) algorithm beat 615 of 906 human teams in a competition, demonstrating its effectiveness.

Feature Stores

A feature store is a central repository where features are stored and organized for the explicit purpose of training models or making predictions. It allows data scientists to create or update groups of features, ensuring consistency and reusability across different models and applications.