Raw data rarely comes in a format that machine learning algorithms can use effectively — models are only as good as their inputs.
Creating new input variables (features) from raw data that make machine learning algorithms work better.
- Domain knowledge: create features based on expert understanding of the problem
- Mathematical transforms: polynomials, interactions, logarithms of existing features
- Aggregations: rolling windows, group-by statistics, time-based features
- Encodings: target encoding, frequency encoding, embedding representations
- Selection: keep only features that improve model performance
- Model-critical: often more impactful than algorithm choice
- Domain-dependent: best features require understanding the problem context
- Creative: no fixed recipe, requires experimentation and intuition
- Computationally-expensive: complex features can slow training significantly
- Built from: EDA — EDA reveals what features to create
- Builds into: Data Modeling — engineered features are model inputs
- Related: Data Transformation — transformation is a type of feature engineering
- Related: Data Science — feature engineering bridges data and models
- Data leakage: creating features from test set information during training
- Overfitting: engineering too many features for small datasets
- Feature importance illusion: correlated features distort importance rankings
- Ignoring feature stability: features that change meaning over time