Models that perform perfectly on training data but fail on new data are useless in practice — they’ve memorized noise instead of learning patterns.
When a model learns training data too well, including noise and idiosyncrasies, resulting in poor generalization to new data.
- Model has high capacity (too many parameters relative to data)
- Trains to minimize training error, reaching near-zero training loss
- Model captures noise, outliers, and sampling artifacts as “patterns”
- On test data, these spurious patterns don’t generalize
- Gap between training and test performance becomes large
- Capacity-driven: more parameters + less data = higher overfitting risk
- Detectable: large train-test performance gap is the signature
- Reducible: regularization, more data, or simpler models can fix it
- Universal: affects all machine learning models to some degree
- Built from: Train-Test Split — the tool to detect overfitting
- Contrasts with: Underfitting — opposite problem (model too simple)
- Related: Supervised Learning — overfitting is a supervised learning risk
- Related: Cross-Validation — helps detect and mitigate overfitting
- Overfitting can happen even with simple models on very small datasets
- Data augmentation and regularization can mask underlying overfitting
- Validation set overfitting: tuning hyperparameters too aggressively on validation set
- Multiple comparison problem: testing many models increases chance of “significant” overfitting