We need to predict an output variable (price, category, probability) given input features, but the relationship is too complex to code manually.
Machine learning paradigm where models learn a mapping from input features to output labels using labeled training examples.
- Collect labeled data: input features X and correct output labels y
- Choose model family: decision trees, linear models, neural networks, etc.
- Define loss function: measure prediction error (MSE for regression, cross-entropy for classification, impurity for trees)
- Optimize: adjust model parameters to minimize loss on training data
- Validate: assess generalization to unseen data
Decision trees use a different approach: instead of optimizing parameters, they recursively partition the data using attribute tests (Information Gain or Gini Index) until pure subsets are reached. This makes them interpretable but prone to overfitting without pruning.
- Labeled data required: need ground truth for every training example
- Generalization goal: minimize error on unseen data, not just training data
- Bias-variance tradeoff: simpler models have high bias, complex models have high variance
- Inductive: learns specific-to-general mapping from examples
- Built from: Data Modeling — supervised learning is a modeling approach
- Builds into: Regression — supervised task for continuous outputs
- Builds into: Classification — supervised task for categorical outputs
- Builds into: Decision Tree Structure — trees are a supervised algorithm
- Builds into: ID3 Algorithm — foundational decision tree construction method
- Related: Train-Test Split — supervised learning requires careful data splitting
- Related: Decision Tree Interpretability — trees offer inherent explainability
- Label noise: incorrect labels mislead the model
- Class imbalance: rare classes get ignored by models optimizing overall accuracy
- Concept drift: relationship between X and y changes over time
- Covariate shift: training and test data have different feature distributions