Why it matters

Data leakage causes ML projects to fail in production despite great offline metrics. Getting splits right is essential to honest evaluation.

Advertisement

The architecture

Train set: 70-80% typically. Model learns weights here.

Validation set: 10-15%. Tune hyperparameters, decide model architecture, early stopping.

Test set: 10-15%. Report final metrics. Never used during development.

Data split rolesTrain (70-80%)fit weightsVal (10-15%)hyperparamsTest (10-15%)final evalTest set touched only once at end; iterating on test leaks info and inflates metrics
Three-way split.
Advertisement

How it works end to end

Time-based split: for time-series or streaming data, split by time. Simulates production where you predict future.

Cross-validation: k-fold rotates val set through data. Estimates generalization when data limited.

Leakage sources: preprocessing on full data before split, target-derived features, temporal leakage (using future info to predict past).