The quiet ways data leakage inflates model performance

Rows are not always independent

Repeated measurements from the same patient must be considered together when designing a split. Otherwise a model may recognise a person rather than generalise to a new patient. Site-level and temporal dependencies can create similar problems. Choose a split strategy that matches the intended prediction scenario.

Preprocessing is part of fitting

Imputation, scaling, feature selection and tuning can transmit information from a test set if performed before resampling. Fit these steps within each training fold. Preserve a final evaluation set that is untouched by tuning and model selection.

Ask when each feature becomes available

A discharge code, a future laboratory result or a treatment decision may be unavailable at the moment a prediction is intended. Build a feature-availability table with timestamps and ask a domain reviewer to challenge it. High predictive power from an implausible feature deserves investigation.

Make a leakage review repeatable

Keep a split manifest, a pipeline diagram and tests for identifier overlap. Re-evaluate when data linkage or refresh logic changes. Report the intended scope of generalisation honestly: performance in a historical sample does not by itself establish performance in a new organisation.

Sources & further reading

• tidymodels — Resampling

Sources checked 20 September 2026. Implementation commentary reflects a practical review perspective; source material may change.