Rows are not always independent
Repeated measurements from the same patient must be considered together when designing a split. Otherwise a model may recognise a person rather than generalise to a new patient. Site-level and temporal dependencies can create similar problems. Choose a split strategy that matches the intended prediction scenario.
Technical background: [1] [2]. Practical recommendations below are VSPJ Consulting commentary.
Preprocessing is part of fitting
Imputation, scaling, feature selection and tuning can transmit information from a test set if performed before resampling. Fit these steps within each training fold. Preserve a final evaluation set that is untouched by tuning and model selection.
Ask when each feature becomes available
A discharge code, a future laboratory result or a treatment decision may be unavailable at the moment a prediction is intended. Build a feature-availability table with timestamps and ask a domain reviewer to challenge it. High predictive power from an implausible feature deserves investigation.
Make a leakage review repeatable
Keep a split manifest, a pipeline diagram and tests for identifier overlap. Re-evaluate when data linkage or refresh logic changes. Report the intended scope of generalisation honestly: performance in a historical sample does not by itself establish performance in a new organisation.
Sources & further reading
References reviewed 21 September 2026. Sources support the technical background; workflow recommendations are independent implementation commentary. Release-watch details are dated snapshots. Check current documentation before adoption.
Have a related programming or implementation challenge?
Discuss it with Sai Prasad ↗