Rows are not always independent

Repeated measurements from the same patient must be considered together when designing a split. Otherwise a model may recognise a person rather than generalise to a new patient. Site-level and temporal dependencies can create similar problems. Choose a split strategy that matches the intended prediction scenario.

Technical background: [1] [2]. Practical recommendations below are VSPJ Consulting commentary.

Preprocessing is part of fitting

Imputation, scaling, feature selection and tuning can transmit information from a test set if performed before resampling. Fit these steps within each training fold. Preserve a final evaluation set that is untouched by tuning and model selection.

Ask when each feature becomes available

A discharge code, a future laboratory result or a treatment decision may be unavailable at the moment a prediction is intended. Build a feature-availability table with timestamps and ask a domain reviewer to challenge it. High predictive power from an implausible feature deserves investigation.

Make a leakage review repeatable

Keep a split manifest, a pipeline diagram and tests for identifier overlap. Re-evaluate when data linkage or refresh logic changes. Report the intended scope of generalisation honestly: performance in a historical sample does not by itself establish performance in a new organisation.

Sources & further reading

  1. tidymodels — Resampling
  2. scikit-learn — Common pitfalls: data leakage

References reviewed 21 September 2026. Sources support the technical background; workflow recommendations are independent implementation commentary. Release-watch details are dated snapshots. Check current documentation before adoption.

Have a related programming or implementation challenge?

Discuss it with Sai Prasad ↗