Why
The reason applied results do not survive contact with production is almost never the model, and treating it as a modelling problem is how teams spend months on the wrong thing. A leak is any path by which information that would not exist at prediction time reaches the estimator during training. It does not announce itself. It shows up as a validation score that is too good, which is the one signal nobody investigates, because a good number looks like success rather than like a bug.
Three leaks account for most of it, and they are structural rather than clever. The first is temporal: data with an order was split at random, so the model was allowed to see the future while predicting the past. The second is grouped: the same entity appears on both sides of the split — the same user, the same document, the same address — so the model memorises the entity rather than learning the pattern, and scores brilliantly on entities it has already met. The third is preprocessing: a scaler, an imputer, a target encoder or a vocabulary was fitted on the whole dataset before splitting, so statistics from the test set were baked into the training features. The third is the most common and the least discussed, because it hides inside code that looks like data preparation rather than like modelling.
On-chain data triggers the first two simultaneously, which is why this belongs in this catalogue rather than in a textbook. Every record has a block timestamp, so it is ordered; every record is attached to an address, so it is grouped. A random split violates both at once. And the labels here usually depend on a forward window — did this address get drained within thirty days, did this position get liquidated before expiry — which means a training label overlaps in time with the validation period even after a clean date cut. That is exactly the case purged cross-validation with an embargo exists for: drop the training samples whose label horizon reaches into the validation window, and leave a gap after it before training resumes.
The honest way to report any of this is a gap, not a score. A single number from a leaky split is unfalsifiable — nobody can tell from the outside whether it is skill or contamination. Two numbers from two splits are a measurement, and the distance between them is the only part of it that carries information. The gap is also the cheapest possible experiment, because it needs no new data, no new features and no tuning: the same estimator, evaluated twice, honestly.
And it is worth naming what the discipline buys beyond correctness. A pipeline that splits properly is a pipeline that can be re-run when the data changes, because the boundary between what is known and what is being predicted has been written down explicitly instead of assumed. That boundary is the same thing a backtest needs, the same thing an audit asks for, and the same thing that makes a result reproducible six months later by somebody who was not there.
How it works
Three leaks, and the split that closes each
| Leak | How it gets in | What the score looks like | The fix |
|---|---|---|---|
| Temporal | Random split on ordered data | Excellent, and it degrades the moment it goes live | Split by time; never shuffle |
| Grouped | The same entity on both sides | Excellent on seen entities, chance-level on new ones | Split by group (address, user, document) |
| Horizon overlap | Labels depend on a forward window that reaches past the cut | Slightly too good; survives a naive date split | Purge the overlapping training samples, then embargo a gap |
| Preprocessing | Scaler / imputer / encoder fitted before the split | Uniformly and mildly too good across all folds | Fit inside the fold, always |
The horizon overlap row is the one that survives the obvious fix, which is what makes it worth naming separately. A team that has already learned to split by date will still leak if the label looks thirty days ahead and the training data runs up to the cut — the last thirty days of training labels are partly about the validation period.
Why the answer is a gap and not a score
| Split | What it measures | Honest use |
|---|---|---|
| Random | An upper bound produced by contamination | Only as the top of the gap |
| Time-ordered | What the model knows about the future given the past | A real estimate, still optimistic if grouped |
| Time + group + embargo | What it would have scored deployed | The number to report |
Report all three. A single honest number invites the question "could a better model do more?"; three numbers answer a better question — how much of the apparent performance was never real. And in a portfolio or a write-up, the gap is more persuasive than the score, because it demonstrates the one thing a reader cannot verify from outside: that you went looking for your own contamination.
The step people skip, stated concretely
Anything fitted belongs inside the fold. Not just the model — the scaler's mean and variance, the imputer's median, a target encoder's per-category averages, a vocabulary or tokenizer built from the corpus, a feature-selection step that looked at the labels, and any resampling done to balance classes. The test for whether something belongs inside is simple: would this quantity be computable on the day of prediction, using only what existed then? If not, computing it once over the whole table has already leaked.
Where it lands in this project
The pairing is deliberate. the-70-has-to-be-wrong is about scoring a probabilistic output honestly; this card is about whether the scoreboard was contaminated before scoring began. The two failures compose badly — a leaked split produces a model that looks well-calibrated on data it has effectively already seen, and the calibration plot will look fine right up until deployment. Fix the split first, because a calibration measured on a leak is a measurement of nothing.