Lead Score Leakage

The defect

Someone assembles a feature table from the CRM to predict "will this lead convert". It contains demo_requested_at, mql_date, opportunity_stage. The backtest reports 94%.

Every one of those is populated as a consequence of conversion. The model has learned that leads which requested a demo request demos. In production — where the field is empty at the moment you need a score — it is worth nothing.

This is the most common defect in home-grown scoring, and it is invisible *because it makes the metrics look excellent.*

  auditFeatures([{ name: "demo_requested_at", population: "post_outcome" }]);
  // [{ severity: "blocking", detail: "…the model learns the answer from the answer.
  //    Backtest accuracy will look excellent and mean nothing…" }]
  
  trainable(findings); // false

Leakage is blocking, not a warning. A warning on a leaking feature gets read as "a bit optimistic" and shipped anyway; the honest description is that the evaluation number is meaningless, and the audit should stop the release.

The one nobody catches

sales_activity features — call_count, emails_logged, owner_assigned. A rep logs a call only for leads they already like, so the feature encodes the rep's opinion. The model reproduces the team's existing preference and is then cited as objective evidence for it.

Warned, not blocked: sometimes it is genuinely the best signal you have. But it should be a decision, not an accident.

Decay, and why it is exponential

A threshold ("older than 30 days is cold") makes a lead's rank jump discontinuously overnight, which surfaces as a queue that reshuffles itself for no visible reason. Halving is smooth: a 90 from last week still beats a 60 from today, and a 90 from last quarter does not.

Withholding is not ranking low

  applyDecay(lead, { now, maxUnknownFeatures: 2 });
  // { withheld: true, reason: "3 feature(s) unknown — too little is known to rank this
  //   lead, which is not the same as ranking it badly" }

A lead you know nothing about is not a bad lead. Ranking it low is a claim you cannot support and it buries records that only need enrichment. rankLeads returns ranked and withheld separately so the second list reads as a data problem.

Missing-as-zero is flagged for the same reason: an unknown company size scoring identically to a company of size 0 sorts incomplete records to the bottom silently.

What it does not do

| not included | why | |---|---| | Training or fitting a model | it audits the feature set and decays the output | | Calibration | genuinely valuable and genuinely different; needs outcome data over time | | Deciding what a good lead is | that is your business, not a library's |

FeaturePopulation is a hand-classification — nothing here can infer from a column name whether a field is populated before or after the outcome. That judgement is the work; this makes it explicit and enforces it.