Pilot Decision Gate

A pre-registered pass/fail rule for whether an AI feature actually helped, decided BEFORE the data arrives. That is the only version of the question that means anything: choosing the threshold after seeing the result is how every pilot passes.

Uses McNemar's test for paired outcomes, because the same ticket answered both ways is paired data, and treating it as two independent samples overstates significance. Exact binomial below 25 discordant pairs, chi-squared with continuity correction above.

Built after finding a live p-value bug in a production system: an erf approximation missing its 1/sqrt(2) scaling reported p = 0.05 at z = 1.163, where the true value is 0.122. Pilots that genuinely failed were reported significant, silently, in a product whose whole premise was verification.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function bootstrapLiftCI(pairs: Array<[number, number]>, iters = 5000, alpha = 0.05): BootstrapCI;
  export function evaluatePilotDecision(mc: McNemarResult, liftCI: BootstrapCI): PilotDecisionResult;
  export function mcnemar(b: number, c: number): McNemarResult;

  export type PilotDecision = "PASS" | "HONEST_NO" | "INCONCLUSIVE";