Numerically Stable Streaming Statistics

Welford variance and compensated summation, both mergeable for sharded or parallel computation.

THE SILENT FAILURE. The textbook variance formula -- mean of squares minus square of the mean -- is catastrophically wrong for data with a large mean and small spread. It can return a NEGATIVE variance, and code that then takes a square root gets NaN, which propagates through a dashboard as a blank rather than an error. More often it returns a small positive number wrong by orders of magnitude, and nobody checks, because variance has no obvious expected value.

Naive summation is the same story: adding a million floats left to right accumulates error proportional to the number of terms, so a total that should be exact is off in the last digits -- which matters the moment it is compared for equality or used as a divisor.

Variance can never return a negative number. Both accumulators MERGE correctly, which is where naive implementations break and so has its own tests. Exact integer summation is available where inputs are integers, because exactness beats compensation when you can have it. NaN and Infinity are REFUSED rather than absorbed, since a poisoned running total leaves no trace of where it came from.

THE CLAIM IS PROVEN, NOT ASSERTED: the suite compares against the naive formula on a pathological dataset and the README prints both numbers. An independent verifier checked the results against exact rational arithmetic and ran ten million pushes hunting for a negative variance, finding none.

VERIFIED: 73 tests. Every mutation was observed FAILING before the source was restored -- a test never seen to fail is a decoration. Independently reviewed by a verifier whose job was to find what is wrong, not to agree.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function createStreamingStats(): StreamingStatsState;
  export function streamingStatsPush( state: StreamingStatsState, value: number ): StreamingStatsState;
  export function streamingStatsPushAll( state: StreamingStatsState, values: Iterable<number> ): StreamingStatsState;
  export function streamingStatsMerge( a: StreamingStatsState, b: StreamingStatsState ): StreamingStatsState;
  export function streamingStatsSnapshot( state: StreamingStatsState ): StreamingStatsSnapshot;
  export function createExactIntegerSumState(): ExactIntegerSumState;
  export function exactIntegerSumPush( state: ExactIntegerSumState, value: number ): ExactIntegerSumState;
  export function exactIntegerSumPushAll( state: ExactIntegerSumState, values: Iterable<number> ): ExactIntegerSumState;
  export function exactIntegerSumMerge( a: ExactIntegerSumState, b: ExactIntegerSumState ): ExactIntegerSumState;
  export function exactIntegerSumValue(state: ExactIntegerSumState): bigint;
  export function exactIntegerSumValueAsNumber(state: ExactIntegerSumState): number;
  export function describeNonFinite(value: unknown): string;