Batch Schema Drift Detector

Infers a structural schema from records and diffs two batches into typed drift findings.

THE SILENT FAILURE. A batch load is built once against the shape the data had that day. When an upstream producer adds a column, renames one, changes a type, or starts sending null where it never did, the load usually SUCCEEDS. The new column is dropped because the target has no place for it. The retyped column is coerced -- 00123 becomes 123 and loses the leading zeros that made it an identifier, or a numeric string becomes NaN and lands as null. The pipeline reports success and the warehouse diverges from the source, discovered weeks later by someone querying a column that is now half null.

Findings are typed: field added, removed, type changed, nullability widened, and a rename CANDIDATE that is reported for confirmation and never applied automatically. Inference fails closed: a field seen only as null is UNKNOWN, not a null type, and never reported as a type change when real values appear later.

A REAL BUG FOUND BEFORE RELEASE. Cardinality divided a CAPPED distinct count by an UNCAPPED total, so any identifier column past 500 distinct values was classified as LOW cardinality -- the opposite of the truth -- and because cardinality feeds the rename scorer, genuine renames silently vanished from the report on ordinary batch sizes. It now reports UNKNOWN when the cap was hit, and everything consuming cardinality handles UNKNOWN without treating it as a mismatch. Sampling is visible in the result, because a schema inferred from a truncated sample that claims completeness is this product's own failure mode.

VERIFIED: 77 tests. Every mutation was observed FAILING before the source was restored -- a test never seen to fail is a decoration. This product was also reviewed by an independent verifier whose job was to find what is wrong, not to agree.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function diffBatches( baseRecords: readonly unknown[], incomingRecords: readonly unknown[], options: DiffBatchesOptions = {}, ): DriftReport;
  export function diffSchemas(base: SchemaSnapshot, incoming: SchemaSnapshot, options: DiffOptions = {}): DriftReport;
  export function inferSchema(records: readonly unknown[], options: InferSchemaOptions = {}): SchemaSnapshot;
  export function nameSimilarity(a: string, b: string): number;
  export function typeCompatibility(a: FieldSchema, b: FieldSchema): number;
  export function cardinalityCompatibility(a: FieldSchema, b: FieldSchema): number;
  export function scoreRenameCandidate(fromField: FieldSchema, toField: FieldSchema): RenameScore;
  export function matchRenameCandidates( removed: readonly FieldSchema[], added: readonly FieldSchema[], threshold: number, ): RenameScore[];

  export type ConcreteType = "string" | "number" | "boolean" | "object" | "array" | "unknown";
  export type InferredType = ConcreteType | "mixed";
  export type CardinalityBucket = "unknown" | "constant" | "low" | "high" | "unique-like";
  export type Severity = "breaking" | "compatible" | "info";