Search Relevance Scoring

BM25 and Reciprocal Rank Fusion that refuse to produce a confidently wrong ranking. A scoring library, not an index -- it stores nothing and retrieves nothing; tokenization, stemming and stop-word policy stay the caller's decisions.

THE SILENT FAILURE. A search ranking is never obviously broken. It returns ten results, they all look related to the query, and nobody can tell that the right answer was eleventh. Ordinary BM25 code hits four well-documented mechanisms that produce this, and none of them raise an error.

NEGATIVE IDF. The classic Robertson/Sparck-Jones inverse document frequency goes negative once a term's document frequency passes about half the corpus -- a document that matches that term then scores lower than a document that does not match it at all. On a small or domain-specific corpus this is routine, not an edge case, and there is no warning. This library computes log(1 + (N - df + 0.5)/(df + 0.5)), matching Apache Lucene's BM25Similarity, and never computes the classic formula anywhere, including internally.

PER-SHARD STATISTICS. Document frequency and average document length are properties of the whole corpus. Computed separately per index shard and merged without recomputing them, every shard's documents end up scored on a different numeric scale -- each shard's own top-k looks perfectly ranked, and no single shard has the information to notice. corpus.scope must be declared 'global'; 'per-shard' is refused outright.

FUSING SCORES OF DIFFERENT SCALES. Hybrid search adds an unbounded BM25 score to a cosine similarity bounded 0 to 1. The BM25 side dominates completely and the vector side becomes decoration; a weight tuned to fix the balance on one corpus silently stops meaning anything once term statistics shift on the next. Reciprocal Rank Fusion combines ranks instead of scores, so it has no scale to distort.

MISSING TERM STATISTICS DEFAULTED TO ZERO. If a query term has no entry in the caller-supplied document frequencies, defaulting to df=0 silently hands that term the corpus's maximum possible IDF -- turning 'we forgot to compute this' into 'this term is maximally rare and should dominate the ranking.' This library throws a MISSING_TERM_STATISTICS error instead of ever defaulting it.

VERIFIED: 68 tests, re-measured by running the suite, 10/10 mutations caught, no survivors.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function computeNonNegativeIdf(documentFrequency: number, totalDocuments: number): number;
  export function scoreBM25( query: readonly string[], document: BM25Document, termDocumentFrequencies: TermDocumentFrequencies, corpus: CorpusStats, params: BM25Params = {} ): BM25Explanation;
  export function rankBM25( query: readonly string[], documents: readonly BM25Document[], termDocumentFrequencies: TermDocumentFrequencies, corpus: CorpusStats, params: BM25Params = {} ): readonly BM25Explanation[];
  export function combinePrecalibratedScores( scoreSets: readonly WeightedScoreSet[], acknowledgement: CalibrationAcknowledgement ): readonly CombinedScore[];
  export function reciprocalRankFusion(rankedLists: readonly (readonly string[])[], options: RrfOptions =;
  export function assertNonEmptyString(value: unknown, label: string, code: RelevanceScoringErrorCode): asserts value is string;
  export function assertFiniteNumber(value: unknown, label: string, code: RelevanceScoringErrorCode): asserts value is number;
  export function assertNonNegativeInteger(value: unknown, label: string, code: RelevanceScoringErrorCode): asserts value is number;
  export function assertPositiveInteger(value: unknown, label: string, code: RelevanceScoringErrorCode): asserts value is number;
  export function assertPositiveFiniteNumber(value: unknown, label: string, code: RelevanceScoringErrorCode): asserts value is number;
  export function assertUnitRange(value: unknown, label: string, code: RelevanceScoringErrorCode): asserts value is number;
  export function validateCorpus(corpus: CorpusStats): void;

  export type TermDocumentFrequencies = Readonly<Record<string, number>>;