Grounded Abstain

The bug this exists to prevent

Confabulation does not look like a failure. Ask a document assistant for a number that was never ingested — next year's budget, a counterparty's internal figures, a lease nobody uploaded — and it does not say "not found." It produces a figure of the right magnitude, in the right format, with the right units, in a confident sentence.

There is nothing downstream that can catch it. The answer is well-formed, so no schema check fires. The number is plausible, so no range check fires. The citation may even be real — a document about the right subject, that simply does not contain that figure.

This was measured, not assumed: on a 42-question golden set the single largest failure class was not bad retrieval or bad ranking, it was answering confidently about material that was never in the corpus at all.

Layer 1 — the probe, before the model speaks

probeSignal() classifies what retrieval came back with as empty, weak, or ok, and buildGroundingSteering() turns that into instructions the model actually gets.

The threshold is calibrated rather than picked. Against a real corpus, queries whose subject was genuinely absent still matched something — full-text search almost always does — but topped out around 0.0005. Genuinely answerable document questions scored **0.0286 to 0.30**. WEAK_RANK_THRESHOLD = 0.01 sits between those with margin on both sides.

That gap is the entire reason this works, and it is why "no results" is the wrong test: the absent-subject case is not zero hits, it is weak hits. A system that only abstains on an empty result set abstains almost never.

Layer 2 — check the figures against the context

extractFactClaims() pulls the numeric claims out of a draft answer and claimSupportedByStructured() checks each one against what the tools actually returned.

Two details that took getting wrong to find:

• Digit-boundary containment. Searching for 412500 inside the context with a plain
• substring test matches 14125001. The check is boundary-aware, or it silently approves a
• figure that merely shares digits with a real one.
• The year is part of the claim. A 2026 figure quoted as the 2027 figure is supported by
• every naive text search and is still wrong. Claims carry their qualifiers.

Three modes, because you cannot ship this straight to on

parseGroundedAbstainMode() reads off / shadow / on.

shadow is the one that matters: it runs the whole decision and logs what it would have suppressed, changing nothing. An abstention gate turned straight on either suppresses good answers nobody sees, or suppresses nothing and you never learn. Shadow first, read the log, then enforce.

What this does NOT do

No retrieval, no ranking, no embeddings, no model calls. It does not decide whether a document is relevant — it decides whether what you got back is strong enough to answer from, and whether the answer's figures survive the context. Bring your own retriever.

It cannot catch a confident non-numeric falsehood. The figure check is exactly that: figures.

Verified

14 tests covering the probe thresholds, the steering text, claim extraction, digit-boundary containment, year-qualifier mismatch, and the three modes.