Trafilatura Extraction Confidence

The confidence gate trafilatura does not have. Decides whether an extraction should be TRUSTED as article content, from signals you compute. Pure function: no I/O, no HTML parsing, no dependency on trafilatura.

Built for trafilatura 2.2.0 (Apache-2.0). Not affiliated with or endorsed by the trafilatura project.

THE SILENT FAILURE. extract() returns a string or None, and a truthy string is NOT evidence that an article was found. Measured:

  real article               -> 'T\nReal sentence about the subject...'
  nav and footer only        -> 'Home\nAbout'
  paywall stub               -> 'Headline\nSubscribe to continue reading.'
  empty body                 -> None

Two of those are not articles and return perfectly ordinary strings that pass every `if text:` check ever written. A pipeline that stores them has replaced the article with a navigation menu, or with the reason it could not read the article -- and because a nav bar sits on every page of a site, an index built from it fills with the same menu repeated thousands of times.

THERE IS NO CONFIDENCE FIELD, and that is verified by enumeration rather than asserted. All 21 fields of bare_extraction(with_metadata=True) were listed; not one expresses confidence, score, probability or quality. The signals must be computed by the caller, and this names exactly which ones and what each means.

THE MEASUREMENT THAT DECIDES THE DESIGN. A paywall stub is 39 characters. A genuinely short article is 41. Length CANNOT separate them, so the obvious guard -- len(text) > 50 -- admits the paywall notice and rejects the real article, silently, forever. This package therefore refuses to answer on length alone when the text is short, and asks for the sentence count, which does separate them: 1 against 6. The link-text ratio is the cleanest signal measured: 0.90 for the nav-only page against 0.00 for everything real.

IT REPORTS RATHER THAN JUDGES WHERE IT CANNOT KNOW. Short text that reads as prose is reported as below the length floor, not as a stub, because a genuine short piece and a teaser above a paywall are indistinguishable from the text alone. None is reported as its own outcome, because it is the one thing trafilatura signals unambiguously and therefore the only one a `if not text:` guard catches -- which is exactly why the others reach production.

THRESHOLDS ARE POLICY. Defaults are the measured figures and are overridable, and every verdict names the threshold it applied so it can be argued with rather than merely obeyed.

ON TRAFILATURA'S LICENCE: it is Apache-2.0 from v1.8.0 onward and GPLv3+ before that. Because this package does not depend on it at runtime there is no version to pin and no way for an old GPL release to be pulled into your build by installing this.

VERIFIED: 19 tests, measured by running the suite, every mutation observed FAILING before restore.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function evaluateExtraction( signals: ExtractionSignals, thresholds: Thresholds = {}, ): Verdict;

  export type Verdict = | { status: "TRUSTED";