URL Dedup Key

A canonicalization-aware dedup key for crawled URLs, and a SAME / DIFFERENT / NOT_DETERMINABLE comparison between two of them. A pure function of the URL string: no network, no runtime dependencies.

THE SILENT FAILURE. Scrapy's default request fingerprinter, which its duplicate filter uses out of the box, fingerprints a URL without stripping marketing tracking parameters or an explicit default port. Two URLs that are unambiguously the same page get different SHA1s, so the crawler fetches the page again.

IT IS SILENT ON EVERY SIGNAL. Reproduced against a real CrawlerProcess and a real local HTTP server with zero settings changes: no exception, ZERO warning-or-above log records, and the dupefilter/filtered stat key ABSENT from stats entirely -- not zero, absent, because the filter only counts when it DOES detect a duplicate. There is nothing to notice. You pay for the extra fetches, the extra bandwidth and the extra rate-limit budget, and your dataset carries duplicate rows. The upstream issue asking for exactly this has been open since 2015.

WHAT IS SAFE TO STRIP BY DEFAULT, AND WHAT IS NOT. This is the line the product turns on, and it is drawn explicitly. Third-party ad-platform parameters -- 26 of them, utm_*, gclid, fbclid, msclkid, mc_cid, yclid and the rest -- are never a site's own routing state, so stripping them is safe generically and is ON by default. Trailing-slash equivalence, index-file equivalence and site-specific parameters are NOT: whether they change the page varies per site, so they are caller-supplied and OFF by default.

AND IT REFUSES RATHER THAN GUESSING. Where you have not told it whether a parameter is significant and it cannot know, the answer is NOT_DETERMINABLE, not SAME. A confident SAME on two genuinely different pages would silently drop a page from your crawl, which is worse than the duplicate it was meant to prevent.

IT ALSO DOES NOT RE-FIX WHAT IS ALREADY CORRECT: Scrapy already canonicalizes query-parameter order and path case, and this says so rather than claiming credit.

VERIFIED: 164 tests, measured by running the suite. Every mutation observed FAILING before restore. All 26 tracking parameters are individually pinned -- an adversarial reviewer found the README claiming that when only 4 actually were, and dropping any one of them now fails a test.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.