clip-token-truncation-guard

CLIP's text encoder holds 77 tokens: 75 for your words plus two markers. Write more and the tail is dropped in silence — the image still arrives, still plausible, and the clause you added because the first four attempts were wrong never reached the model. This counts exactly, using the real byte-pair encoder against the vocabulary your own pipeline already ships, and names the text that did not survive. An ordinary prompt ending 'with a canon eos r5 and an 85mm f/1.2 lens' comes to 89 tokens: the camera body is cut in half and the lens spec is gone. The count is not the word count, because CLIP gives every digit its own token — '1920x1080 35mm f/1.4' is nineteen tokens, not four. It refuses to estimate: without a vocabulary it errors and says where to find one, since a guard that is sometimes wrong about the limit is the same silent failure wearing a hat. Verified against the reference tokenizer on 40 cases including emoji, accents, CJK, Cyrillic, digit runs and the literal marker text; all 40 agree. 19 tests run anywhere, 5 more against a real tokenizer.

Contributed by a community creator from https://github.com/michaelcho87/clip-token-truncation-guard at commit 9dde080edb9e. Packaged from the approved paths only: src, test, README.md, LICENSE, package.json.