tokenx

johannschopplich/tokenx is an open-source project by johannschopplich: 📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle. Indexed here so it can be found — not resold.

It is free. Get it from the upstream repository: https://github.com/johannschopplich/tokenx

From the project's own README (excerpt, reproduced for discovery under its MIT license):

tokenx

Fast and lightweight token count estimation without requiring a full tokenizer.

Estimates are calibrated against OpenAI's o200kbase encoding – the tokenizer of all current GPT models. Counts for other LLM families will differ somewhat; the defaultCharsPerToken and languageConfigs options let you tune the heuristics for your model. For precise counts, use a full tokenizer like gpt-tokenizer.

Features • ⚡ 95%+ average accuracy, and no single sample below 90% • 📦 Just 2kB bundle size with zero dependencies • 🖥️ Bundled CLI – count, slice and split from the shell • 🌍 Multi-language support with configurable language rules • 🗣️ Built-in rules for accented scripts (German, French, Spanish, Slavic), Cyrillic, and Greek • 🀄 CJK (Chinese, Japanese, Korean) character handling • 😀 Emoji-aware pricing (emoji cost more tokens than their character count suggests)

Benchmarks

The following chart shows how close the estimates come to actual GPT token counts for different input texts:

Bars grow left when tokenx underestimates and right when it overestimates. The axis spans the ±10% per-sample deviation bound.

Accuracy depends on the kind of text, not its length: a short excerpt deviates about as much as the full document it came from. A holdout corpus that no ratio was ever fitted against is held to a looser ±15% bound, so a retune cannot silently overfit the chart.

Three cases are knowingly outside these bounds, all underestimates: • High-entropy strings – base64, hashes, digests: ≈-70%. Pricing them would cost every caller runtime for a case ordinary traffic rarely carries. • Traditional and classical Chinese – the hanzi rate is calibrated on contemporary simplified script: ≈-10% to -20%. • Scripts without a built-in rule – Arabic ≈-35%, Hindi ≈-30%,…