Public Endpoint Rate Limiter

The bug this exists to prevent

Public endpoints get one global limit, usually a memorable number, applied uniformly. That number is wrong in both directions at once.

It is far too generous for the endpoint that costs money. A search path that spends a frontier-model call per request, at a limit tuned so ordinary browsing never trips it, is an uncapped bill with a rate limit in front of it — and the failure is not an outage but an invoice.

It is simultaneously too tight for the cheap reads, so real users hit 429s on endpoints that cost nothing to serve, and everyone learns to treat rate limiting as noise.

Budgets here are assigned by cost: model-calling paths get the tight budget, cheap reads stay generous, and there is a test asserting the expensive budget is meaningfully tighter — not tighter by a token amount that satisfies a reviewer and changes nothing.

Getting the client identity right

The client key comes from the right end of the x-forwarded-for chain, not the nearest proxy. Read the wrong end and every request appears to come from your own load balancer, so the limiter either throttles all users as one client or, worse, never fires because the key is constant and the count resets.

This is the detail that makes a limiter look installed while doing nothing.

The window admits exactly the budget

Allows up to the limit, then refuses — asserted at the boundary, since an off-by-one is either a free extra request forever or one fewer than documented, and neither shows up in normal use.

Denials carry the retry delay, so a well-behaved client backs off rather than spinning. A 429 with no Retry-After turns a limiter into a load generator.

What this does NOT do

In-process only, and that is a real limit. Each instance has its own counters, so N instances means roughly N times the effective budget. For a serverless deployment with many short-lived instances, that multiplier can be large. This is a guard against abuse and runaway loops, not a precise global quota — use a shared store if you need exactness.

No authentication, no per-user quotas, no billing integration, no middleware for any particular framework.

Related

rate-limit-policy in this catalogue is the complementary half: it defines what the limits should be per bucket and does no counting. This one counts and decides in-process. They come from different codebases and are not designed to compose — pick the one matching the problem you have.

Verified

13 tests: budget assignment by path cost, model paths on the tight budget, cheap reads left generous, the expensive budget meaningfully tighter, the window admitting exactly the budget and no more, client-key extraction from the correct end of the proxy chain, and the retry delay on denial.

Not covered: no multi-instance test, and no test of clock changes mid-window.