Rate Limit Policy

The bug this exists to prevent

The usual rate limiter is one global budget: N requests per minute per org. It fails in a specific and expensive way the first time you have more than one expensive surface.

A burst of image generation, or an agent retrying a research call, consumes the shared budget. Now the chat assistant is throttled too. The user is not doing anything wrong and has no idea why the app stopped answering — the feature they are using is fine, and the limit they hit belongs to a feature they never touched.

The second failure is the mirror image: limits tuned tight enough to contain generation make ordinary chat unusable, so they get raised until they contain nothing.

Rule 1 — one bucket per route family

Buckets are declared separately — chat, voice, research, media generation, an external agent bridge — so a burst on one cannot exhaust another. That is the whole design: these surfaces have costs that differ by orders of magnitude, and a single number cannot be right for all of them.

Rule 2 — the defaults are generous on purpose

These limits exist to stop a runaway loop, an abusive caller, or a leaked key from burning spend. They are not there to shape normal usage.

That distinction decides every number. A limit set where normal use occasionally trips it trains everyone to treat 429 as noise, and then the one that matters is ignored too. Every value is overridable from the environment, so tuning does not require a deploy.

Rule 3 — an external bridge has no per-user identity

A shared inbound key cannot attribute requests to individuals, so that bucket carries org and hourly ceilings and no per-user sub-limit — rather than a per-user limit that silently applies to a single synthetic identity and therefore throttles the integration as though it were one person.

Rule 4 — say how long to wait

Deny decisions carry the retry delay. A 429 with no Retry-After produces clients that spin, which converts a rate limit into a load generator — the failure mode where the limiter is the thing causing the outage.

What this does NOT do

No counting and no storage. This is the policy half only: given a bucket, its limits, and the current count, it decides. The durable counter — Redis, Postgres, whatever you already run — is yours, and it is the easy half.

No middleware, no framework integration, and no clock at all — not even an injected one. The functions work on the count you pass and the window's remaining TTL, which is what makes them testable without freezing time. Reading the counter and knowing its TTL is your storage layer's job.

Verified

14 tests covering per-bucket resolution, per-minute and per-hour boundaries, org versus user limits, the shared-key bucket having no per-user limit, environment overrides, the disable switch, and the retry delay reported on denial.

Not covered: no test exercises a real counter store, and there is no concurrency test. The functions are pure, so races — the hard part of rate limiting — belong entirely to the storage layer you supply. Buying this does not solve distributed counting.

Related

public-endpoint-rate-limiter in this catalogue is the complementary half: an in-process sliding window that actually counts and decides. This module defines what the limits should be and does no counting. They come from different codebases and are not designed to compose — pick the one matching the problem you have, rather than buying both expecting a system.