Prompt-Cache-Aware Batcher

Prompt caching keys on an EXACT prefix. Requests sharing a large system prompt hit the cache only if they are issued while the entry is alive and the prefix matches byte for byte. Naive round-robin or arrival-order dispatch interleaves requests with different prefixes, so each one misses or evicts the other’s entry and the cache does nothing at all. Nothing errors. The only symptom is cost.

It groups a queue by exact prefix and orders the groups so each is issued contiguously, inside the TTL window. A group larger than the window is split -- and the split is REPORTED, because a silently split group means a second cache write at 1.25x and the caller should know it happened.

IT REFUSES TO CLAIM A BENEFIT THAT CANNOT OCCUR. Below a model’s minimum cacheable prefix length, nothing caches at all -- which is why small prompts never seem to benefit. Reporting a saving for a prefix under that minimum is the bug, so it reports none.

The projection -- estimated writes, reads and the cost delta versus arrival order -- is labelled an ESTIMATE, because actual hits depend on wall-clock timing the planner does not control. A confident number here would be the same overclaiming this catalogue is sold against.

Pure and deterministic: takes a queue and a clock value, returns a plan. No I/O, no timers, fully testable.

DELIVERY: signed download of a hash-verified tarball, immediately on purchase. Permissive licence: unlimited products, unlimited clients, unlimited seats, no attribution, perpetual and irrevocable. One restriction, do not republish the source as source.

Interface

What you call, and what comes back. Types and signatures only — the implementation ships with the source.

  export function isKnownModel(model: string): boolean;
  export function planBatch( queue: QueuedRequest[], clockMs: number, options: PlanBatchOptions = {}, ): BatchPlan;

  export type CacheTtl = "5m" | "1h";
  export type CacheAction = "write" | "read" | "uncacheable";