Search Degradation Signal

The incident this exists for

A production model key hit its usage cap. Query understanding fell back to the deterministic regex path and stayed there.

Nothing broke in a way anyone could see. Results still came back. HTTP status was still 200. The only tell was a mode: "lexical_fallback" field on a response nobody reads.

92.7% of the previous two days' searches had already degraded before anyone noticed.

That is the failure mode of every graceful fallback: it is designed to be invisible, and it succeeds. The service is up, the users are served, the quality quietly halved, and the monitoring was built to answer "is it up?"

Rule 1 — ok stays true while status says degraded

healthPayload reports ok: true and status: "degraded" at the same time, because both are true — the service is serving and it is serving badly.

Collapsing that into one boolean is what produced the incident. ok: false would page someone for a service that is working; ok: true alone says nothing is wrong. There is a test named for this — "if status cannot differ from ok, this whole module is theatre" — which is what the original { ok: true } payload was.

Rule 2 — observed and predicted are separate fields

degraded is what broke while running. misconfigured is what was never going to work on this instance at all.

They are never merged, because they are different pages for whoever gets woken up. One means "something changed"; the other means "this deployment was born broken."

Rule 3 — the boot check, because serverless hides the truth

A misconfigured instance is knowable before any traffic proves it. That matters more than it sounds: serverless runs many short-lived instances, so a monitor polling /health keeps hitting instances that have served no search yet and keeps being told everything is fine — on a deployment where every search is guaranteed to degrade.

Rule 4 — degradation clears, so a blip is not permanent

recordHealthy clears the record. Without it, one transient failure marks the service degraded forever and the signal becomes noise you learn to ignore.

The predicate is injected

healthPayload(base, smartPathAvailable) takes "is the smart path available?" as an argument. This package reads no environment variables and carries no provider SDK.

Pass the same predicate your search path actually branches on — not a second reading of the environment. Two copies of "is a model available", in two places, is precisely the shape that lets a keyless instance pass a green test run while degrading every real request.

> Changed from the original. Upstream this read ANTHROPIC_API_KEY and AMOS_AI_MODE > directly. Those are one provider's names, and a package that reads them is not reusable — so > the predicate moved to the caller and the two env cases collapsed into one boolean, since both > produce identical behaviour. The inherited tests were updated to pass the predicate explicitly > rather than manipulate the environment.

What this does NOT do

No searching, no ranking, no alerting, no metrics export. It is an in-process record plus a health payload. It does not persist across restarts — deliberately, since a restarted instance has not yet observed anything.

No paging or notification. Wire status into whatever you already run.

Verified

13 tests: ok-with-degraded, naming what and why, separate observed and predicted fields, the boot check with no traffic, recovery clearing the record, and preservation of caller-supplied fields.

Not covered: nothing tests behaviour across instances or restarts, because there is none — the record is per-process by design.