The Uptime workflow failed ten consecutive runs. Those failures were reported as CI trouble, and they were not. Production really was answering db_ok: false, from 00:45:23 UTC until at least 10:06.
The reason this took nine hours to resolve rather than nine minutes is that every instinct about where to look was wrong, because the database was completely fine the entire time.
The database was fine the entire time
Every container up 24 hours. Disk at 22%. SELECT 1 through postgres-meta fine. And /rest/v1/jobs?limit=1 answering a plain curl in 90 milliseconds, using the same service key the backend uses.
Caddy, Kong and PostgREST logged nothing at all for 20 minutes. Not errors - nothing. Which is the actual clue, because it means the requests never left App Runner:
httpcore.LocalProtocolError: Max outbound streams is 250, 250 openOne connection, 250 streams, none of them coming back
postgrest/_sync/client.py:102 hardcodes http2=True.
HTTP/2 multiplexes every request onto a single TCP connection, under a cap on concurrent streams, 250 as advertised here. Driven from many threads through asyncio.to_thread, streams were opened and never released, and httpcore never resets that counter. Once it reaches the cap the process fails every later request, permanently, for as long as it lives.
The measurement that says leak rather than saturation
This is the part worth keeping, because saturation and a leak look identical while traffic is high.
There were 1,665 failures over nine hours. And there were still 8 in a 12-minute window while traffic was near zero. Streams in use drain when the work stops. Leaked streams do not.
Two things follow from that immediately. No amount of concurrency limiting would have fixed this, because the streams were not in use. And only a restart clears an already-wedged process, because nothing in the process is ever going to give those 250 slots back.
What readers actually saw
Partly shielded, and not fully. The instance served 3,168 answers from cache and 45 stale ones, which is a caching layer doing exactly the job it was added for. But 71 reads failed outright, and those readers got zero results with no indication that anything was wrong.
The fix, and the trap inside the fix
HTTP/1.1 cannot fail this way. It pools connections under a bound, so a request that outlives its connection costs that one connection rather than poisoning a shared multiplex. The theoretical throughput loss does not apply here either: the box is one hop away and every call is small. Measured after the change, against production, jobs?limit=1 at 0.130s and rpc/search_jobs at 0.390s.
The trap is in how you get there. The obvious move is to inject a client through SyncClientOptions(httpx_client=...). That does not work, and it fails in the most expensive possible way:
# postgrest's own constructor
self.session = http_client or Client(base_url=..., headers=...)A client passed in never receives base_url or the apikey header. Every call would then 401, and the repository reports a 401 as the database being unreachable. Which is the exact symptom being fixed. You would ship the fix, watch the symptom persist, and have no reason to suspect the fix itself.
So the session is swapped after construction instead, copying base_url, headers and timeout off the session supabase already built. Four of the nine new tests exist only to pin that: base_url, apikey, Authorization and Accept-Profile all survive the swap.
One more test guards the premise rather than the behaviour. It asserts that a default postgrest client really is on HTTP/2, so that if the library ever stops forcing it, this suite fails loudly instead of passing for the wrong reason.
The nine tests fail against the old code, and one of them fails on a NameError from a missing logging import. That is how the missing import got found rather than shipped.
What I have not established
The root cause of the leak itself. HTTP/1.1 removes the failure mode; it does not explain which code path fails to release a stream. Sentry's httpx wrapper is in every frame of the traceback and is the obvious candidate, but that is a suspicion and not a finding.
Removing a failure mode without explaining it is still a fix. It is just not an answer, and the difference matters a great deal to whoever is next tempted to turn HTTP/2 back on.
Two things left deliberately: storage3/_sync/client.py:80 forces http2=True in exactly the same way. Storage is not on this app's hot path, so it is not patched, but it is the same landmine with the same pin in it. And AutoDeploymentsEnabled is true from main, so any push deploys the backend - worth knowing before pushing late in the day, because the first big scheduler fan-out after a deploy is what exposed this in the first place.