Latency regression misattributed to LLM retry-broadening; real cause was connection-pool contention

An ETL pipeline slowed from 19s/pair to 40-47s/pair after one "resilience hardening" commit changed three perf-relevant knobs at once: a Neo4j withSession auto-retry wrap, driver config (connectionAcquisitionTimeout 5s to 30s with maxConnectionPoolSize left at 20), and a broadened LLM callWithRetry predicate (429 + socket codes, 8 retries, 60s cap). The team reverted all three serially, and TWO commits 15 minutes apart each claimed to be "the actual cause" of the SAME slowdown, with no measurement between them. The retry revert left a header comment asserting "deliberately narrow: 5xx only, no 429" that a later commit silently contradicted by re-broadening the predicate — so the file documented a policy it did not implement, and nobody could tell whether the regression had returned. The real fix was the driver config (pool 20 to 40, drop the acquisition-timeout override). Narrowing the retry predicate back to 5xx-only was a coincidental, unverified revert that cost real resilience.

DIAGNOSE FROM DISTRIBUTION SHAPE, NOT A SUSPECT LIST. The diagnostics recorded "per-tier durations uniformly 40-47s with 7s spread." Uniform, low-variance, fixed per-unit overhead is the signature of a synchronous resource wait (pool acquisition). Retry-induced latency is heavy-tailed and bimodal: it fires on a minority of calls, so most units are unaffected and a few pay +30-80s. Retries CANNOT produce a uniform 7s spread. That single check exonerates the retry path, and it is free — it comes from the accused commit's own logged evidence.

DOUBLE-COUNTING IS THE TELL. The driver revert said "delta = ~24s/pair" against a "19s/pair" baseline; 19 + 24 = 43 already accounts for the whole regression. The retry revert 15 minutes later then claimed sole causation for the same delta. Two commits cannot each be the whole cause of one number.

A REVERT MESSAGE IS A HYPOTHESIS, NOT A MEASUREMENT. "The actual cause" in a commit body carries no evidentiary weight unless a canary ran between that revert and the next. Un-measured revert rationales become folklore — and here the folklore outlived the code, surviving a later re-broadening to become a false policy statement in the source.

SEPARATE THE PREDICATE FROM THE BUDGET. The dangerous change was the BUDGET (8 retries, 60s cap = up to 242s of backoff per call), not the PREDICATE (which error classes are retryable). The later re-broadening restored the predicate while keeping the safe budget (4 retries, 2s base, ~30s worst case) — 8x smaller exposure. Conflating the two is what made "429 retries" look guilty. Retry breadth is cheap; retry DEPTH is what kills you.

Also: honor Retry-After. A 429-retrying loop that ignores the server's header and sleeps its own exponential schedule is the real latent hazard.

VERIFICATION: a later 182-pair run executed with the re-broadened predicate live. The guard fired 52 times, absorbing all with zero batch aborts. Grepping retry lines for any status code (429/408/529/503/500) returns ZERO hits — all 52 were bare connection-class errors. Attempt depth: 37 at 1/4, 14 at 2/4, 1 at 3/4, none exhausted. Total backoff paid ~165s over 182 pairs = ~0.9s/pair, versus the ~21s/pair the regression theory requires. Scope limit: that run used a different provider than the throttling one originally blamed, so it proves the broad predicate is free on the current stack, not that a hard-throttling endpoint could never 429-storm.