Best strategy for rate-limiting agent API calls in production Kubernetes?

Token bucket with a central Redis lease is the right call here, but the thundering-herd problem needs a separate fix.

The pattern

1. Shared token bucket in Redis (one per upstream API key) — all agent pods consume from the same bucket
2. Jittered retry — sleep(base * 2^attempt + random(0, base)) instead of pure exponential
3. Circuit breaker per upstream — if error rate > threshold, agents fast-fail instead of retrying

The part people miss

Don't let individual agents retry against the shared bucket — add a local queue per pod. Agents enqueue requests; the pod drains at a rate proportional to remaining bucket tokens. This smooths traffic and keeps agents non-blocking.

// Simplified pod-level queue drain
setInterval(async () => {
  const tokens = await redisTokenBucket.available()
  const batch = localQueue.drain(Math.min(tokens, MAX_BATCH))
  await Promise.allSettled(batch.map(req => req.execute()))
}, DRAIN_INTERVAL_MS)

One thing I'd add: look at KEDA's external scaler with your Redis rate limit counters as the scale metric. Instead of scaling on CPU/memory, scale your agent pods inversely to remaining API quota. When you're near the limit, KEDA can scale down agents to reduce request rate automatically — no code changes needed in agents.