Rate-limit counters are held only in the ratelimiter's Olric cluster memory, and they are lost when the pods holding them go away. A caller then regains its full budget in the middle of a window, with nothing logged and no metric to distinguish it from a new window.
Reproduced on a self-hosted install with three ratelimiter pods and a function limited to 10 per hour:
- Spend 4 of the budget, replace all three pods one at a time, then invoke again: all 10 further calls are admitted, so 14 are admitted against a limit of 10.
- Same sequence with no pod replacement: exactly 10 admitted, then denied. Enforcement itself is correct.
- Replacing only one or two of the three pods keeps the counter, because a pod that still holds a copy survives.
Cause: each counter lives on the member that owns its partition, and the Olric replica count is left at the default of 1, so there is no second copy. Olric also only seeds a member on write. Its balancer moves a partition when the current holder stops owning it, and nothing back-fills a backup owner that joined while a counter sat idle, so a freshly started pod holds nothing for existing counters. Interleaving a single request between pod replacements is enough to keep the counter, which confirms writes are the only path that seeds a new member.
Impact: a rolling upgrade of the ratelimiter replaces every pod in turn, so every counter is lost and enforcement is effectively off for the duration of the rollout. Any pod churn (eviction, node drain, OOM kill, scale-down) has the same effect on the counters that pod held.
Suggested direction: give each counter a backup copy, hand a terminating pod's counters to the current owners before it exits, and surge the replacement in first so there is a live member to receive them.
Note that this still would not make counters durable: losing every member at once has nothing to recover from, since nothing is persisted.
Rate-limit counters are held only in the ratelimiter's Olric cluster memory, and they are lost when the pods holding them go away. A caller then regains its full budget in the middle of a window, with nothing logged and no metric to distinguish it from a new window.
Reproduced on a self-hosted install with three ratelimiter pods and a function limited to 10 per hour:
Cause: each counter lives on the member that owns its partition, and the Olric replica count is left at the default of 1, so there is no second copy. Olric also only seeds a member on write. Its balancer moves a partition when the current holder stops owning it, and nothing back-fills a backup owner that joined while a counter sat idle, so a freshly started pod holds nothing for existing counters. Interleaving a single request between pod replacements is enough to keep the counter, which confirms writes are the only path that seeds a new member.
Impact: a rolling upgrade of the ratelimiter replaces every pod in turn, so every counter is lost and enforcement is effectively off for the duration of the rollout. Any pod churn (eviction, node drain, OOM kill, scale-down) has the same effect on the counters that pod held.
Suggested direction: give each counter a backup copy, hand a terminating pod's counters to the current owners before it exits, and surge the replacement in first so there is a live member to receive them.
Note that this still would not make counters durable: losing every member at once has nothing to recover from, since nothing is persisted.