Bound concurrent generations and surface saturation #47

Parent

Spec: #22 — Generate text, an LLM action node for workflows.

What to build

Generations run on a shared Celery queue. Without a bound, one team publishing a high-volume workflow can occupy every worker and stall unrelated work for everyone.

Add a fleet-wide cap and a per-team cap on concurrent generations, following the pattern already used by the analytics query task: two stacked concurrency limiters over a Redis sorted set, the outer one global and the inner one keyed by team. The existing pattern hardcodes its numbers; these should come from settings so they can be turned without a deploy. The slot expiry must exceed the task's hard time limit, or a worker that dies can free a slot while its work is still notionally live.

Saturation is invisible by design, and that is the point. A team at its cap does not get an error. The task retries in the broker while retrieve keeps answering pending, so the workflow simply takes longer. Only when the retry budget is exhausted does it become a terminal capacity failure, written through the task's failure hook as a normal terminal outcome rather than an exception surfaced to the caller. Preview needs no special case — it waits like everything else, and hits its own preview deadline if the wait is too long.

Also here: an expiry on the task itself, so a task nobody will collect is discarded rather than picked up and billed.

The measurement that decides the next step. Stamp the enqueue time on the task payload and record the wait as a histogram. This is the graduation signal: the feature deliberately shares an existing queue because production reads its worker queues from the charts repository, so a queue added only in this repo would be consumed by nothing, silently. If queue-wait at the high percentile turns bad during the pilot, that is the evidence for splitting to a dedicated queue — and the charts entry has to land before the flag widens. Saturation itself needs no new gauge; it already shows on the shared concurrency-limit-exceeded counter.

Acceptance criteria

• Concurrent generations are bounded fleet-wide and per team, with both limits read from settings.
• The slot expiry is longer than the task's hard time limit.
• A team at its per-team cap does not block other teams.
• A capped task retries rather than failing, and retrieve keeps answering pending while it waits.
• Exhausting the retry budget produces a terminal capacity failure written through the failure hook, not an exception to the caller.
• A task that has expired is discarded rather than executed.
• Celery queue wait is recorded as a histogram from an enqueue stamp on the payload.
• Limiter behaviour is tested directly against the fake Redis used in tests, including the over-limit case. Note the fake server is memoized across tests, so leftover sorted-set state has to be cleared or later tests inherit occupied slots.

Blocked by

• Generation endpoint, task, and first pass #45 — the task these caps wrap.