Skip to content

AI decision step: let large batches through admission without throttling people #251

Closed as not planned

Description

@Silthus

Part of #240
Spec: #247 (Spec doc: AI decision step for workflows).

What to build

Make a large batch workflow get through the AI decision step without people failing as throttled.

Today the decide route admits with per-team and global token buckets, and a denial returns 429 with a Retry-After that names the next free token, not the person's place in the queue. A batch of N people therefore polls the route at about N/2.5 requests per second, and many of them reach the spec's 30-minute throttled limit (found in the #248 build, see its progress comment).

Pick the simpler of these two and record why:

  1. Queue-aware admission in Django: a GCRA-style "next free slot" so Retry-After reflects the person's place in the queue.
  2. A per-team concurrency cap in the Node handler, so a batch drains at the admission rate instead of polling.

Also add a per-organization admission bucket, so one organization with many projects cannot fill the global bucket.

Acceptance criteria

  • A simulated batch far above the per-team rate drains with no person failing as throttled within the spec's bounds, and the route sees roughly the admission rate, not N/2.5 requests per second.
  • One organization's projects together cannot take more than the per-organization budget.

Test seam

The Django route contract tests (spec seam 1) with a fake clock, or the Node handler tests (spec seam 3), depending on the choice.

Base branch

Stacks on the Node runtime branch from #249 when the fix lives in the handler, else on feat/workflows-ai-decision-route from #248.

Cast

Opus 5.5 (GPT-6.1 Sol through codex:codex-rescue when Codex answers).

Mandatory skills

/tdd, /using-redis-token-buckets, /writing-tests, /writing-pr-descriptions, then qa-swarm.

Activity

  1. Silthus commented on Oct 5, 2026

    @Silthus
    OwnerAuthor

    Superseded: Michael redrew v1 as one minimal PR (see the map's Destination). This ticket moves to v2. Its branch, if any, stays as the starting point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions