Skip to content

Make cloud task queue operations atomic - #102

Merged
mrubens merged 5 commits into
developfrom
codex/bullmq-task-queue
Jul 10, 2026
Merged

Make cloud task queue operations atomic#102
mrubens merged 5 commits into
developfrom
codex/bullmq-task-queue

Conversation

@mrubens

@mrubens mrubens commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace the cloud task queue read/modify/write sequence with versioned Redis storage and atomic Lua enqueue/dequeue scripts
  • persist each task run queue scope and require owner-matched lock release
  • synchronously cancel superseded pending runs and sync task state in a database transaction
  • replace process-local orphan suppression with a PostgreSQL FOR UPDATE SKIP LOCKED recovery lease
  • add the queue_scope migration plus concurrency, ownership, persistence, and recovery coverage

This takes the recommendation fallback path: it keeps the current controller dispatch contract while making the custom queue transactionally robust.

Validation

  • cloud job queue unit suite: 40 tests passed
  • enqueue database/Redis integration suite: 14 tests passed
  • controller and orphan recovery suites: 13 tests passed
  • package lint and TypeScript checks for cloud-agents, controller, and db
  • pre-push lint:fast, check-types:fast, and knip

Rollout

The Redis storage keys are versioned as v2 so old and new controllers can coexist during a rolling deployment while continuing to contend on the same scope locks.

@roomote-roomote

roomote-roomote Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

No new code issues found. See task

  • 🔴 Migration slot 0002 collision / task_shares regression — Resolved. The branch merged current develop, so the task_shares removal is preserved (0002_remove_task_shares.sql restored, and no task_shares in schema.ts, 0003_snapshot.json, or 0004_snapshot.json) and the queue-scope migration was moved to a sequential slot 0004_task_run_queue_scope.sql. The _journal.json entries are now sequential 00000004 and the snapshot prevId chain is intact (000200030004). The PR reports mergeable: true.
  • 🟢 Blocking dequeue busy-polls instead of BLPOP — Addressed. The latest commit documents the tradeoff in a comment above DEQUEUE_POLL_INTERVAL_MS (packages/cloud-agents/src/server/cloud-job-queue.ts:160), explaining that BLPOP cannot atomically combine queue removal with the scope-lock claim, that polling costs ~4 idle EVAL calls per controller per second for a bounded 250ms wake-up delay, and that a future dedicated wake-up connection can remove the tradeoff. This matches the earlier suggestion to note the tradeoff.

The only change since the last review is test-only: the pr_review queue scope dedup suite now isolates its queue from the shared Redis service by injecting a dedicated CloudJobQueue instance (timeout: 1) through the public CloudJobQueue.queue static in beforeEach, and restoring/flushing/disconnecting it in afterEach (which also let the temporary 15s test timeout be reverted). The injection is type-safe and matches the existing supersedes test's Redis-cleanup pattern. No new issues were found, prior checklist items remain resolved, and the PR remains mergeable.

@mrubens
mrubens merged commit 941eb83 into develop Jul 10, 2026
1 check passed
@mrubens
mrubens deleted the codex/bullmq-task-queue branch July 10, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants