[Fix] Retry the worker's job-claim callbacks while the public edge settles - #21
Merged
Merged
Conversation
…ttles The first callbacks a docker worker makes (cloudJobs.dequeue / resume) go through the public URL, and a freshly (re)started ngrok/Caddy edge can briefly answer 5xx or without a JSON content-type. The worker failed the whole job on the first bad response, orphaning it in 'dequeued'. Claiming a job is idempotent server-side (FOR UPDATE SKIP LOCKED), so add the claim mutations to the retryable-path allowlist with a longer budget (6 attempts, ~31s of backoff). Also have the dev CLI poll the API health route through the public edge after starting services, so local stacks don't hand jobs to workers before the edge is serving. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
No code issues found. See task |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first callbacks a docker worker makes —
cloudJobs.dequeue(andcloudJobs.resumefor resumed jobs) — go through the public URL (ngrok → Caddy edge). Right after the stack (re)starts, the edge can briefly answer 5xx or without a JSON content-type while its upstreams settle. The worker's fetch wrapper failed the whole job on the first bad response:The job is orphaned in
dequeued(later recovered toidleby the controller) and the worker container lingers. Observed on a local dev stack: job created at 20:12:00, worker failed at 20:12:15, while the same URL served healthy JSON minutes later; an identical retried job succeeded end-to-end.Fix
packages/sdk: the transport-retry allowlist (RETRYABLE_WORKER_TRPC_MUTATION_PATHS) becomes a map with optional per-path retry budgets.cloudJobs.dequeueandcloudJobs.resumeare added with a longer budget (6 attempts, ~31s of exponential backoff) — claiming a job is idempotent server-side (FOR UPDATE SKIP LOCKEDreturns nothing on a second attempt), so replaying the claim is safe. Batched requests take the smallest budget across their paths, and explicit wrapper options still win so callers/tests can pin behavior.apps/dev: after starting services, the dev CLI now polls the API health route through the public edge (<public-url>/_roomote-api/, up to 45s, warn-only) instead of only checking pm2 process state, so local stacks don't hand jobs to workers before the edge is serving.Verification
packages/sdk/src/client/index.test.ts(21 total pass): retries a no-content-type 502 on dequeue, the 6-attempt default budget, resume transport retry, and explicit-options precedence.waiting_for_prompt.🤖 Generated with Claude Code