Fence Backfiller tasks by generation - #11311
Conversation
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
The Backfiller validated its delayed tasks by comparing the task's wall-clock scheduled time against LastProcessedTime, which for a range backfill is a schedule-time cursor into the (often historical) range. The two clocks never reconcile: for historical ranges an already-executed continuation stays valid, so redelivery re-runs it and advances the cursor prematurely; for forward-dated ranges the fresh continuation is invalidated immediately and the backfill stalls after the first batch. Stamp a generation on each BackfillerTask and track it on BackfillerState. scheduleTask bumps the generation, and Validate accepts a task only while it matches the current generation, so a superseded or redelivered task is rejected. LastProcessedTime is now used purely as the range cursor. Backward compatible: in-flight tasks (generation 0) stay valid until superseded on their next reschedule. Also fix two fidelity gaps in the shared chasmtest engine that these tests surfaced: - Supply a default namespace entry (and WithNamespace option) so components that read namespace-filtered config can run under the engine. - Make NextTransitionCount stable within a transaction, advancing once per commit, so newly scheduled tasks are actually persisted rather than silently dropped by the task-persistence gate. Add Engine.FirePureTasks to deliver a component's due pure tasks through the framework's real timer path, and drive the new backfiller lifecycle tests through the actual create/patch handler.
# Conflicts: # chasm/chasmtest/test_engine.go
Use persisted execution attempts to recognize generation-zero continuations created by an older binary, reject the task it already processed, and restore the generation invariant on the next execution. Exercise the persisted logical-task lifecycle on the updated CHASM test engine and keep the regression setup local to its tests.
chaptersix
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 18eea9ba96
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if taskGeneration == 0 { | ||
| // An old binary schedules generation-zero tasks and advances only Attempt. | ||
| valid = currentGeneration == 0 || attempt >= currentGeneration |
There was a problem hiding this comment.
During a rolling upgrade, a forward-dated backfill can still stall before this compatibility branch is reached: if an old binary receives the generation-bearing continuation, its old Validate ignores the generation and calls validateTaskHighWaterMark; because LastProcessedTime is in the future while the delayed task uses wall-clock time, it rejects and removes the continuation without scheduling the generation-zero successor assumed here. The mixed-version test only increments Attempt and therefore skips this real old-validator path; preserve a continuation across that handoff or explicitly gate mixed-version processing.
Useful? React with 👍 / 👎.
1 participant
What changed?
Background
Expected CHASM pure-task lifecycle
A CHASM pure timer is a physical wake-up for due logical tasks in the component tree. For each due logical task, CHASM calls its validator immediately before execution. An invalid task is a successful no-op:
Executeis skipped, the node is marked dirty, and transaction close removes logical tasks whose validators return false (ExecutePureTask,closeTransactionCleanupInvalidTasks).After a successful
Execute, the handler must change component state so the processed logical task becomes invalid. It may also schedule a valid successor. CHASM explicitly relies on the processed task becoming invalid and being removed at transaction close (pure-task iteration contract).stateDiagram-v2 direction LR [*] --> Due Due --> Validate: physical timer finds logical task Validate --> Invalid: returns false Invalid --> Acknowledged: skip Execute and remove logical task Validate --> Execute: returns true Validate --> Retry: returns error Execute --> Retry: returns error Execute --> Complete: deletes Backfiller Execute --> Successor: schedules generation N+1 Successor --> Committed: generation N is invalid and N+1 is valid Complete --> Acknowledged: commit deletion Committed --> Acknowledged: commit state and tasks Retry --> Due: physical queue retryFor a normal continuation, the expected validation results are:
Validate(N) == true.N+1:Validate(N) == falseandValidate(N+1) == true.Nremains retryable.What happens when validation is wrong?
The timer executor treats an invalid task as success and commits the cleanup (active pure-timer executor). A boolean validation result therefore does not consume a retry attempt and does not enter the DLQ path.
Executeis skipped. Transaction close removes the logical task, the physical timer completes successfully, and there is no retry or DLQ entry. If no other task can recreate the continuation, the Backfiller stalls.Backfiller validation
LastProcessedTimeis the Backfiller's cursor in schedule time. A delayed CHASM task'sScheduledTimeis in wall-clock time. The current validator treats the task as valid when:The range bounds do not directly decide validity; they determine where the cursor can move after each partial batch. The same comparison therefore fails differently depending on the cursor's position relative to wall-clock time.
Existing behavior
Range only in the past
stateDiagram-v2 direction LR [*] --> Due Due --> Executing: Validate = true Executing --> CursorAdvanced: Execute partial batch CursorAdvanced --> Retained: post-Execute Validate = true Retained --> Executing: redeliveryRange from the past through now
stateDiagram-v2 direction LR [*] --> Due Due --> Executing: Validate = true Executing --> Compare: cursor advances toward now Compare --> Retained: cursor remains before task time Retained --> Executing: redelivery Compare --> OldTaskRemoved: cursor reaches task time OldTaskRemoved --> NextTask: continuation remains validRange from the past into the future
stateDiagram-v2 direction LR [*] --> Due Due --> Compare: Validate before Execute Compare --> Executing: task time is after cursor Executing --> CursorAdvanced: Execute partial batch CursorAdvanced --> Retained: cursor remains before task time Retained --> Executing: redelivery Compare --> Removed: cursor is at or after task time CursorAdvanced --> Removed: cursor advances past continuation time Removed --> Stalled: Execute skipped and Attempt unchangedAttemptdoes not advance, and the range stalls.Desired behavior
Task validity should depend only on task lifecycle state, not on where the requested range falls on the calendar.
Why this change?
A generation fence enforces the lifecycle above when the task is handled by the new code. Scheduling a replacement advances the persisted generation, making the prior payload invalid without consulting the range cursor.
A plain generation equality check was not sufficient for rolling upgrades. When an old binary successfully executes a generation-bearing task, it preserves the unknown state generation, advances
Attempt, and schedules a generation-zero continuation. The attempt-aware compatibility path recognizes only that successor and re-establishes the generation invariant when a new binary executes it.That compatibility path applies only after the old binary executes the task. Historical continuations pass the old high-water-mark validator and can use this handoff. A forward-dated continuation can be rejected by the old validator before
Attemptadvances, so the compatibility path cannot recover it. Forward-dated partial backfills already stall this way on the current implementation; this PR does not make that behavior worse, but the fix is not guaranteed for a continuation evaluated by an old executor during a rolling deployment.Potential risks
The rolling-upgrade compatibility path relies on the persisted Backfiller attempt count advancing once per successful task execution. Tests cover ordinary generation invalidation and a new-binary → old-binary → new-binary continuation handoff.
An old binary can still remove a forward-dated continuation before executing it. Avoiding that limitation requires preventing mixed-version CHASM task execution or separately gating the new behavior; this PR does not add recovery for a logical task already removed by the old validator.