Skip to content

fix(ci): let the DCO gate be republished instead of only refreshed in place - #7408

Merged
ko3n1g merged 1 commit into
NVIDIA:mainfrom
ko3n1g:ko3n1g/ci/dco-gate-recreate
Sep 17, 2026
Merged

ko3n1g merged 1 commit into
NVIDIA:mainfrom
ko3n1g:ko3n1g/ci/dco-gate-recreate

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Scope corrected after live testing. This PR originally claimed that publishing a
new DCO gate check run re-binds a merge box stuck on "Expected — waiting for status".
That was tested on four real pull requests and is false. The recreate path is kept
because it is the right shape for the reconcile sweep, but it does not fix that symptom.
The negative result is recorded below so the next person does not repeat it.

Background / motivation

  • A small number of approved, fully green pull requests sit at mergeable_state: blocked
    with the merge box reporting the required DCO gate as "Expected — waiting for status
    to be reported"
    , while a green DCO gate check run exists on the head SHA and
    gh pr checks <n> --required lists it as pass.
  • Separately, the refresh path could only PATCH the existing gate, so a gate carrying a
    verdict the DCO App has since revised could never be corrected: reconcile_pull_request
    returned present for any head that already had a gate, whatever it said.

What changed

  • The reconcile sweep now repairs a blocking gate the DCO App has since cleared, not just
    a missing one — new refreshed outcome.
  • dco-gate.yml takes a pull-request number instead of only a SHA, so a manual refresh
    does not need a head-SHA lookup.
  • publish_gate gains recreate, which POSTs a replacement check run rather than PATCHing.

Details

  • .github/scripts/dco_gate.py
    • create_gate — the shared POST path; publish_gate(..., recreate=True) routes to it.
    • head_sha_of_pull_request + _requested_sha — resolve DCO_GATE_PULL_REQUEST to the
      current head, fall back to DCO_GATE_SHA, fail loudly on a dispatch with neither.
    • gate_conclusion — extracted from gate_payload so the sweep can compare a published
      gate against the current verdict.
  • .github/workflows/dco-gate.yml — pull_request / sha / recreate inputs, plus
    pull-requests: read for the head lookup.

Read budget — why a passing gate is not re-verified

The repo has 954 open pull requests and the sweep already spends ~1 request per PR against
the hourly GITHUB_TOKEN budget (the 12:37 UTC run took 5m46s). Re-reading every verdict would
double that and push the sweep over the limit. So the sweep re-reads a verdict only behind a
gate that is not already passing
— the class that actually blocks a PR. A verdict that flips
after a gate has passed stays with the event-driven publisher, which fires on that flip.
test_reconcile_reads_a_passing_gate_exactly_once pins the invariant.

Negative result: recreating the check run does not clear a stuck merge box

Measured across all 954 open PRs, exactly 4 were approved with all three required contexts
green and still BLOCKED: #7354, #7350, #7347, #7329.

A one-off Actions run published a brand-new DCO gate check run for each
(104905317481, 104905333508, 104905352882, 104905366800). All four stayed blocked,
and #7329 still rendered "Expected — waiting for status" afterwards. Closing and reopening
#7329 did not clear it either.

Every API-visible attribute of a stuck PR matches a CLEAN one:

#7329 (blocked) #7397 (clean)
statusCheckRollup.state SUCCESS SUCCESS
reviewDecision APPROVED APPROVED
required contexts 3× success, app 15368 3× success, app 15368
pending check runs 0 0
gate check suite fork branch, skipped fork branch, success/skipped
gate in the CI suite? no no
base main main

Ruled out: check-run identity, check-run or merge-box staleness (a CLEAN PR untouched since
2025 exists), check-suite shape, base branch, labels, auto-merge, unresolved threads, required
deployments, conversation resolution, linear history, commit signatures.

Durable fix worth considering separately

Nemo_CICD_Test and multi-approval-bot-summary are never stuck, and they differ from the gate
in exactly one way: they are real Actions jobs in the pull-request/<n> mirror suite, while
DCO gate is an API-created check run filed into the fork-branch suite. Publishing the gate
as a job on the mirror push — polling briefly for the DCO App verdict — would put it on the same
delivery path as the two contexts that never fail to bind. That is a larger change and is not
attempted here.

Tested

  • python3 .github/scripts/test_dco_gate.py → 23 passed (was 20; 3 new, 2 adjusted).
  • ruff check .github/scripts/dco_gate.py .github/scripts/test_dco_gate.py → clean.
  • Workflow YAML parses; all touched lines ≤ 100 chars.
  • Live: PATCH refresh of ci: preserve dependency build cache across README changes #7347's gate (104581997123) changed only completed_at; POST of four
    replacement gates changed the IDs and nothing else. Both recorded above.

🤖 Generated with Claude Code

@ko3n1g
ko3n1g requested a review from a team as a code owner September 16, 2026 17:25
@ko3n1g

ko3n1g commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft September 16, 2026 17:25
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@copy-pr-bot

copy-pr-bot Bot commented Sep 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

A check run created through the Checks API is pinned by GitHub to the first
check suite for its head SHA. For a fork pull request that suite is keyed to the
contributor's branch name, which does not exist in this repository, and when a
merge box fails to bind it the required context reports "Expected — waiting for
status to be reported" indefinitely.

That state is unrepairable through the API: a check run cannot move between
suites, so neither updating the existing gate nor publishing a replacement
clears it. Both were tried against four stuck pull requests and changed nothing.
A commit status is not an answer either, because the required context is pinned
to app 15368 and a status published by anything else is not counted.

Report the gate the way `Nemo_CICD_Test` and `multi-approval-bot-summary` report
theirs instead — as a job on the `pull-request/<n>` mirror push, whose check run
lands in a suite keyed to a real branch of this repository. The job only reads
the trusted DCO App verdict, waits for it when it has not landed yet, and fails
closed if it never does.

Delete the Checks API publisher and its reconcile sweep rather than leaving them
alongside. Two producers of one context race: branch protection reads whichever
check run is newer, so an API-published gate landing last would put the
unbindable run back in front of the merge box and reintroduce the bug. Dropping
the sweep also removes ~950 API requests every six hours.

`dco-gate-merge-group.yml` still owns the gate for merge-queue refs, which this
job does not run for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Sep 16, 2026
@ko3n1g
ko3n1g enabled auto-merge September 16, 2026 18:27
@ko3n1g
ko3n1g added this pull request to the merge queue Sep 17, 2026
@nemo-automation-bot

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35217238977

@nemo-automation-bot

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35217283399

@ko3n1g
ko3n1g removed this pull request from the merge queue due to a manual request Sep 17, 2026
@nemo-automation-bot

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/35217310563

@ko3n1g
ko3n1g merged commit cd99369 into NVIDIA:main Sep 17, 2026
106 of 108 checks passed

This branch was previously deployed

2 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved All necessary approvals have been made complexity: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants