Skip to content

fix(ci): make issue triage labeling deterministic - #1206

Merged
phernandez merged 8 commits into
mainfrom
codex/fix-claude-issue-triage
Aug 9, 2026
Merged

fix(ci): make issue triage labeling deterministic#1206
phernandez merged 8 commits into
mainfrom
codex/fix-claude-issue-triage

Conversation

@phernandez

@phernandez phernandez commented Aug 6, 2026

Copy link
Copy Markdown
Member

Why

The Claude issue-triage workflow advertises priority, component, complexity, and status labels that do not exist in this repository. Its only label tool accepted arbitrary candidate labels, silently ignored unknown names, and immediately added every candidate that happened to exist. Because the tool was add-only, label discovery mutated the live issue and Claude could not undo mistakes.

This has recurred since the helper was introduced on July 16: at least 12 triage comments explicitly report accidental label application. On issue #1205, an enhancement tracker ended up labeled as a bug, documentation request, duplicate, good first issue, question, and Cloud issue as well.

What Changed

  • Replace the generic --add-label interface with one semantic classification:
    --type bug|enhancement|documentation|question --component cloud|none.
  • Reject unsupported values, duplicate options, and candidate-probing arguments before any GitHub mutation.
  • Remove only obsolete triage-owned labels and add the requested classification without replacing the issue's full label set.
  • Abort before mutation if the current labels cannot be read.
  • Add the desired classification before removing obsolete labels, so an additive API failure leaves the prior classification intact.
  • Re-read labels after the additive update so competing bot-owned labels introduced during the first read/POST window are included in reconciliation.
  • Serialize workflow runs by issue number so two triage attempts cannot delete each other's selected type.
  • Update the workflow prompt to call the helper exactly once, treat priority and complexity as prose, classify MCP as a component rather than a fifth type, and avoid duplicate/status labels that the bot does not own.
  • Require a concise triage comment without tool logs, taxonomy discussion, or task checklists.
  • Add regression coverage for targeted mutations, read failures, concurrent conflicting labels, workflow serialization, invalid probes, required arguments, and workflow YAML/prompt syntax.
  • Select Git Bash from the installed Git for Windows and use LF-only, Bash-compatible fixture paths so the shell-helper coverage runs on Windows.

Implementation Details

The helper first verifies that current labels are readable before any mutation. It then uses GitHub's additive labels endpoint to apply the requested type and optional Cloud component. After that POST succeeds, it reads the labels again and deletes obsolete values from the automated triage ownership set (bug, enhancement, documentation, question, and cloud). The second read includes a competing owned label added during the initial read/POST window, while targeted DELETE requests leave template, maintainer, and unrelated automation labels untouched.

The workflow uses claude-issue-triage-${{ github.event.issue.number }} as its concurrency group with cancel-in-progress: false. Runs for different issues remain independent, while attempts for the same issue are serialized so one complete classification wins instead of two DELETE loops removing each other's desired labels.

Repeated valid calls converge on one type and one Cloud/non-Cloud component choice. Unsupported attempts and failed preflight reads stop before the first write. A failed additive request leaves the prior classification intact. A failed post-add reconciliation read leaves the desired label added but performs no destructive calls, allowing a later rerun to converge safely. Duplicate detection remains read-only; Claude links the original issue in prose rather than labeling or closing the new issue.

The tests invoke scripts/edit-issue-labels.sh with ordinary bash on POSIX. On Windows they derive bash.exe from the installed Git executable, because the unqualified Windows bash command launches WSL rather than Git Bash. Temporary shell fixtures use LF line endings and POSIX-form paths so Git Bash parses them consistently.

Testing

Automated

  • uv run pytest -p pytest_mock --no-cov -q tests/test_claude_issue_triage.py: 13 passed
  • just fast-check: Ruff checks and formatting passed; ty passed
  • bash -n scripts/edit-issue-labels.sh: passed
  • uv run ruff check tests/test_claude_issue_triage.py: passed
  • uv run ruff format --check tests/test_claude_issue_triage.py: passed
  • git diff --check: passed

The focused suite parses the edited workflow as YAML and asserts the single semantic helper contract, per-issue non-cancelling concurrency, the four-type/MCP-component taxonomy, exact additive POST/targeted DELETE ordering, preservation of non-triage labels, no mutation after a failed preflight read, no deletion after a failed additive or reconciliation read, and reconciliation of bot-owned labels added during the initial mutation window.

Manual

  • Inspected the live repository label taxonomy and recent Claude triage comments.
  • Confirmed at least 12 self-reported accidental-label incidents since July 16.
  • No live issue labels were mutated while testing this fix.
  • Inspected the original Windows failure and confirmed all 10 helper cases failed before script execution with WinError 193.
  • Inspected the first portability rerun: direct-script execution was fixed, 25 checks passed, and the Windows job exposed that unqualified bash selected the WSL launcher rather than Git Bash.
  • Reproduced Codex's concurrent-owned-label scenario in the fake GitHub boundary and verified the fresh reconciliation read removes the competing type and Cloud labels.
  • Evaluated the two-simultaneous-run interleaving and placed serialization at the Actions workflow boundary rather than adding locks or retries to the shell helper.

Risks / Follow-ups

  • The workflow itself cannot be exercised end to end without opening a real issue; the first issue opened after merge is the live acceptance check.
  • GitHub does not provide an atomic operation for replacing only a label subset. Per-issue workflow serialization prevents competing helper runs, and the post-add read reconciles other owned-label changes observed during the initial mutation window. A distinct external actor can still mutate labels after any final operation; a later valid triage run converges without replacing unrelated labels.
  • This PR prevents new label accumulation but intentionally does not clean labels on previously affected issues.
  • The pushed head relies on the Windows CI job for platform-specific confirmation of the Git Bash selection and fixture paths.

Signed-off-by: phernandez <paul@basicmachines.co>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31ca315e07

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/edit-issue-labels.sh Outdated
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76f0f02db2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/edit-issue-labels.sh
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee06e942f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/edit-issue-labels.sh
Comment thread .github/workflows/claude-issue-triage.yml
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 1c15d00736

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 1c15d00736

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e95cce6ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/edit-issue-labels.sh Outdated
Signed-off-by: phernandez <paul@basicmachines.co>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a081c1d163

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/edit-issue-labels.sh
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez
phernandez merged commit a9e845e into main Aug 9, 2026
27 of 28 checks passed
@phernandez
phernandez deleted the codex/fix-claude-issue-triage branch August 9, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant