Skip to content

feat: add 5 new error entries (concurrency-timing x3, known-unsolved x2) #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
htekdev merged 1 commit into from
Jun 2, 2026
Merged

feat: add 5 new error entries (concurrency-timing x3, known-unsolved x2) #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions errors/concurrency-timing/always-cleanup-5min-forced-kill.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
id: concurrency-timing-011
title: "always() Cleanup Jobs Forcibly Killed After 5-Minute Cancellation Timeout"
category: concurrency-timing
severity: warning
tags:
- always
- cancellation
- cleanup
- forced-termination
- notification
- timeout
- teardown
patterns:
- regex: "The runner has received a shutdown signal"
flags: "i"
- regex: "Job was cancelled"
flags: "i"
- regex: "The operation was canceled"
flags: "i"
error_messages:
- "The runner has received a shutdown signal. This can happen when the runner service is stopped, a new job is started, or the runner is in the process of shutting down."
- "Job was cancelled"
root_cause: |
When a workflow run is cancelled (manually or via `cancel-in-progress`), GitHub Actions
re-evaluates the `if:` condition for every currently running job. Jobs marked with
`if: always()` continue running — this is the intended mechanism for cleanup, notifications,
and teardown steps.

However, GitHub enforces a **5-minute hard termination window** after cancellation is
initiated. Once 5 minutes have elapsed since the cancellation signal, ALL remaining jobs
are forcibly killed by the server, regardless of their `if:` conditions — including jobs
explicitly marked `if: always()`.

This means:
- Cleanup jobs that take more than 5 minutes (Terraform destroy, test result uploads,
Slack notifications with retries, database teardown) will be killed mid-execution.
- The job may appear partially completed in the logs with no clear failure message —
it simply stops, often leaving infrastructure in a partial or inconsistent state.
- Developers are surprised that `always()` does not guarantee the job completes after
a workflow cancellation.

Common failure scenarios:
- Artifact upload in an `if: always()` post-job step when the upload is slow
- Terraform `destroy` as a cleanup job when a long-running deployment is cancelled
- Notification jobs that retry on transient failures and consume more time than expected
- Integration test teardown (database resets, container removal) that exceeds 5 minutes

Source: GitHub Docs — Canceling a workflow: "After the 5 minute cancellation timeout
period, the server will forcibly terminate all jobs that are still running."
fix: |
Design `always()` cleanup jobs to complete well within 5 minutes. Add a job-level
`timeout-minutes: 4` to any cleanup job that runs after cancellation so it fails
cleanly rather than being force-killed at an unpredictable point.

For teardown that cannot be shortened, trigger cleanup from a separate workflow using
`workflow_run: [completed]` — it runs after the cancelled run fully settles and is
not subject to the 5-minute window.

Use the `cancelled()` expression to detect cancellation and take a fast code path.
fix_code:
- language: yaml
label: "Guard cleanup job with timeout-minutes to fail fast before forced kill"
code: |
jobs:
deploy:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- run: ./deploy.sh

cleanup:
needs: deploy
if: always()
runs-on: ubuntu-latest
timeout-minutes: 4 # Stay under the 5-min forced-kill window
steps:
- name: Teardown infrastructure
run: ./teardown.sh
timeout-minutes: 3 # Per-step guard too

- language: yaml
label: "Use cancelled() to take a fast notification path on cancellation"
code: |
jobs:
build:
runs-on: ubuntu-latest
steps:
- run: ./slow-build.sh

notify:
needs: build
if: always()
runs-on: ubuntu-latest
steps:
- name: Quick notification (cancellation — must be fast)
if: cancelled()
run: |
curl -s -X POST "$SLACK_WEBHOOK" \
-H 'Content-type: application/json' \
-d '{"text":"⚠️ Workflow cancelled — cleanup may be incomplete"}'

- name: Full notification (success or failure path — has time)
if: "!cancelled()"
run: ./full-notify.sh "${{ needs.build.result }}"

- language: yaml
label: "Post-cancellation teardown via workflow_run — not subject to 5-min window"
code: |
# cleanup.yml — separate workflow triggered after any completion including cancellation
on:
workflow_run:
workflows: ["Deploy"]
types: [completed]

jobs:
teardown:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Emergency cleanup when deploy was cancelled
if: github.event.workflow_run.conclusion == 'cancelled'
run: ./emergency-teardown.sh

- name: Normal cleanup on success or failure
if: github.event.workflow_run.conclusion != 'cancelled'
run: ./standard-teardown.sh
prevention:
- "Keep `if: always()` cleanup jobs under 4 minutes — add `timeout-minutes: 4` as a safety guard."
- "Use `if: cancelled()` to detect cancellation and take a fast code path rather than the full teardown path."
- "For cleanup that takes longer than 5 minutes, use a separate `workflow_run: [completed]` workflow that runs outside the cancellation window."
- "Test cancellation behavior by manually cancelling a long-running workflow and verifying cleanup jobs complete before 5 minutes."
docs:
- url: "https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/canceling-a-workflow"
label: "GitHub Docs: Canceling a workflow (5-minute forced termination)"
- url: "https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/evaluate-expressions-in-workflows-and-actions#status-check-functions"
label: "Status check functions: always(), cancelled()"
- url: "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#workflow_run"
label: "workflow_run event — trigger cleanup after completed workflows"
160 changes: 160 additions & 0 deletions errors/concurrency-timing/required-check-pending-path-filter-skip.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
id: concurrency-timing-013
title: "Required Status Check Stuck in Pending When Workflow Skipped by Path or Branch Filter"
category: concurrency-timing
severity: warning
tags:
- required-status-check
- path-filter
- branch-filter
- pending
- pull-request
- branch-protection
- paths
- blocked-pr
patterns:
- regex: "Some checks haven't completed yet|Required status check.*pending"
flags: "i"
- regex: "Waiting for status:.*pending"
flags: "i"
error_messages:
- "Some checks haven't completed yet"
- "Required status check is pending"
- "Waiting for status: CI / test (pending)"
root_cause: |
GitHub Actions workflows that use `paths:`, `paths-ignore:`, `branches:`, or
`branches-ignore:` filters will NOT run — and will NOT report ANY status — for
commits that don't match the filter criteria.

When a required status check is configured in a branch protection rule and the
workflow providing that check is skipped by a filter:
- The check is NEVER created for that commit — it remains in "Pending" state indefinitely
- The PR is blocked from merging with "Some checks haven't completed yet"
- The check CANNOT be manually re-triggered without pushing a commit that matches the filter

Common scenario:
A repository has a `ci.yml` workflow with `paths: ['src/**', '*.ts']` and
`CI / test` configured as a required status check. A developer opens a PR that only
changes `README.md` or `.github/docs/`. The `CI / test` check never runs, shows as
"Pending" forever, and the PR is permanently blocked from merging without an admin
override or a dummy code commit to trigger the workflow.

This is explicitly documented behavior but frequently misunderstood:
- The workflow appears to work correctly for code-change PRs (the common case)
- The bug only surfaces on documentation-only, config-only, or administrative PRs
- Developers and reviewers see a pending check with no way to trigger it

Note: This is distinct from skipped-needs-cascade (job dependency skipping) — this
is specifically about the WORKFLOW TRIGGER filter preventing the run from ever starting,
so no job status is reported at all.

Source: GitHub Docs — Troubleshooting required status checks: "If a workflow is skipped
due to path filtering, branch filtering or a commit message, then checks associated
with that workflow will remain in a 'Pending' state. A pull request that requires those
checks to be successful will be blocked from merging."
fix: |
Two main approaches:

1. **Always-succeeding bypass job** — remove path filters from the workflow trigger,
run the workflow for all PRs, use `dorny/paths-filter` or `tj-actions/changed-files`
to detect changes inside the workflow, and add a sentinel job that always produces a
status. Configure the required check to point at the sentinel job name.

2. **Split workflow** — keep the path-filtered workflow for actual CI work, and add a
separate always-running workflow that provides the required status check name
(succeeds immediately for non-code PRs, waits for CI for code PRs).
fix_code:
- language: yaml
label: "Fix: always-running workflow with internal path detection and sentinel job"
code: |
name: CI
# No path filter — workflow always runs for all PRs
on:
pull_request:
branches: [main]

jobs:
changes:
runs-on: ubuntu-latest
outputs:
code: ${{ steps.filter.outputs.code }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
code:
- 'src/**'
- '*.ts'
- 'package*.json'

test:
needs: changes
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci && npm test

# Branch protection required check: "CI / ci-gate" (not "CI / test")
# Always produces a status — green for docs PRs, waits for test on code PRs
ci-gate:
needs: [changes, test]
if: always()
runs-on: ubuntu-latest
steps:
- name: Confirm CI passed or code was not changed
run: |
CODE_CHANGED="${{ needs.changes.outputs.code }}"
TEST_RESULT="${{ needs.test.result }}"
if [[ "$CODE_CHANGED" == "false" ]]; then
echo "✅ No code changes — CI gate passes automatically"
elif [[ "$TEST_RESULT" == "success" ]]; then
echo "✅ Tests passed"
else
echo "❌ Tests $TEST_RESULT"
exit 1
fi

- language: yaml
label: "Alternative: run all steps always but skip expensive ones via filter"
code: |
name: CI
on:
pull_request:
branches: [main]
# No workflow-level path filter — status always reported

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
code: ['src/**', '*.ts', 'package*.json']

- name: Install dependencies
if: steps.filter.outputs.code == 'true'
run: npm ci

- name: Run tests
if: steps.filter.outputs.code == 'true'
run: npm test
# Job always completes with success — check is always reported
prevention:
- "Never use workflow-level `paths:` or `branches:` filters as the sole trigger for a required status check."
- "Use `dorny/paths-filter` or `tj-actions/changed-files` INSIDE an always-running workflow instead of workflow-level path filters."
- "Name required status checks after jobs that always produce a status, even on non-code PRs."
- "Test branch protection rules by opening a documentation-only PR to verify all required checks complete."
- "Consider admin overrides as a last resort, not a workflow fix — the root cause will keep happening."
docs:
- url: "https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/troubleshooting-required-status-checks"
label: "Troubleshooting required status checks (path filter skip documented)"
- url: "https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore"
label: "Workflow syntax: paths and paths-ignore filters"
- url: "https://github.com/dorny/paths-filter"
label: "dorny/paths-filter — detect changed files inside workflow"
Loading
Loading