Skip to content

fix(ci): unparallelize action-downloading steps - #93766

Closed
rnegron wants to merge 1 commit into
masterfrom
rauln/ci-serialize-parallel-action-downloads
Closed

rnegron wants to merge 1 commit into
masterfrom
rauln/ci-serialize-parallel-action-downloads

Conversation

@rnegron

@rnegron rnegron commented Sep 2, 2026 •

Copy link
Copy Markdown
Member

Problem

  • Scheduled Backend CI runs go red with no test findings: a job dies during setup, and a rerun passes.
  • The runner fails with Operations that change non-concurrent collections must have exclusive access, or leaves GITHUB_EVENT_PATH empty so the next action crashes on SyntaxError: Unexpected end of JSON input.
  • Cause: parallel: step branches that each download an action write the runner's action registry from two threads.
  • Sample of 12 scheduled runs: 4 went red on this alone (16:32, 12:40, 11:28, 01:42). The product-test matrix is ~46 jobs, so one bad job is enough.

Changes

  • Setup steps that were parallel now run in order, in the 3 workflows whose branches downloaded actions concurrently. Nothing else about them changes: every step keeps its name, uses, with, and if untouched.
  • parallel: blocks whose branches are all run: steps keep the speedup. They download nothing.
  • cd-sandbox-base-image.yml also keeps it, with a comment saying why: those branches are Depot image builds, and running three multi-platform builds one at a time costs more than the crash has ever cost there.
  • Mirrored into .depot/workflows/ci-backend.yml to keep the shadow apples-to-apples.
  • New entry in docs/internal/ci-things-already-tried.md, and a warning at each site, so the pattern is not reintroduced.

How did you test this code?

  • Parsed both revisions of every touched workflow and compared the step lists with the parallel blocks flattened. Identical in every job, so the change is nesting only.
  • bin/hogli lint:workflows, actionlint, and hogli ci:preflight pass.
  • The failure is a runner race, so I could not reproduce it locally or prove absence. The check is that CI still passes and the crash stops appearing on scheduled runs.
  • The blocks came from #76651, which reported no before/after timings, so there is no measured win to weigh against this.

Automatic notifications

  • Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Claude Code (Opus 5), starting from a master-red alert in #alerts-devex.
  • Skills invoked: /authoring-ci-workflows, /code-review, /simplify.
  • The first draft also serialized cd-sandbox-base-image.yml. Review caught that those branches are image builds rather than setup steps, so they are left alone.
  • Considered keeping parallel: and pre-warming the action downloads instead. Rejected: nested uses: inside a composite resolve lazily, so there is no reliable point to pre-warm them.
  • Two other reds in the same window are separate and not addressed here: a temporal.download fetch failure, and a warehouse-sources snapshot-pin flake that #93155 already targets.

https://claude.ai/code/session_013SombsNtSjMRnGQPaSF55i

@trunk-io

trunk-io Bot commented Sep 2, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

ℹ️ Docs preview — preview build triggered

Docs from this PR will be published at posthog.com.

Project Preview Updated (UTC)
posthog.com Open preview Sep 2, 2026, 8:35 PM

The preview should be ready in about 10 minutes. Open the preview at /handbook/engineering/.

@rnegron
rnegron force-pushed the rauln/ci-serialize-parallel-action-downloads branch from eca667f to 63d5ed8 Compare September 2, 2026 20:25
@rnegron
rnegron force-pushed the rauln/ci-serialize-parallel-action-downloads branch from 63d5ed8 to 3b654ec Compare September 2, 2026 20:29
@trunk-io

trunk-io Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@rnegron rnegron closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant