Repository navigation
perf(ci): run the python static checks concurrently - #131571
fuziontech wants to merge 1 commit into
Conversation
"The `code-quality` job ran every static check one after another on one runner, so the job wall was their sum. Measured over 7 pull-request runs of `.github/workflows/ci-python.yml`, that job is 279s at the median and its serial steps add up to the same 279s: ty 49s, mypy 134s, the generated-projections check 18s, and the lint/pytest group 15s. They read the same tree and share no state, so they now run as one `parallel:` block and the wall becomes the longest of them. mypy is the long pole, so the job is bounded by mypy's own duration instead of mypy's plus every other check's. The job keeps its runner: the extra checks are light next to mypy, which already pins its worker count to the runner's memory. Every command, severity and target is unchanged, and the block holds only `run:` steps. An action that writes `$GITHUB_PATH` or `$GITHUB_ENV` must stay out of a `parallel:` block: that is the race the runner's implementation crashed on in `#76651`, which is why the setup actions stay outside it. This job already used the keyword for its lint group, so this extends a pattern the workflow relies on rather than adding one. `Save mypy cache`, the `schema.py` freshness check and the complexity steps move after the block, because mypy now finishes inside it. " Co-authored-by: Shelley <shelley@exe.dev>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Hey @fuziontech! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
Risk: No findings This PR moves the ty, mypy, and generated-projections checks into the existing Sentinel reviewed |
PR overviewNo security issues were found in this pull request. Reviews
|
🤖 CI report
|
|
Measured, and it is a regression. Closing without merging. What I expected: the What happened: the job took 528s against a like-for-like baseline of 382s, which is 38% slower. Both sides ran with every gated pytest suite switched on, because this PR edits Per step, same job, same gated suites:
The block itself took 402s and finished after mypy's 415s step had already ended, so the concurrency bought nothing and cost a lot. Why it fails: That also means the idea is not reachable from this job as it stands. It would need the Recording it under |
Problem
Every pull request that touches Python waits on the
Python code qualityjob inPython CI, and for the pull requests it ends, that job is the whole wait: it is 4.5 min at the median and 8.1 min at p90, second only to the Security lane. The job runs every static check one at a time on one runner, so its wall is the sum of six checks that read the same tree and share nothing.Changes
Python code qualitynow runs its checks as oneparallel:block, so it ends on the slowest check instead of on the total. Nothing a person sees changes: the same checks, the same commands, the same severities, and the same fail-on-finding behaviour.Check ty type checking(49s),Check static typing/ mypy (134s) andCheck if the generated projections are up to date(18s) join the lint and pytest group that already ran concurrently. mypy is the long pole, so the job is bounded by mypy alone.Save mypy cache, theschema.pyfreshness check and the complexity steps move below the block, because mypy now finishes inside it. This is mechanical.run:steps are in the block. An action that writes$GITHUB_PATHor$GITHUB_ENVstays out: that is the race the runner'sparallel:implementation crashed on in #76651, recorded under "Put the setup actions in aparallel:block" indocs/internal/ci-things-already-tried.md. The job already used the keyword for its lint group.flowchart LR subgraph before["Before: wall = sum"] direction LR b1[checkout 33s] --> b2[ty 49s] --> b3[mypy 134s] --> b4[projections 18s] --> b5[lint + pytest 15s] endflowchart LR subgraph after["After: wall = max"] direction LR a1[checkout 33s] --> a2 subgraph grp["parallel block"] direction TB a2[mypy 134s] a3[ty 49s] a4[projections 18s] a5[lint + pytest 15s] end a2 --> a6[cache save, schema, complexity] a3 --> a6 a4 --> a6 a5 --> a6 endHow did you test this code?
Test rationale: no new test. The change moves steps inside one job and rewrites no check, so the existing coverage is the checks themselves: the same
mypy,ty,build:projections --check, ruff and pytest commands run, and the job still fails when any of them fails. The evidence is the job's own duration on this PR's run against seven pre-change runs of the same job.Baseline, the
Python code qualityjob on 7 pull-request runs of.github/workflows/ci-python.ymlbefore this change:Serial steps inside those runs, medians: checkout 33s, dependencies 11s, ty 49s, mypy 134s, projections 18s, lint + pytest group 15s.
After, measured on this PR's own run: [pending: this PR's run]
Verification run locally:
hogli lint:workflows: 15 checks pass across 144 workflows.tools/workflow-plan: 491 tests pass, including the evaluator pass that says which jobs run on a draft, ready, fork, merge-queue and schedule event.Not run here: the job itself. This environment cannot build the full dev stack, so the timing above comes from CI rather than from a local run.
Docs update
No docs change:
docs/internal/ci-things-already-tried.mdalready records theparallel:rule this change relies on, and no new behaviour is documented.🤖 Agent context
Autonomy: Fully autonomous
Agent: Shelley, DeepSeek V4 Flash (
deepseek-flash-api-deepseek-com)Skills:
/authoring-ci-workflows,/writing-pr-descriptionsMore context for AI agents
depot-ubuntu-24.04-4instead..github/workflows/ci-python.yml. Every branch must stay arun:step.