diff --git a/TODO.md b/TODO.md index a79ad420..e9251ffb 100644 --- a/TODO.md +++ b/TODO.md @@ -245,7 +245,7 @@ One pull request per stage moving a standard workflow out of every repo and into - **Checked** - `develop` at `7c67328` on 2026-08-15, where the report counts 20 copies of `test-pull-request.yml` in 13 variants and 13 copies of `validate-task.yml` in 11, and the doc-lint block (markdownlint, cspell, actionlint, editorconfig-checker) repeats in every one. - **Open** - Whether the per-type lint steps are selected by an input the stub sets or read from the repo's registry entry through a hub checkout at `github.job_workflow_sha`, since the second needs no per-repo input and the first needs no network read. - **Open** - Whether a `validate` hook that runs a domain compile (an ESPHome build, a KiCad ERC) is one hook or several, given the two repos carrying such a step run it as a separate job today. - - **Settled** - Pilots are HomeAutomation-Config, the smallest operational tree, then a C# repo, so both trigger shapes are exercised before the sweep. + - **Settled** - Pilots are PhotoCleaner, which piloted the merge-bot stub in ptr727/PhotoCleaner#53 on 2026-08-15 as a release-model repo with Dependabot, C#, executable and Docker targets, then HomeAutomation-Config for the operational trigger shape, so both shapes are exercised before the sweep. - **Settled** - The step gated on `hashFiles('.github/actions/validate/action.yml') != ''` runs the caller's hook from its own checkout, else the default from a hub checkout under `.hub/`, and a local composite action resolves at step time from the workspace, which is what makes the fallback expressible at all. - **Host the pure functions: `get-version-task.yml` and `publish-plan-task.yml`.** Neither has a repo-specific line, and the plan job is missing where D4.1 needs it. @@ -518,7 +518,7 @@ Regenerate [reports/divergences.md][divergences-report] before using it as the w - **Adopt the merge-bot caller stub, which is one file per repo replacing the copied job bodies.** The audit reports the missing `merge-bot` caller job on every copy until the repo adopts, which is the work list. - **Hub state** - Done on `develop`, where `.github/workflows/merge-bot-task.yml` is the task and the hub's own `merge-bot-pull-request.yml` is the stub. The stub a repo copies is in [`docs/reusable-workflows.md`][reusable-workflows-doc] "Adopting the Merge-Bot", and its pin is the first hub release carrying the task, so no repo can adopt before that release. - - **Outstanding** - Every repo carrying the file, 16 today, pilot on HomeAutomation-Config, then homeassistant-purpleair for the `rules` input. + - **Outstanding** - Every repo still carrying the job bodies rather than the stub, 15 of the 16 copies after PhotoCleaner adopted in ptr727/PhotoCleaner#53 and promoted it in ptr727/PhotoCleaner#54 on 2026-08-15, then HomeAutomation-Config for the operational path and homeassistant-purpleair for the `rules` input. - **Issue** - [#521][issue-521], whose hub half is done and whose sweep half this is. - **Rides with** - The `verbatim` re-vendor above. - **Detail** - The unused `GITHUB_TOKEN` grants #521 names are gone with the copy, since the task declares none and the stub sets `permissions: {}`. diff --git a/docs/reusable-workflows.md b/docs/reusable-workflows.md index 1c36c5c1..866b32dd 100644 --- a/docs/reusable-workflows.md +++ b/docs/reusable-workflows.md @@ -60,7 +60,7 @@ A downstream `uses:` reads `ptr727/ProjectTemplate/.github/workflows/-task A downstream pull request may pin a hub feature-branch SHA to test a hub change that is still in flight, and re-pins to a released `main` SHA before it merges. `scripts/repo_gate.py check_sha_pin` reads the owner and repository out of the reference and confirms the SHA resolves, for a reusable workflow exactly as for an action. -The sequencing consequence is that a hub task lands on `develop`, promotes to `main`, and is released before any downstream carries a bumpable pin. The catalog snippet for a caller stub therefore lands one release after the task it names, since a snippet under `catalog/snippets/workflows/` is scanned by the same pin gate and cannot carry a placeholder SHA. The [Rollout](#rollout) section carries that ordering as checkboxes per stage. +The sequencing consequence is that a hub task lands on `develop`, promotes to `main`, and is released before any downstream carries a bumpable pin. The catalog snippet for a caller stub therefore lands one release after the task it names, since a snippet under `catalog/snippets/workflows/` is scanned by the same pin gate and cannot carry a placeholder SHA. The [Rollout][rollout] section carries that ordering as checkboxes per stage. ### Secrets and Permissions @@ -105,7 +105,7 @@ Docker Hub README publishing is a hub task of its own, `publish-docker-readme-ta This section is the tracker a session resumes from, and git is its only persistence. Every item is a checkbox with the evidence that closed it, a pull request, a commit, or a release tag, written into the item by the change that closed it. A session picking this work up reads this section first, takes the first unchecked item whose stage is open, verifies its claim against the current tree before acting on it, does the work in its own worktree, and ticks the item in the same pull request. Nothing here is ticked by intention: an item is ticked when the thing it names is on `develop`, or, for an adoption, on the named repo's ground-truth branch. [`TODO.md`][todo] "Hub-Hosted Reusable Workflows" carries the reasoning behind each stage, the open questions and what is settled, and this section carries the state. -A stage carries three kinds of item, plus a proof item where a claim needs a live run. **Hub** is the hub pull request that ships the task and its stub, and the catalog snippet that follows the release. **Release** is the promotion and release that gives the task a pinnable `main` commit, since a downstream stub pins a released tag and nothing can adopt before one exists. **Adoption** is one checkbox per repo, ticked when that repo's ground-truth branch carries the stub and the audit reports no `interface` finding on the file. **Proof** is a checkbox for a behavior only a live run demonstrates, ticked with the run URL. Stage 0 is the merge-bot's hub and release items plus its two proofs, and stage 1 is its adoption, split so the adoption list is a stage of its own. The exit metric per stage comes from [reports/workflow-reuse.md][workflow-reuse-report]: downstream copies of the stage's files fall to zero, callers rise to the number of repos that need the workflow, and downstream workflow lines fall from 10,964 toward the stubs plus the genuinely repo-specific hooks. Regenerate that report in the pull request that ticks a stage's last adoption, so the number and the tick land together. +A stage carries three kinds of item, plus a proof item where a claim needs a live run. **Hub** is the hub pull request that ships the task and its stub, and the catalog snippet that follows the release. **Release** is the promotion and release that gives the task a pinnable `main` commit, since a downstream stub pins a released tag and nothing can adopt before one exists. **Adoption** is one checkbox per repo, ticked when that repo's ground-truth branch carries the stub and the audit reports no `interface` finding on the file. **Proof** is a checkbox for a behavior only a live run demonstrates, ticked with the run URL. Stage 0 is the merge-bot's hub and release items plus its two proofs, and stage 1 is its adoption, split so the adoption list is a stage of its own. The exit metric per stage comes from [reports/workflow-reuse.md][workflow-reuse-report]: every downstream copy of the stage's files becomes a caller, so the callers column equals the copies column, and the report's downstream line total, 10,964 on the first run, falls toward the stubs plus the genuinely repo-specific hooks. Regenerate that report in the pull request that ticks a stage's last adoption, so the number and the tick land together. ### Stage 0: Design, Measurement, and the Merge-Bot Task @@ -117,10 +117,13 @@ A stage carries three kinds of item, plus a proof item where a claim needs a liv ### Stage 1: Merge-Bot Adoption -Adoptable since `2.0.338`. Each repo replaces the whole of its `.github/workflows/merge-bot-pull-request.yml` with the stub in [Adopting the Merge-Bot](#adopting-the-merge-bot), on its own feature branch, and the audit's `missing required job 'merge-bot'` finding on that file is the work list. The pilot goes first and records what the hub cannot prove, cross-repository resolution of the pin, the `rules` input where the repo has a tracker, and the first Dependabot bump of the pin, in its audit report. +Adoptable since `2.0.338`. Each repo replaces the whole of its `.github/workflows/merge-bot-pull-request.yml` with the stub in [Adopting the Merge-Bot][adopting-the-merge-bot], on its own feature branch, and the audit's `missing required job 'merge-bot'` finding on that file is the work list. The pilot goes first and records what the hub cannot prove, cross-repository resolution of the pin, the `rules` input where the repo has a tracker, and the first Dependabot bump of the pin, as proof items here. -- [ ] HomeAutomation-Config (pilot, operational model) -- [ ] homeassistant-purpleair (second, `rules: '[{"head-prefix": "ha-version-bump/", "base": "develop"}]'` and `delete-branch: true`) +- [x] PhotoCleaner (pilot, chosen as a release-model repo with Dependabot, C#, executable and Docker targets and a fresh resync, so what it shows is the mechanism): adopted on `develop` in ptr727/PhotoCleaner#53 at `a3158ce` and promoted to `main`, its ground-truth branch, in ptr727/PhotoCleaner#54 at `4efcae8`, both on 2026-08-15, where `python3 spec/audit.py PhotoCleaner` reports no `interface` finding on the file. The live proofs it owes, cross-repository resolution of the pin with a Dependabot PR to `develop` merged with `--squash` through the callee, and Dependabot bumping the pin, are the two proof items directly below, ticked with their evidence when they happen. +- [ ] Proof: the first `pull_request_target` run on PhotoCleaner `develop` after `a3158ce` resolves the owner-scoped `uses:` and merges the Dependabot PR that opened it. Tick with the run URL. +- [ ] Proof: Dependabot opens a `Bump ptr727/ProjectTemplate` PR on PhotoCleaner after the next hub release. Tick with the PR. +- [ ] HomeAutomation-Config (operational model, the direct-to-develop path) +- [ ] homeassistant-purpleair (third, `rules: '[{"head-prefix": "ha-version-bump/", "base": "develop"}]'` and `delete-branch: true`) - [ ] ESPHome-NonRoot (`delete-branch: true`, built-in upstream-version pairs cover its tracker) - [ ] NxWitness (`delete-branch: true`, drops the Dependabot semver-major filter per D8.1 unless the open decision lands first) - [ ] KiCadLibrary (drops the Dependabot semver-major filter per D8.1 unless the open decision lands first) @@ -131,7 +134,6 @@ Adoptable since `2.0.338`. Each repo replaces the whole of its `.github/workflow - [ ] Blog - [ ] ESPHome-Config - [ ] HomeAssistant-Config -- [ ] PhotoCleaner - [ ] PlexCleaner - [ ] Utilities - [ ] Vantage-Config @@ -140,7 +142,7 @@ Adoptable since `2.0.338`. Each repo replaces the whole of its `.github/workflow - [ ] EspDinIoT (same) - [ ] Financial-Modeling (same) - [ ] HolidayLights (same) -- [ ] `reports/workflow-reuse.md` regenerated with `merge-bot-pull-request.yml` at 0 copies carrying job bodies and callers equal to the adopters. +- [ ] `reports/workflow-reuse.md` regenerated with `merge-bot-pull-request.yml` showing callers equal to copies. ### Stage 2: The Gates @@ -150,10 +152,10 @@ Hub: `validate-task.yml` hosts the per-type doc-lint block once and calls the `v - [ ] Promoted and released, tag recorded here. - [ ] Catalog snippets for both stubs pinned to that release. - [ ] Hook fallback observed on a hub pull request run (default path) and on the pilot (override path), run URLs recorded here. -- [ ] HomeAutomation-Config (pilot, operational trigger shape) -- [ ] One C# repo (second pilot, release trigger shape with smoke) +- [ ] PhotoCleaner (pilot, release trigger shape with smoke, the same repo that piloted stage 1) +- [ ] HomeAutomation-Config (second pilot, operational trigger shape) - [ ] The remaining repos, one checkbox each added when the pilots close, since the sweep list is every cataloged repo. -- [ ] `reports/workflow-reuse.md` regenerated with `validate-task.yml` and `test-pull-request.yml` at 0 copies carrying job bodies. +- [ ] `reports/workflow-reuse.md` regenerated with `validate-task.yml` at 0 copies (a hub-only file no repo carries) and `test-pull-request.yml` showing callers equal to copies. ### Stage 3: The Pure Functions @@ -162,20 +164,21 @@ Hub: `get-version-task.yml` and `publish-plan-task.yml` hosted, and the downstre - [ ] Hub pull request on `develop`. - [ ] Promoted and released, tag recorded here. - [ ] Adoption, one checkbox per carrier added when the hub pull request merges: today `get-version-task.yml` has 8 carriers and `publish-plan-task.yml` 3. -- [ ] `reports/workflow-reuse.md` regenerated with both at 0 copies. +- [ ] `reports/workflow-reuse.md` regenerated with `get-version-task.yml` and `publish-plan-task.yml` at 0 copies, since both are hub-only files no repo carries. ### Stage 4: The Release Chain and the Docker Core -Hub: `build-release-task.yml` with `build-executable`, `build-nuget`, `build-pypi` and `release-assets` hooks, `publish-release-task.yml`, and `build-docker-task.yml` per [The Docker Family](#the-docker-family). The three no-asset release shapes collapse into `expect_release_assets`. +Hub: `build-release-task.yml` with `build-executable`, `build-nuget`, `build-pypi` and `release-assets` hooks, `publish-release-task.yml`, and `build-docker-task.yml` per [The Docker Family][the-docker-family]. The three no-asset release shapes collapse into `expect_release_assets`. - [ ] Hub pull request on `develop`. - [ ] Promoted and released, tag recorded here. -- [ ] PhotoCleaner and PlexCleaner (pilots, vanilla Docker plus executable) +- [ ] PhotoCleaner (pilot, vanilla Docker plus executable) +- [ ] PlexCleaner (second, the same shape) - [ ] VSCode-Server-DotNetCore (vanilla Docker only) - [ ] ESPHome-NonRoot (`docker-prepare` hook for the upstream pin) - [ ] NxWitness (matrix hook and `build-base`) - [ ] The NuGet, PyPI and remaining release repos, one checkbox each added when the pilots close. -- [ ] `reports/workflow-reuse.md` regenerated with `build-release-task.yml`, `publish-release.yml` and `build-docker-task.yml` at 0 copies. +- [ ] `reports/workflow-reuse.md` regenerated with `build-release-task.yml` and `build-docker-task.yml` at 0 copies (hub-only files) and `publish-release.yml` showing callers equal to copies. ### Stage 5: The Type-Specific Tasks @@ -230,7 +233,7 @@ A repo that needs either input appends the block to the `merge-bot` job. This is The task's inputs are `app-login` (default `ptr727-codegen[bot]`), `rules` (a JSON array of `{"head": ""}` or `{"head-prefix": ""}` plus `"base"`, default `[]`), and `delete-branch` (default `false`). The merge method follows the base, `develop` squashes and `main` merges, so a rule carries none. An App pull request that matches no rule is annotated with a warning rather than merged, so a renamed tracker branch is visible in the run rather than silent. -Two copies today filter Dependabot by ecosystem and semver tier before merging. [WORKFLOW.md D8.1][workflow-d8] says every Dependabot tier auto-merges and the required checks are the gate, so those two repos drop the filter on adoption unless the [Open Decisions](#open-decisions) below settle otherwise. +Two copies today filter Dependabot by ecosystem and semver tier before merging. [WORKFLOW.md D8.1][workflow-d8] says every Dependabot tier auto-merges and the required checks are the gate, so those two repos drop the filter on adoption unless the [Open Decisions][open-decisions] below settle otherwise. ## What a Pilot Proves @@ -244,13 +247,20 @@ Four things the hub cannot prove fall to the first downstream adopter. They are - **The Dependabot semver-major filter.** Two repos skip a nuget semver-major bump. Either it drops on adoption per D8.1, or the task grows a `skip-semver-major-ecosystems` input with a `dependabot/fetch-metadata` step run under the App token. Decide before those two repos adopt, everything else adopts unaffected. - **A `requiredHubUses` audit contract.** The interface check today asserts the task filename token in the caller job. A field asserting the full owner-scoped form on a downstream copy and the `./` form on the hub is a small schema extension. It waits for the first adoption to show whether the token check misses anything. + + +[adopting-the-merge-bot]: #adopting-the-merge-bot +[open-decisions]: #open-decisions +[rollout]: #rollout +[the-docker-family]: #the-docker-family + [governance-hub-hosted-tooling]: ../GOVERNANCE.md#hub-hosted-tooling [governance-workflow-yaml-conventions]: ../GOVERNANCE.md#workflow-yaml-conventions [secrets]: ../spec/secrets.json +[todo]: ../TODO.md [workflow]: ../WORKFLOW.md [workflow-d8]: ../WORKFLOW.md#d8---bots--automation [workflow-reusable-task-parameter-contract]: ../WORKFLOW.md#reusable-task-parameter-contract -[todo]: ../TODO.md [workflow-reuse-report]: ../reports/workflow-reuse.md diff --git a/reports/workflow-reuse.md b/reports/workflow-reuse.md index 58a09f9a..3e247df9 100644 --- a/reports/workflow-reuse.md +++ b/reports/workflow-reuse.md @@ -1,12 +1,12 @@ # Fleet workflow reuse report -Generated by `python3 spec/workflow_reuse.py --report` at hub `d08fea7` - do not hand-edit. Each row reads a repo's ground-truth branch at generation time and compares it against the hub canonical of the same name after line-ending, action-pin, and job-needs normalization, per [`spec/fidelity-model.md`][fidelity-model] "Normalization". Git dates this file. The target model and the migration phases are in [`docs/reusable-workflows.md`][reusable-workflows]. +Generated by `python3 spec/workflow_reuse.py --report` at hub `982b03d` - do not hand-edit. Each row reads a repo's ground-truth branch at generation time and compares it against the hub canonical of the same name after line-ending, action-pin, and job-needs normalization, per [`spec/fidelity-model.md`][fidelity-model] "Normalization". Git dates this file. The target model and the migration phases are in [`docs/reusable-workflows.md`][reusable-workflows]. ## Fleet Total -- **108 workflow files, 10,964 lines** across 20 downstream repos, 103 of them named for a hub canonical. No workflow at all in EspDinIoT. -- **4,201 lines (38%) are byte-identical to a hub canonical** after normalization, which is the confirmed duplication. The rest is mostly a per-repo edit of the same canonical rather than independent code. -- **Files reaching a hub reusable workflow or composite action through a pinned `uses:`: 1.** That is the state every carried copy converges to, so this number rises and the two above fall as the migration lands. +- **109 workflow files, 10,914 lines** across 20 downstream repos, 104 of them named for a hub canonical. No workflow at all in EspDinIoT. +- **4,266 lines (39%) are byte-identical to a hub canonical** after normalization, which is the confirmed duplication. The rest is mostly a per-repo edit of the same canonical rather than independent code. +- **Files reaching a hub reusable workflow or composite action through a pinned `uses:`: 2.** That is the state every carried copy converges to, so this number rises and the two above fall as the migration lands. ## Per Workflow @@ -15,17 +15,17 @@ Downstream copies of each hub canonical. A variant is a cluster of copies each a | File | Copies | Lines | Identical to hub | Variants | Callers | | --- | --- | --- | --- | --- | --- | | `build-release-task.yml` | 10 | 1,934 | 1,011 | 6 | 0 | -| `merge-bot-pull-request.yml` | 16 | 1,694 | 191 | 8 | 0 | -| `test-pull-request.yml` | 20 | 1,652 | 486 | 13 | 0 | -| `publish-release.yml` | 17 | 1,345 | 539 | 12 | 0 | -| `validate-task.yml` | 13 | 1,289 | 452 | 11 | 1 | +| `test-pull-request.yml` | 20 | 1,628 | 493 | 14 | 0 | +| `merge-bot-pull-request.yml` | 16 | 1,626 | 203 | 8 | 1 | +| `publish-release.yml` | 17 | 1,341 | 548 | 12 | 0 | +| `validate-task.yml` | 14 | 1,335 | 490 | 12 | 1 | | `build-docker-task.yml` | 5 | 625 | 298 | 4 | 0 | | `get-version-task.yml` | 8 | 495 | 364 | 4 | 0 | | `publish-plan-task.yml` | 3 | 252 | 207 | 1 | 0 | | `build-executable-task.yml` | 2 | 214 | 154 | 1 | 0 | | `deploy-site-task.yml` | 1 | 191 | 119 | 1 | 0 | | `run-codegen-pull-request-task.yml` | 2 | 161 | 140 | 1 | 0 | -| `check-upstream-version-task.yml` | 1 | 133 | 109 | 1 | 0 | +| `check-upstream-version-task.yml` | 1 | 133 | 108 | 1 | 0 | | `deploy-site.yml` | 1 | 55 | 41 | 1 | 0 | | `run-periodic-codegen-pull-request.yml` | 2 | 48 | 40 | 2 | 0 | | `build-datebadge-task.yml` | 1 | 37 | 27 | 1 | 0 | @@ -42,34 +42,35 @@ Each variant names the repos whose copies cluster together, so a hub task's inpu - 2: PhotoCleaner, PlexCleaner - 1: aiopurpleair - 1: homeassistant-purpleair -- `merge-bot-pull-request.yml` - - 1: Blog - - 6: ESPHome-Config, HomeAssistant-Config, HomeAutomation-Config, PlexCleaner, Utilities, Vantage-Config - - 3: ESPHome-NonRoot, NxWitness, homeassistant-purpleair - - 1: KiCadLibrary - - 1: LanguageTags - - 2: MediaTools, aiopurpleair - - 1: PhotoCleaner - - 1: VSCode-Server-DotNetCore - `test-pull-request.yml` - 2: AudioCleaner, Financial-Modeling - 1: Blog - 2: DevKitCIoT, HolidayLights - 1: ESPHome-Config - 1: ESPHome-NonRoot - - 2: HomeAssistant-Config, Vantage-Config + - 1: HomeAssistant-Config - 1: HomeAutomation-Config - 1: KiCadLibrary - 4: LanguageTags, MediaTools, Utilities, aiopurpleair - 1: NxWitness - 1: PhotoCleaner - 2: PlexCleaner, VSCode-Server-DotNetCore + - 1: Vantage-Config - 1: homeassistant-purpleair +- `merge-bot-pull-request.yml` + - 1: Blog + - 6: ESPHome-Config, HomeAssistant-Config, HomeAutomation-Config, PlexCleaner, Utilities, Vantage-Config + - 3: ESPHome-NonRoot, NxWitness, homeassistant-purpleair + - 1: KiCadLibrary + - 1: LanguageTags + - 2: MediaTools, aiopurpleair + - 1: PhotoCleaner + - 1: VSCode-Server-DotNetCore - `publish-release.yml` - 1: Blog - - 3: ESPHome-Config, HomeAssistant-Config, Vantage-Config + - 2: ESPHome-Config, Vantage-Config - 1: ESPHome-NonRoot - - 2: Financial-Modeling, HomeAutomation-Config + - 3: Financial-Modeling, HomeAssistant-Config, HomeAutomation-Config - 1: KiCadLibrary - 2: LanguageTags, MediaTools - 1: NxWitness @@ -83,6 +84,7 @@ Each variant names the repos whose copies cluster together, so a hub task's inpu - 1: Blog - 1: ESPHome-NonRoot - 1: Financial-Modeling + - 1: HomeAssistant-Config - 1: HomeAutomation-Config - 1: LanguageTags - 1: NxWitness @@ -128,16 +130,16 @@ Each variant names the repos whose copies cluster together, so a hub task's inpu | Blog | 6 | 560 | 317 | 0 | - | | DevKitCIoT | 1 | 58 | 25 | 0 | - | | ESPHome-Config | 3 | 450 | 90 | 0 | - | -| ESPHome-NonRoot | 11 | 1,154 | 463 | 0 | `check-upstream-dependency.yml`, `check-upstream-version.yml` | +| ESPHome-NonRoot | 11 | 1,154 | 462 | 0 | `check-upstream-dependency.yml`, `check-upstream-version.yml` | | Financial-Modeling | 3 | 233 | 124 | 0 | - | | HolidayLights | 1 | 53 | 25 | 0 | - | -| HomeAssistant-Config | 3 | 229 | 88 | 0 | - | +| HomeAssistant-Config | 4 | 247 | 142 | 0 | - | | HomeAutomation-Config | 4 | 248 | 144 | 0 | - | | KiCadLibrary | 6 | 772 | 221 | 0 | - | | LanguageTags | 7 | 724 | 265 | 0 | - | | MediaTools | 5 | 530 | 173 | 0 | - | | NxWitness | 10 | 1,102 | 363 | 0 | `build-base-images-task.yml` | -| PhotoCleaner | 8 | 827 | 457 | 1 | - | +| PhotoCleaner | 8 | 759 | 469 | 2 | - | | PlexCleaner | 8 | 805 | 455 | 0 | - | | Utilities | 6 | 621 | 242 | 0 | - | | VSCode-Server-DotNetCore | 8 | 563 | 297 | 0 | - |