Shard exhaustive CI runtime suites under five minutes

Motivation

CI is the exhaustive correctness authority: every applicable TypeScript test runs under Deno, Node, and Bun. The corpus has grown beyond the useful wall-clock range of one job per runtime. At 5542c92, the standalone Deno suite took 22m07s, and the same suite reached the composability battery's former 30-minute ceiling.

This issue shortens the exhaustive CI critical path by splitting each runtime suite into deterministic weighted shards. It does not change which tests run.

Owner ruling (2026-08-22): the local affected-test selector formerly included in this issue is dropped. Deno's native --changed and --related commands remain the documented local affected-test mechanism. Closed PR #311 is not an implementation dependency.

Owner ruling (2026-08-22): choose the smallest measured shard counts that keep every exhaustive runtime suite below 300 seconds after runtime and dependency setup, for five consecutive CI runs. Setup, checkout, queue time, and dependency downloads are reported separately. If a run misses the target, update weights or shard counts and begin a fresh sequence of five runs.

Exhaustive sharding contract

For Deno, Node, and Bun, CI runs the complete discovered TypeScript test corpus minus only that runtime's recorded exclusions. No test runs in fewer runtimes than it does before this change.

scripts/lib/test-files.ts is the single source of discovery. Its result is the same corpus Deno's workspace test glob discovers. The recorded exclusions in scripts/runtime-test-exclusions.ts are applied to that set per runtime, retaining every reason and issue link.

Each runtime's applicable corpus is partitioned into deterministic, disjoint shards whose union is exactly that corpus. Every applicable file runs exactly once per runtime. A new or unmeasured file still belongs to exactly one shard and receives a conservative non-zero fallback weight.

Weights come from the corresponding runner's own per-file timing on the GitHub-hosted runner. They are versioned repository data changed only by an explicit remeasurement command; ordinary verification never rewrites them. Shards use deterministic longest-processing-time-first assignment, with stable tie-breaking, to minimize the longest shard rather than equalizing file counts.

The shard-count decision is empirical. Select the smallest counts that satisfy the five-run, 300-second target above. Record before/after per-file, per-shard, per-runtime, and overall makespan evidence together with the Actions run URLs and attempts.

All shard jobs run concurrently and preserve #279's ownership, cancellation, output, and clean-worktree invariants. A shard runs its assigned files serially through the runtime's existing runner; Deno --parallel and equivalent in-shard parallelism are outside this issue.

CI aggregation

The existing required green aggregate remains the delivery authority. It requires every runtime shard job and fails if any shard is missing, skipped unexpectedly, cancelled, crashes, or reports a test failure. The workflow regression parses the shard jobs, proves green.needs covers them, and proves the partition for each runtime is deterministic, disjoint, and exhaustive.

Check and log names identify the runtime and shard. Each shard retains complete first-failure output. The current exhaustive commands remain available for humans, debugging, composability verification, and weight validation.

Scope

Included:

deterministic weighted sharding of test-deno, test-node, and test-bun;
one discovery source and the existing recorded runtime exclusions;
explicit weight measurement and versioned weights;
shard-count calibration and five consecutive timing runs;
green and workflow regressions covering every shard; and
documentation of the exhaustive CI topology and remeasurement procedure.

Not included:

a repository-specific affected-test selector or any revival of PR ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) #311;
changes to the documented local affected-test policy;
Deno --parallel or other parallel execution inside a shard;
changes to what the composability job runs or when it runs; or
a new runtime-tests required check or a repository-ruleset mutation.

Plan gate

Before code, submit a short plan that identifies:

the one discovery and exclusion flow shared by all three runtimes;
the runner-native timing seam and versioned weight format;
deterministic LPT assignment, fallback weights, and exact-once partition proof;
the smallest measured shard counts proposed for the 300-second target;
workflow job and green.needs structure, including failure and cancellation behavior;
regressions for discovery equivalence, exclusions, deterministic/disjoint/exhaustive partitioning, unmeasured files, and workflow aggregation; and
the before/after measurement and five-consecutive-run validation procedure.

Acceptance criteria

Make the full verification battery safe to run concurrently #279 is closed and its concurrency, cancellation, output, and cleanliness invariants remain intact.
test-deno, test-node, and test-bun each run as deterministic weighted shards.
Every applicable test runs exactly once per supported runtime, minus only that runtime's recorded exclusions; no existing runtime coverage is lost.
Versioned runner-native timing weights and an explicit remeasurement command exist; ordinary verification does not rewrite the weights.
New or unmeasured tests receive a conservative non-zero weight and run in every applicable runtime.
The selected shard counts are the smallest measured counts for which five consecutive CI runs keep every runtime suite below 300 seconds after setup.
Timing evidence records run URLs and attempts, setup-excluded boundaries, per-file and per-shard durations, per-runtime makespan, overall makespan, and runner cost.
green requires every shard, and the CI workflow regression proves coverage plus deterministic, disjoint, exhaustive partitioning.
Logs identify runtime and shard and retain complete first-failure output.
The full standing delivery battery and compiled smoke checks pass on the final PR head.

Sequencing ruling

Research and mechanism planning for #280 may proceed in parallel with the approved Stage-1 repair for #279, because they create no repository changes or competing PR.

Implementation remains blocked until the #279 repair and Stage 2 have merged and #279 is closed. Do not open a #280 implementation branch or PR before then. Revalidate the plan against the final #279 interfaces before code.

Parallel work should use an isolated clean clone and scratch caches so it cannot interfere with #279. Prioritize the two empirical uncertainties that determine the design: the exact supported seam for capturing Deno 2.9.1's native selected test-file set, and per-file/per-runtime timing plus runner overhead for choosing deterministic shard counts. Produce the complete plan required by this issue and stop for planner review; do not implement opportunistically.

Owner rulings for the #280 mechanism plan

The owner approved the following conservative selection and rollout policy:

A deletion and the old side of a rename trigger the full runtime-test corpus. The source side of a copy is not a deletion and must not be classified as one.
Runtime-loaded documents and test fixtures trigger the full corpus. Do not narrow fixture changes by scanning test source for literal path strings; that can miss computed cross-member reads.
If Deno cannot prove that a changed TypeScript path reaches tests, including a target reachable only through a non-analyzable dynamic import, run the full corpus and explain the fallback.
The ignored generated browser bundle is not a branch/worktree change input. Its tracked generator inputs trigger the full corpus; merely touching the ignored generated output does not.
Add an explicit no runtime tests class for known test-irrelevant paths such as CI-only and lint/format-only configuration. Anything not deliberately classified remains unknown and triggers the full corpus.
Deno's currently failing intra-suite --parallel mode remains separate work. Shard exhaustive CI runtime suites under five minutes #280 obtains parallelism through isolated exhaustive CI shards; Make the full verification battery safe to run concurrently #279 owns safe concurrency among the repository commands.
After the sharded workflow has landed with the stable aggregate check name, configure the repository ruleset so runtime-tests is required. This is a separate owner-controlled post-merge configuration step; do not claim the check is required merely because the workflow defines it.

These rulings amend the #280 contract and must appear in the next mechanism-plan revision before code.

Planner review of `enchanted-swimming-sundae.md` — FAIL

Reviewed the complete 522-line artifact, SHA-256 6c19bbcf6781610ab163a01e71f5576e923c496211c79ebd77333ba7bc03e151, against current #280, all #279 amendments, current origin/main (4a13be7), and open PR #304. The empirical selection and timing research is strong, and the per-candidate Deno seam plus weighted-sharding direction are retained. The plan cannot pass while its trigger table contradicts its own probes and the newly recorded owner rulings.

Prescriptive revision prompt:

Revise ~/.claude/plans/enchanted-swimming-sundae.md only; do not create a branch, change repository code, or begin #280 implementation. Preserve the measured Deno 2.9.1 per-candidate --no-run --changed seam, the single selected set flowing to all three runtimes, LPT weighting, exhaustive CI, failure aggregation, and the dependency on completed #279.

Incorporate every owner ruling in Shard exhaustive CI runtime suites under five minutes #280 comment 5161001826 and remove §12's resolved questions. Deletions and only the old side of R trigger the full corpus; the source side of C is not a deletion. Runtime-loaded documents and all non-test fixture/data files under member tests/ trigger the full corpus. Remove the literal-path scan and the narrower member-local fixture rule. Keep Deno --parallel out of Shard exhaustive CI runtime suites under five minutes #280 and state that isolated CI shards provide this issue's parallelism.

Close the non-analyzable dynamic-import hole. §1.4 proves a variable-import target can select zero tests, but §2 currently classifies every remaining .ts/.tsx path as native-authoritative and supplies no fallback. Specify a supported Deno-native proof/fallback that makes any changed TypeScript path whose test reachability cannot be proven trigger the full corpus, without implementing a second module graph. Measure its cost and add the controlled variable-import regression that fails against the current table. If no acceptable native seam can establish that distinction, say so and use the approved full-corpus fallback; do not silently rely on a registry or source-text heuristic.

Make the changed-path classification internally consistent and total. The current research calls .github/workflows/ci.yml and .oxlintrc.json correctly test-irrelevant, but the table has no such class, so both land in unknown and run everything. Add an exact, deliberately bounded no runtime tests class for known CI-only and lint/format-only paths; keep anything unmatched as unknown → full corpus. Remove packages/web/generated/** from change inputs: it is ignored, and both Deno and the proposed git ls-files --others --exclude-standard input omit it. Retain full triggers for its tracked generator sources. Add regressions for the no-test class, unknown fallback, ignored generated output, and tracked generator inputs.

Resolve the failure-policy contradiction. §2 says an unresolvable base is a hard error, while §3 says unreadable git state and an unresolvable base may escalate to the full corpus. State one exact policy: inability to resolve the requested base, find the repository, or obtain a trustworthy changed-path set exits non-zero with remediation because no comparison state exists; an unexpected candidate-selection probe failure after a trustworthy change set exists falls back to the full corpus, names the failed probe, and still returns the exhaustive runners' result. Pin both paths with regressions.

Make controlled-change validation automated. The plan currently mixes a checked-in harness with a “documented manual matrix” while claiming a future Deno semantic change will fail a fixture. Every selection/blind-spot and trigger-table row that protects correctness must run in repository tests against scratch git repositories; reserve manual evidence for timing calibration only. Include type-only imports, additions, staged/unstaged/untracked state, deletion, rename, copy status, variable dynamic import, documents, fixtures, generated inputs, ignored output, configuration, and unknown paths.

Split implementation into a sequential small-PR chain, one PR open at a time. PR A lands the discovery/selection/sharding libraries, affected command, automated regressions, weight-measurement command, and the default-branch workflow_dispatch measurement hook, but does not yet make affected mode the AGENTS default or replace exhaustive CI. After PR A merges and stops for review, PR B records CI-hosted weights, installs the exhaustive shard matrix and aggregate check, updates specs/testing-spec.md and AGENTS.md, runs the five timing validations, and closes Shard exhaustive CI runtime suites under five minutes #280. Keep Stage 2 of Make the full verification battery safe to run concurrently #279 merged and Make the full verification battery safe to run concurrently #279 closed before opening PR A.

Correct the required-check rollout. The repository currently has a pull-request ruleset but no required status checks. Defining a job named runtime-tests does not make it required. The plan must keep that aggregate name stable, validate it on PR B, merge PR B only after owner review of its green final head, then identify a separate owner-controlled post-merge ruleset update that makes runtime-tests required. Include an API verification of the resulting ruleset; do not attempt to require a not-yet-present check before merge.

Make the five-run timing procedure executable on the PR B head. Because the measurement workflow hook lands in PR A on the default branch, dispatch it against PR B's ref five times; record Actions run URLs/attempts, setup-excluded start/end boundaries, every shard duration, per-runtime and overall makespan, and runner cost. Every run must execute the exhaustive applicable corpus exactly once and remain under 180 seconds. Any miss is corrected by new CI-derived weights or shard counts and five fresh consecutive passing runs.

Add final acceptance verification to the plan: rebase/remeasure after Make the full verification battery safe to run concurrently #279 and PR 🤖 Freeze the lock at the task boundary, and prove builds change nothing #304/Stage 2 change the corpus and flags; run setup once; run the complete pinned battery (lint, check, test, check:jsr, Node tsc, Node tests, Bun tests, core xmd test, and applicable site checks), the new affected controlled-change matrix, exhaustive shard-union/disjointness checks, failure/cancellation cleanup tests, and tracked-tree cleanliness. Implementation must follow the repository and linked Effection AGENTS rules: generator operations, yield*, allSettled when all three runtimes must finish, observed cleanup via ensure, @effectionx/fs where available, no casts, Promises, async, or await, and the current PR template/gitmoji rules.

Return the complete revised plan and stop for another planner verdict. Do not ask the implementor to choose among fallback behavior, PR boundaries, or ruleset rollout; those decisions are fixed above.

Planner review of `enchanted-swimming-sundae.md` revision 2 — FAIL

Reviewed the complete 583-line artifact, SHA-256 8ba7219ecac7d36ca1609b76fda6a42b0657306d4264890a4443cb58846e670d, against the current issue contract, owner rulings, repository AGENTS.md, Effection v4 AGENTS.md, and the live repository ruleset. The revised Deno-native reachability proof, total trigger table, failure policy, automated scratch-repository validation, sequential PR split, and delayed required-check rollout resolve the previous review findings.

One blocking factual defect remains. Section 6 says ruleset main (id 13643935) has an empty rules array. The live API response instead contains three active rules: deletion, non_fast_forward, and pull_request with its current parameters. There is no required_status_checks rule, which is the narrower fact the plan needs. Updating a ruleset requires supplying its rules, and GitHub also requires an explicit strict/loose status-check policy; neither may be left for the implementor to choose or allowed to erase existing protection.

Prescriptive revision prompt:

Revise ~/.claude/plans/enchanted-swimming-sundae.md only; do not create a branch, change repository code, begin #280 implementation, or modify the GitHub ruleset. Preserve revision 2's selection, reachability, trigger-table, failure, regression, sharding, timing, two-PR, #279 dependency, and AGENTS.md-compliance design.

Correct §6 and every derived claim to match the live ruleset. Ruleset main id 13643935 is active and currently contains deletion, non_fast_forward, and pull_request rules; only required status checks are absent. Record the existing pull-request parameters exactly from a fresh API read instead of calling the rules array empty.

Make the post-PR-B owner-controlled operation explicitly additive and non-destructive: immediately before any mutation, re-read the complete ruleset; preserve its enforcement, target conditions, bypass actors, every existing rule, and every existing rule parameter; add only a required_status_checks rule naming the stable runtime-tests context; then re-read and assert both that runtime-tests is required and that all pre-existing rules and parameters remain unchanged. If the live ruleset differs from the reviewed snapshot, stop and return the new state to the planner rather than overwriting it.

GitHub requires strict_required_status_checks_policy when adding that rule, and strict versus loose changes whether a PR must be tested against the latest base. Do not assign that product behavior to the implementor. State that the separate ruleset operation remains blocked on an owner ruling for strict versus loose policy, to be recorded on #280 before the mutation. Keep the ruleset mutation outside PR A and PR B, and stop again for planner review after revising the plan.

Planner review of `enchanted-swimming-sundae.md` revision 3 — PASS

Reviewed the complete 627-line artifact, SHA-256 9a7977dfdc499dcd10291178e8957e6813110b73d7c824d54482a3c386ca165c, against the current #280 contract and owner rulings, repository AGENTS.md, Effection v4 AGENTS.md, current main (4a13be72ee2cecbb09c25b2bd5878fc0bddbef13), and live ruleset 13643935.

Revision 3 resolves the remaining defect. Its snapshot exactly matches the live active ruleset: deletion, non_fast_forward, and pull_request with the recorded parameters; no required_status_checks rule; unchanged default-branch conditions and empty bypass list. The later owner-controlled operation is additive, preserves and verifies every existing protection, stops on stale state, remains outside PR A and PR B, and does not ask the implementor to choose strict versus loose status policy.

The unresolved strict-versus-loose choice does not block either implementation PR. It remains an explicit owner decision that must be recorded on #280 before the separate post-PR-B ruleset mutation. All previously approved selection, reachability, conservative fallback, automated validation, weighted exhaustive sharding, sequential-PR, #279 dependency, and AGENTS.md-compliance mechanisms remain intact.

Planner review of PR #311 at 490de0b65d6b80a4ddc6cda196fcbaf0dc47b909: FAIL.

All ten checks are now complete and green, the head is mergeable, and there are no review threads. Those gates do not resolve the contract failures below.

First, PR #311 was opened before its explicit prerequisite. #280 comment 5160700394, the approved revision-3 plan, and the plan's delivery section all say PR A must not open until #279 Stage 2 has merged and #279 is closed. #279 is still OPEN, no Stage-2 PR has merged, and #311's own body acknowledges that Stage 2 remains open. The required final-interface and flag revalidation therefore could not have happened.

Second, the implementation violates the repository and approved-plan code rules. scripts/affected-tests.ts and scripts/measure-weights.ts add top-level await main(...); scripts/affected-tests.ts adds as const; and the scratch dynamic-import fixture adds an explicit Promise annotation. The approved plan explicitly requires no casts, Promises, async, or await.

Third, the PR body claims the full automated regression suite, but the approved §7.1 matrix is incomplete. The current tests do not exercise a real cross-workspace transitive edge; the --no-check negative half of the type-only proof; a real copy through changeSet; staged git rm; rename into and out of a triggering directory; an analyzable template dynamic import; a non-analyzable dynamic target that the test actually loads at runtime; every full-trigger and no-runtime class through the scratch-repository selection pipeline; the tracked bundle-generator input alongside the ignored output; or command-level aggregation/cancellation/cleanliness. There is no regression proving an unreadable base produces the promised remediation at the command boundary, all three runners finish when one fails, full first-failure output is retained, cancellation leaves no child process, or a full affected run leaves tracked content and modes unchanged.

On the user's question: agents do not currently know to run affected tests. AGENTS.md is unchanged and still requires the exhaustive Deno test task. That is intentional for PR A under the approved two-PR rollout; PR B is the PR that changes the agent default, documents separately applicable checks and --all, installs exhaustive sharded CI, and closes #280. Do not move that policy change into PR A merely to answer the question early.

Prescriptive repair prompt:

Stop #280 implementation and preserve the current branch, but do not merge PR #311. Close PR #311 rather than leaving a competing implementation PR open. Complete #279 Stage 2 in its own reviewed PR, merge it, and close #279 first, exactly as the recorded dependency requires.

After #279 closes, rebase the preserved #311 work onto final origin/main, re-read the final #279 task flags, harness, scratch-path, cancellation, and cleanliness interfaces, and return a short revalidation note before retaining code. If those interfaces require a nontrivial redesign, submit the mechanism amendment for planner review before editing further.

Then repair and reopen PR A:

Restore AGENTS compliance without changing PR A's policy boundary. Replace both top-level await main(...) entrypoints with the repository's observed main(...) pattern; replace as const with an explicitly typed runtime list/union that uses no cast; remove the scratch fixture's explicit Promise annotation while preserving the dynamic-import case. Run a canonical added-line sweep for Promise, async, await, and as casts.

Complete the approved scratch-repository matrix. Add controlled cases for a real cross-workspace transitive import; type-only selection with checking on and the same candidate not selected under --no-check; a production-path copy; staged git rm; renames into and out of a trigger; analyzable template and actually runtime-loaded non-analyzable dynamic imports; every full-trigger class and every bounded no-runtime class through selection rather than regex-only unit tests; and the exact ignored generated-output/tracked-generator-input pair.

Test the affected command's behavior, not only its selection library: unreadable base exits nonzero with the documented remediation and starts no runner; a failed probe widens and runs all three; one runtime failure does not cancel the others, preserves complete first-failure output, and makes the aggregate exit nonzero; cancellation halts every child process group; and success, failure, and cancellation leave no owned scratch state or tracked content/mode change. If making these observable needs a new nontrivial runner/process seam not already authorized by the approved plan, stop with a short mechanism amendment rather than hiding behavior behind untested hardwired commands.

Correct the PR body so it claims only evidence the reopened head contains. Keep the explicit statement that PR A does not change agent policy: agents continue using the exhaustive instructions until PR B updates AGENTS.md. Do not add the affected default, sharded CI, weights, testing spec, or required-check policy to PR A.

Run the complete pinned battery and the finished controlled-change/cancellation matrix on the rebased final head, prove tracked-tree cleanliness, push, wait for every latest-head check to complete green, and verify zero unresolved threads before requesting review.

Stop after PR A. PR B remains sequential and is where agents are instructed to run deno task test:affected plus separately applicable non-test checks, while CI becomes exhaustive and sharded.

Owner ruling (2026-08-22): rescoped to shards only

This issue now delivers one thing: the exhaustive CI suites run as weighted shards to minimize wall-clock makespan. The local affected-test selector (PR A, #311) is dropped from this issue's scope.

Why now: the Deno suite is 22m07s standalone at 5542c92 (305 files, 677 tests), and under the composability battery it outgrew its 30m ceiling and reddened main (#538, #539). The sharding half of the approved revision-3 plan was measured and proven (weighted split executed at 75/71/68s against 212s serial on the Aug-2 corpus) and does not depend on the selector; the selector half is what failed review on #311 and is what has kept the whole issue stalled since 2026-08-03. Deno-native deno test --changed / --related already gives agents a documented affected-mode command (AGENTS.md, Test selection); a repository selector can return as its own issue if that proves insufficient.

What carries over unchanged from the approved plan: weight measurement from the runner's own timing, longest-processing-time-first assignment, one discovery source (scripts/lib/test-files.ts = Deno's glob), the recorded per-runtime exclusions applied to the same set, green.needs covering every shard job (the ci-workflow regression test enforces it), and CI remaining the correctness authority — every applicable test under every supported runtime, once.

What is not in scope here: --parallel inside a shard (the suite is not parallel-safe; separate issue), and any change to what the composability job runs (separate issue on the battery).

Acceptance for the rescoped issue: test-deno, test-node, test-bun each split into N weighted shard jobs with recorded before/after wall clock; the green aggregate requires every shard; the ci-workflow test parses the shard jobs; no test runs in fewer runtimes than it did.

Navigation Menu

Repository navigation

Shard exhaustive CI runtime suites under five minutes #280

Motivation

Exhaustive sharding contract

CI aggregation

Scope

Plan gate

Acceptance criteria

Sequencing ruling

Owner rulings for the #280 mechanism plan

Planner review of `enchanted-swimming-sundae.md` — FAIL

Planner review of `enchanted-swimming-sundae.md` revision 2 — FAIL

Planner review of `enchanted-swimming-sundae.md` revision 3 — PASS

Owner ruling (2026-08-22): rescoped to shards only

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Shard exhaustive CI runtime suites under five minutes #280

Description

Motivation

Exhaustive sharding contract

CI aggregation

Scope

Plan gate

Acceptance criteria

Activity

taras commented on Aug 2, 2026

Sequencing ruling

taras commented on Aug 3, 2026

Owner rulings for the #280 mechanism plan

taras commented on Aug 3, 2026

Planner review of enchanted-swimming-sundae.md — FAIL

taras commented on Aug 3, 2026

Planner review of enchanted-swimming-sundae.md revision 2 — FAIL

taras commented on Aug 3, 2026

Planner review of enchanted-swimming-sundae.md revision 3 — PASS

taras commented on Aug 3, 2026

taras commented on Aug 22, 2026

Owner ruling (2026-08-22): rescoped to shards only

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Planner review of `enchanted-swimming-sundae.md` — FAIL

Planner review of `enchanted-swimming-sundae.md` revision 2 — FAIL

Planner review of `enchanted-swimming-sundae.md` revision 3 — PASS