Problem
The Iceberg core tests are sharded by a task-agnostic class-name predicate, but iceberg-spark-extensions runs unsharded (iceberg_spark_test_reusable.yml:221-227) and measured 55m24s in queue run 35022044196 — the single longest job in that run and the largest ownable saving.
Change
- Give the extensions job the same shard matrix: pass
-PcometShardIndex / -PcometShardCount and -PcometShardTask for the extensions task, with a task-specific artifact name.
- Add an extensions coverage check alongside the existing core one.
Expected gain
55m24s unsharded → ~15-20 min ⇒ ~35 queue-min/run × ~38 runs ≈ 1,300 runner-min/72 h (~2 %). The largest single ownable saving.
Constraint
dev/ci/check-iceberg-shards.py:43 hard-codes SHARD_COUNT = 4 and asserts every manifest's count equals it, so extensions must use count 4 (or the constant must become per-task). The Gradle predicate is a pure hash of the class name and is task-agnostic (dev/ci/iceberg-test-shards.gradle).
Verification
- One label run produces balanced per-shard candidate counts (extensions balance is unverified until then) and the coverage check passes: the union of shard candidates equals the unsharded inventory.
- Queue wall clock for the extensions job 55m24s → target ~15-20 min.
Risk / rollback
Shard imbalance could make one shard as slow as the unsharded job; the coverage check prevents dropped tests. Revert the workflow and the matrix emitter.
Problem
The Iceberg core tests are sharded by a task-agnostic class-name predicate, but
iceberg-spark-extensionsruns unsharded (iceberg_spark_test_reusable.yml:221-227) and measured 55m24s in queue run35022044196— the single longest job in that run and the largest ownable saving.Change
-PcometShardIndex/-PcometShardCountand-PcometShardTaskfor the extensions task, with a task-specific artifact name.Expected gain
55m24s unsharded → ~15-20 min ⇒ ~35 queue-min/run × ~38 runs ≈ 1,300 runner-min/72 h (~2 %). The largest single ownable saving.
Constraint
dev/ci/check-iceberg-shards.py:43hard-codesSHARD_COUNT = 4and asserts every manifest's count equals it, so extensions must use count 4 (or the constant must become per-task). The Gradle predicate is a pure hash of the class name and is task-agnostic (dev/ci/iceberg-test-shards.gradle).Verification
Risk / rollback
Shard imbalance could make one shard as slow as the unsharded job; the coverage check prevents dropped tests. Revert the workflow and the matrix emitter.