Tooling: overlap bit589 and complex741 in the #741 ancestry replay, 17% (50% with --complex-jobs 3) less wall time (no new kappa)

[rcfc-jreed]

Concurrent bit/complex replay of the #589 + #741 ancestry

Status: tooling improvement. It changes no construction, bound, certificate or
selected record, and claims no new kappa.

What changed

scripts/verify_frontier741.py (the cold ancestry replay that verify_frontier799.py
runs, and hence the selected make frontier1042-verify chain) runs bit589 to
completion and then complex741. Neither reads anything the
other writes: bit589 writes only under <work>/bit, complex741 only under
<work>/complex, and the binding step needs both. The new
scripts/verify_frontier741_concurrent.py runs the same two commands at the same
time, then the same binding script.

verify_frontier741.py is untouched because its sha256 is pinned in
certificates/selected-result.json and the checkpoint certificates. The new driver
is a sibling that writes the same layout (bit/, complex/, bit589.log,
complex741.log, STEPS.json, RESULT.json) plus a CONCURRENCY.json with wall and
summed job times. If either supplier fails, the other is stopped (process group
SIGTERM, then SIGKILL after 20 s), the failing log is named, and nothing is bound.
--complex-jobs N (N>1) swaps in the existing dependency-scheduled complex replay
(verify_complex741_parallel.py, #853); the default of 1 runs verify.py unchanged so
that this change is measured on its own.

Reproduce

python3 -B scripts/verify_frontier741_concurrent.py --work /absolute/fresh/dir
python3 -B scripts/verify_replay_coverage.py --frontier 741 --work /absolute/fresh/dir
python3 -m unittest tests.test_frontier741_concurrent

Expected: the first prints PASS selected finite composition; inherited hypotheses remain, and RESULT.json has status PASS_REBUILT_PR589_PR741_COMPOSITION, kappa
1661202643272239/1000000000000000000. The second prints
PASS_COMPLETE_PR741_REPLAY_COVERAGE. The tests (8) take about 3 s.

Developed and measured at base commit 2921874911956718fcaaaf9396fe8267b3cfa9f0.
The branch is rebased onto ba82555c (main when submitted); verify_frontier741.py,
its pinned hash, the bindings script and both supplier packages are unchanged between
the two, and the tests were re-run on the rebased tree. Benchmark environment (not the
CI image): WSL2 Ubuntu 24.04, 28 logical CPUs, 7 GB RAM; Python 3.12.15, numpy 2.5.4,
scipy 1.18.1, sympy 1.14.0; the C++17 checkers built with zig c++ (ziglang 0.17.0,
clang 22.1.8) against Boost 1.86.0 headers. No system compiler was available.

Measurement

Same checkout, same fresh output path each time, one run at a time on an otherwise idle
machine. The first row used different checkout/output paths and is timing only.

Driver	Wall seconds	bit589	complex741
original, sequential	608	138	469
original, sequential	599	–	–
original, sequential	562	–	–
concurrent	495	141	495
concurrent	479	–	–
concurrent, --complex-jobs 3	293	150	293

Sequential mean 590 s, concurrent mean 487 s: about 17% less wall time, 100 to 120 s
saved. The concurrent run is bounded below by complex741 (critical path 469 to 495 s),
so the bit supplier is hidden but nothing else is. With --complex-jobs 3 the complex
chain shortens to the dependency chain of #853 and the bit supplier is again fully
hidden: 293 s, about 50% less than sequential (one sample). That is 3 sequential, 2
concurrent and 1 combined samples on one machine; treat the ratios as indicative, not
as precise speedups. #853 alone would leave bit589 (about 140 s) in series with the
complex replay; that exact configuration was not run here.

Coverage is unchanged

• tests/test_frontier741_concurrent.py parses verify_frontier741.py and requires the
  concurrent driver's bit589, complex741 and bindings commands, stage order and
  environment scrub (PYTHONOPTIMIZE, WITNESS_DISCOVER, WITNESS_RECORD, SW4_PIN)
  to equal the original's. If the original changes, the test fails.
• Output trees of the original and concurrent drivers (same path) were compared file by
  file: 385 common files, 101 differ. A second original run differs from the first in
  exactly the same 101 files, so the concurrent schedule adds no new difference. After
  removing seconds/elapsed fields and SHA-256 values (digests of files that embed
  timings), every differing JSON file is identical in both comparisons. The remaining
  differing files are logs.
• The concurrent output passes verify_replay_coverage.py --frontier 741 and the
  original bindings check.
• Negative controls on a copy of the concurrent output: a changed bit_lower_decimal
  fails the bindings check (Rebuilt bit differs); one extra byte appended to
  complex/global-replay/ASSEMBLY.json fails coverage (Required ancestry artifact hash differs); a deleted bit/xcheck.json fails.
• Mutation controls on the tests: replacing the bit verifier, adding -O to the complex
  command, removing SW4_PIN from the scrub, not stopping the sibling after a failure,
  and waiting for each job right after launching it are each caught. (The SW4_PIN
  mutation first survived; the test now sets the variable before checking the scrub.)

Scope and open items

• Finite replay tooling only. It is not a formal proof, and every inherited assumption
  listed in certificates/selected-result.json remains.
• The output is accepted as an ancestor by the coverage and bindings checks, so it can
  be passed as --ancestor to verify_frontier799.py. That was run once (WSL, default
  concurrent mode, base 2921874; the 799 scripts are unchanged on later main):
  verify_frontier799.py --work W --ancestor <output> passed in 203 s with
  PASS_REBUILT_PR589_PR799_COMPOSITION, kappa 1661634653669567/10^18, and
  verify_replay_coverage.py --frontier 799 --ancestor <output> printed
  PASS_COMPLETE_PR799_REPLAY_COVERAGE. It records the ancestor as reused
  (reused_ancestry: true), as for any externally produced ancestor. Nothing here
  changes what make frontier1042-verify (or frontier942-verify) runs.
• End to end, cold frontier1042 on the rebased PR tree (WSL, same checkout, one run
  each, one after the other): the original verify_frontier1042.py took 1560 s
  (baseline799-cold 788 s, its own stages 772 s). The new path (concurrent driver 467 s,
  then verify_frontier799.py --ancestor 189 s, then
  verify_frontier1042.py --baseline799 756 s, with the ancestor moved to
  <baseline799>/ancestor as that script expects) took 1412 s, 148 s or 9.5% less.
  Both ended in PASS_COMPLETE_PR1042_COMPOSITION with kappa
  0.001672188119205421; the new path records reused_completed_baseline: true. About
  132 s of the saving is the 741 part (788 s to 656 s); the rest is run-to-run noise in
  1042's own stages (chain 479 s vs 464 s). Single runs, so about +/-2% should be
  assumed. The Conditional κ = 0.001661960547713947: T37 whole-owner packing and all-T bound #942 replay was not run with it.
• Overlap check at submission: Tooling: 1.54-1.71x faster #741 complex replay by running independent pinned stages concurrently #853 (merged) schedules stages inside complex741 only and
  does not overlap the bit supplier; no open or merged PR found by search touches
  verify_frontier741.py or overlaps bit589 with complex741. The two are
  complementary: with --complex-jobs 3 the complex chain hides bit589 entirely (table
  above).
• --complex-jobs 3 was run cold once (293 s). Its output has the same 385 files as the
  sequential tree and passes verify_replay_coverage.py --frontier 741. Against the
  sequential tree it differs in the 101 noise files plus complex741.log, whose 45
  lines differ only in the order of RUN/PASS progress messages; every differing JSON
  file is identical after masking seconds and SHA-256 values.
• The u8-native stage (about 254 s) and the cold Conditional κ = 0.001661634653669567: four paired read exchanges on PR741's frame-coalesced word #799 native stage (about 393 s)
  dominate the remaining serial time and were not examined.
• Peak memory with both suppliers running was not measured; the 7 GB benchmark machine
  completed it.
• Linux CI was not run; the benchmark used the environment above.

Credit and disclosure

Builds on the pinned verify_frontier741.py driver, verify_frontier741_bindings.py
(Douglas Colkitt, with OpenAI Codex assistance), the
dependency-scheduled complex replay of #853 (Aaron / navinor), and the --jobs pattern
of verify_frontier596.py. Track: verification engineering, assignment seed 90, base
commit above. Substantial AI assistance (Claude, in Cursor) in analysis, code, tests
and this note. Contributor: James Reed (@rcfc-jreed).

[rcfc-jreed]

Linux evidence for this PR (addresses the "Linux CI not run" open item in the note).

Run: https://github.com/rcfc-jreed/integer-mult-bounds/actions/runs/38447255855 on fork branch ci-741-concurrent (this PR's head 4329398a plus one workflow-only commit, 8c04742e, which is not part of this PR). The upstream workflows for this PR are waiting for maintainer approval (action_required), so this is the fork run instead.

Setup follows frontier741.yml: ubuntu-24.04, Python 3.12, pip install numpy sympy scipy, apt-get install g++ libboost-dev, repo-pinned action SHAs. Each replay job also ran verify_replay_coverage.py --frontier 741, asserted PASS_REBUILT_PR589_PR741_COMPOSITION with kappa 1661202643272239/10^18, and required git diff --exit-code. The unit-tests job (record check plus the 36 related tests) passed.

Job (own 4-vCPU VM)	Wall seconds	bit589	complex741	vs original
original verify_frontier741.py	1245	279	966
verify_frontier741_concurrent.py	770	222	770	-38%
same, --complex-jobs 3	585	398	585	-53%

All three passed with the same exact result. Caveats: one run each, and each job ran on a different hosted VM, so absolute times are noisy (bit589 alone took 222 to 398 s across the three). Treat the ratios as indicative. They are consistent in direction with the WSL measurements in the note (-17% and -50%). Not run on Linux: the downstream #799 and #1042 replays with this driver's output.

[rcfc-jreed]

Downstream check added: the concurrent driver's output was used as --ancestor\ for \�erify_frontier799.py\ (WSL, default concurrent mode). Result: \PASS_REBUILT_PR589_PR799_COMPOSITION, kappa 1661634653669567/10^18, in 203 s, and \�erify_replay_coverage.py --frontier 799 --ancestor ...\ printed \PASS_COMPLETE_PR799_REPLAY_COVERAGE. It is recorded as reused ancestry (
eused_ancestry: true). The #942/#1042 replays were not run with it. Note updated accordingly.

[rcfc-jreed]

End-to-end result added (WSL, rebased PR tree, same checkout, one run each, back to back). Cold \�erify_frontier1042.py: 1560 s. New path (concurrent 741 driver 467 s -> \�erify_frontier799.py --ancestor\ 189 s -> \�erify_frontier1042.py --baseline799\ 756 s): 1412 s, i.e. 148 s (9.5%) less. Both ended in \PASS_COMPLETE_PR1042_COMPOSITION, kappa 0.001672188119205421. About 132 s of the saving is the #741 part; the rest is run-to-run noise in 1042's own stages. Single runs, so treat as roughly +/-2%. Details and caveats are in the updated note.

[Mister-Meeseeks]

100-PR batch review at 4d45bef9bd0b52cc4989f195cd2dd4597f4a6d08.

Merged to main, preserving the original contributor commits and package attribution.

Reviewed scope: Opt-in concurrent PR589/741 replay driver; focused tests check exact supplier commands, environment scrubbing, completion ordering and failure cancellation. No independent full-chain timing claim.

The pinned scoped command passed; the batch ledger contains its command, input hashes, environment and log. This incorporates the research/tooling contribution and does not change the selected κ or discharge the inherited multiplication-theorem assumptions.

Batch review, attribution and evidence. Thank you for the contribution.