Repository navigation
build(bench): provide the disposable Fly ladder adapter #958
Description
Activity
- changed the title
[-]build(bench): reduce Fly execution to a disposable provisioning adapter[/-][+]build(bench): provide the disposable Fly ladder adapter[/+]on Aug 29, 2026 Live tiny qualification attempt — provider build admission failure, full teardown
Merged executor:
a9014b4378ef4240c907d6f8f8b9e7f50bf4509c(PR #999)One authorized tiny qualification attempt ran from a clean detached checkout:
- Admission passed for fixed region
dfw, smallest current performance presetperformance-1x(1 vCPU / 2048 MiB), and a 10 GiB disposable volume plan. - Execution ran from
2026-08-29T22:03:11Zto2026-08-29T22:03:33Zand failed with typed sanitized statusbuild_failedduring Fly remote image build. - Failure occurred before volume or Machine creation. GraphForge filesystem, disk I/O, phase RSS, and the smoke lifecycle were not exercised. This is not an S18 result.
- No retry was attempted.
Provider evidence is consistent with a Fly organization billing/remote-build entitlement boundary: live organization state reported
EnablePaidHobby: falseandRemoteBuilderApp: null. Fly's current official billing documentation says organizations require a payment method for most deployment activity: https://fly.io/docs/about/billing/#payment-options . Fly's current build documentation confirms remote build is the default supported path and local-only requires a healthy local Docker daemon: https://fly.io/docs/flyctl/integrating/#build-modes . Local Docker was independently unhealthy during diagnosis, and local build would also violate this benchmark controller's remote-only/host-disk policy.Teardown was independently verified:
- global apps baseline:
[] - global apps final:
[] - scoped Machine / volume / secret inventories: app not found
- image attachment: absent with destroyed app
- cleared ledger: no app ownership, resource IDs, digest, secrets, or token material
- Machine and volume cost: zero; only a short-lived empty app existed
Sanitized local artifacts:
result.jsonSHA-256338e143c7c9496bc51f76ba8baf91089fd1d9aa5fc3de37b253e53576de6650d, schemagraphforge-fly-adapter-result/1, failurebuild_failed- cleared
ledger.jsonSHA-25672e21f32b0f29321889d404cfbbe9e640ab69974b39e7e3d480b2b0fe43c45d1 - no qualification evidence JSON exists because the image did not build
Close gate remains unchanged: enable/confirm Fly billing and remote-build capability, then rerun the same merged executor once without changing machine size or test scope; require qualified evidence plus complete teardown.
- Admission passed for fixed region
Correction and second live attempt — app-readiness race isolated
The prior comment inferred a billing/remote-build entitlement boundary from organization metadata. The user confirmed Fly spending is enabled; that inference is withdrawn. Do not use
EnablePaidHobbyas billing authority.A second explicitly authorized attempt ran unchanged from clean detached merged SHA
a9014b4378ef4240c907d6f8f8b9e7f50bf4509c:- auth:
ds.liberty@gmail.com; global app baseline[] - fixed
dfw; smallest currentperformance-1x(1 vCPU / 2048 MiB); planned 10 GiB - execution:
2026-08-30T04:27:17Zto04:27:38Z - terminal result: typed sanitized
build_failed - failure again preceded image resolution, volume creation, Machine creation, smoke execution, and RSS collection
- no S18 or resizing occurred; no automatic retry followed
Independent teardown again proved global apps
[], app-scoped Machine/volume/secret inventories all app-not-found, and a cleared ledger with no digest, IDs, secrets, or token material. Result SHA-256:338e143c7c9496bc51f76ba8baf91089fd1d9aa5fc3de37b253e53576de6650d. Cleared-ledger SHA-256:72e21f32b0f29321889d404cfbbe9e640ab69974b39e7e3d480b2b0fe43c45d1. No evidence JSON exists because no Machine ran.Fly deploy begins by verifying the app through Machines inventory. The executor created the app, persisted ownership, and immediately invoked deploy without requiring propagation to that authority. The repeated narrow failure validates the missing readiness gate. PR #1000 implements the root repair: bounded deadline/backoff polling through
flyctl machine listbefore the single build, typedreadiness_timeout, immediate failure for malformed/nonempty inventory, no build retry, and preserved finally teardown.No further Fly attempt should run until #1000 is merged and exact-head gated.
- auth:
Post-#1000 live attempt — readiness fixed; build configuration remains blocked
Merged repair:
f1606c2b679253012861e0c8fda75d7b69135e04(PR #1000).One authorized post-fix tiny attempt ran unchanged from a clean detached checkout:
- auth
ds.liberty@gmail.com; global app baseline[] - fixed admitted
dfw; smallest currentperformance-1x(1 vCPU / 2048 MiB); planned 10 GiB - execution
2026-08-30T05:01:21Zto05:01:47Z - readiness polling converged, proving fix(bench): await Fly app readiness before remote build #1000 fixed the app-propagation boundary
- the subsequent single remote build still returned typed sanitized
build_failed - failure preceded image digest resolution, volume/Machine creation, smoke execution, filesystem/RSS evidence, or S18
- no retry or scaling followed
Independent teardown again proved global apps
[], scoped Machine/volume/secret inventories app-not-found, and a cleared ledger with no digest, IDs, secrets, or token material. Result SHA-256:338e143c7c9496bc51f76ba8baf91089fd1d9aa5fc3de37b253e53576de6650d. Cleared ledger SHA-256:72e21f32b0f29321889d404cfbbe9e640ab69974b39e7e3d480b2b0fe43c45d1. No evidence JSON/RSS exists. No Machine or volume cost was incurred.The remaining root-cause candidate is now narrower and independent of readiness: the repository has no
fly.toml, and the executor callsflyctl deploy --build-onlywithout--config. Fly documents thatfly deployapplies configuration from localfly.toml, looks for that file in the working directory, and supports an explicit--configpath: https://fly.io/docs/launch/deploy/ and https://fly.io/docs/launch/monorepo/ . The executor intentionally sanitizes provider stderr, so the exact CLI text was not retained and this remains a strongly supported diagnosis rather than a claimed captured error.Root repair should provide a checked-in minimal build-only Fly config with no services and pass it explicitly, while retaining the existing remote-only build, same-app immutable digest, no-public-service contract, typed failure, and finally teardown. No further Fly attempt should run before that repair is reviewed and merged.
- auth
PR #1001 implements the validated explicit-build-config repair from the post-#1000 evidence. It adds an identity-free, service-free checked-in Fly build config, passes it and the Dockerfile by absolute path, preserves dynamic
--app,--remote-only,--build-only,--push,--no-public-ips, commit build args, immutable digest resolution, and finally teardown. Offline tests prove config contents/absence of public surface, working-directory independence, and that build failure precedes volume/Machine creation. No Fly run occurred. Issue 958 remains open pending merge and one post-merge tiny qualification with independent teardown.Post-#1001 live attempt — explicit config valid; diagnostic contract is now the blocker
Merged explicit-config repair:
95887ce937bd8229cc4103558605d56f30ab1738(PR #1001).One authorized configured tiny attempt ran unchanged from a clean detached checkout:
- auth
ds.liberty@gmail.com; global app baseline[] - fixed admitted
dfw; smallest currentperformance-1x(1 vCPU / 2048 MiB); planned 10 GiB - checked-in config parsed to only
[build].dockerfile - execution
2026-08-30T05:22:39Zto05:23:05Z - readiness converged and the explicit config was supplied
- the single remote build still returned typed sanitized
build_failedbefore image digest resolution - no volume, Machine, smoke, filesystem/RSS evidence, S18, retry, or scaling occurred
Teardown independently proved global apps
[], scoped Machine/volume/secret inventories app-not-found, and a cleared ledger with no digest, IDs, secrets, or token material. Result SHA-256:338e143c7c9496bc51f76ba8baf91089fd1d9aa5fc3de37b253e53576de6650d. Cleared-ledger SHA-256:72e21f32b0f29321889d404cfbbe9e640ab69974b39e7e3d480b2b0fe43c45d1. No evidence JSON/RSS exists and no Machine/volume cost was incurred.After teardown,
flyctl config validate --strict --app ... --config containers/fly-filesystem-qualification/fly.build.tomlpassed. Missing/invalid Fly config is therefore not the terminal cause.The close blocker is now the harness diagnostic contract:
FlyctlTransport.runcaptures provider stdout/stderr, butexecutecatchesCalledProcessErrorand emits onlybuild_failed, discarding the bounded provider text. Distinct build causes are therefore indistinguishable after mandatory teardown. Another run would be goal-seeking until the executor preserves a bounded, rigorously redacted local diagnostic without credentials, provider resource IDs, absolute paths, or unbounded logs.One static build-drift candidate also needs deterministic checking: the qualification Dockerfile uses
rust:1.90-bookworm, whilerust-toolchain.tomlpins1.96.0. This is not claimed as the live cause without captured build output.Next root repair: add sanitized bounded provider diagnostic capture and Dockerfile/toolchain parity coverage, merge, then run exactly once.
- auth
Diagnostic root repair is now in PR #1002 at exact head
df4c596a. The executor now maps bounded provider build output to a closed sanitized cause code and never persists raw output; unknown or sensitive diagnostics collapse toprovider_build_unknown. Offline tests cover billing, remote-builder availability, invalid config, Docker/build-step failure, timeout, and unknown sensitive text with token/resource/path/URL leakage checks. The checked-in Fly smoke image was also verified stale at Rust 1.90 versus repository authority 1.96.0 and is aligned in this PR with a static parity regression.Local gates: Ruff green; 31 focused tests green; all 85 benchmark Python tests plus smoke green; Fly filesystem safety-contract checks green; diff check green. No Fly resources were created. Issue 958 remains open pending merge and one subsequent live tiny qualification with teardown evidence.
Local-machine Fly retry at merged commit
0704900657c6836f832fa6614bd7c2672bfd010e:- no-spend admission passed:
performance-1x, 10 GiB, fixeddfw, andfull_run_authorized=false - the one authorized tiny execution failed during the remote image build with the closed sanitized result
build_failed / provider_build_unknown - no Machine ran, no volume was created, no GraphForge smoke or S18 workload executed, and no qualification evidence was produced
- teardown proof:
flyctl apps list --jsonreturned[]; the ownership ledger is cleared (app_owned=false,machine_id=null,volume_id=null,secret_names=[],token_material_present=false)
This is still a provider-build-path failure, not evidence of a GraphForge filesystem, ingest, or RSS failure. Do not retry unchanged or authorize S18 from this result; the next repair must make the unknown provider failure deterministically diagnosable without persisting raw provider output.
- no-spend admission passed:
- modified the milestones: M14: Benchmark Harness and GDC Suites, M11: Benchmark Harness and GDC Suites for v0.6.0
on Sep 4, 2026
Objective
Provide a minimal disposable Fly adapter for the completed qualification controller. The adapter provisions infrastructure and retrieves evidence; it never owns benchmark mechanics, thresholds, rung selection, or success policy.
Requirements
Acceptance criteria
BDD completion scenario
Given a completed controller, an explicitly authorized maximum scale, credentials, and a pinned image digest
When a tiny qualification succeeds or fails
Then available sanitized evidence is retrieved
And every temporary Fly resource is destroyed and independently verified absent.
Non-goals
Running the real S18-S26 ladder as this issue's close criterion, owning thresholds or projection policy, selecting larger hardware to hide amplification, nested Docker on Fly, or persistent provider infrastructure.
Relationships