Gate v0.1.0 on real ultrafuzz-bench threat-model and dynamic-goal runs

Current release policy and status (2026-08-09). The implementation described here shipped to the intended long-lived release/v0.1.0 branch in PR #351 (942e629fd6ad8ac79abb80be919b2af169e1c2b8). The maintainer explicitly deferred exercising the paid full-cohort lane; it is not a prerequisite for the manual release gate. Routine gpt-5.6-luna smoke evidence is the acceptance signal for accumulated branch states. PR #366 remains a separate manual release/v0.1.0 → main decision and is outside this issue. The original full-cohort design below is retained as historical capability documentation, not unfinished release-blocking work.

Summary

Define and execute the real-protocol validation gate for Ultrafuzz v0.1.0 after dynamic nodes (#179), the vulnerability database (#180 (pinned external database integration)), and threat-model /goal discovery (#182) are integrated.

All model-backed validation must use the existing CI ultrafuzz-bench cohort. Do not introduce Uniswap v2 or any additional ad hoc protocol for this work.

Canonical target cohort

Reuse exactly the three immutable targets established by #83 and current CI:

ID	Framework	Repository	Revision
`very-liquid-vaults-foundry`	Foundry	`https://github.com/rheo-xyz/very-liquid-vaults`	`e50384709a696c86ab0440bbbc3dd14a5f4ff6ec`
`venus-isolated-pools-hardhat`	Hardhat	`https://github.com/code-423n4/2023-05-venus`	`9853f6f4fe906b635e214b22de9f627c6a17ba5b`
`stableswap-ng-vyper`	Vyper	`https://github.com/curvefi/stableswap-ng`	`8c78731ed43c22e6bcdcb5d39b0a7d02f8cb0386`

Do not replace, supplement, or silently fall back from this cohort. Preserve the existing pinned toolchain and source-integrity requirements.

Validation layers

1. Deterministic implementation gates

The feature issues own their focused tests, but this gate must confirm the integrated release passes:

format, lint, typecheck, build, and package/release validation;
topology graph and contract validation;
vulnerability database whole-corpus validation;
dynamic expansion and provenance integration tests;
report-bundle and schema compatibility tests.

Do not add a fake-agent runner or synthetic model-success fallback.

2. Quick real-agent protocol checks

Use gpt-5.6-luna with reasoning high through the normal Codex integration, consistent with the existing real smoke workflow.

Run the production v0.1.0 topology against all three CI targets and retain the actual:

THREAT_MODEL.md;
threat-model.json;
goal-plan.json;
dynamic node manifest/status;
selected vulnerability-class records and database digest;
findings, report.json, and report.md.

These are real model outputs. Do not fabricate a threat model, planner output, finding, or success path.

3. Threat-model snapshot examples

Use the three real targets as the source of checked/retained snapshot examples. Since LLM prose is nondeterministic, avoid a brittle byte-for-byte assertion over the entire Markdown document.

Snapshot and assert stable structural facts instead:

schema version and required sections/fields;
well-formed stable threat/capability IDs;
repository evidence references;
target-relevant actors, assets, capabilities, and threat categories;
capability status uses present, absent, or unknown correctly;
goal-plan cardinality follows threats + applicable classes, plus the fixed roaming node;
generated node IDs match the plan;
MDX placeholders resolve to the intended threat/class context;
every produced finding retains one or more real source node IDs.

Retain the complete generated Markdown/JSON as human-reviewable benchmark artifacts even when automated assertions are structural.

Target-specific expectations should be derived from the pinned repository and public benchmark ground truth, not injected into agent prompts. Venus should exercise lending/liquidation relevance; the other two targets provide the existing Foundry/vault and Vyper/StableSwap coverage without adding a fourth repository.

Dynamic fanout checks

For every run record:

number of structured threats;
number of applicable database classes;
expected and actual dynamic child counts;
generated/ready/active/terminal/failed/skipped counts;
requested and effective concurrency;
max_dynamic_nodes setting;
retries, timeouts, and failed goal lanes;
source node IDs for validated findings;
token usage, cost, and wall-clock time.

The gate must demonstrate that dynamic children are independently visible and that a large ready queue respects existing concurrency rather than collapsing into one opaque agent node.

Report checks

THREAT_MODEL.md and threat-model.json contain detailed threat analysis.
report.md changes only as needed for concise artifact links/coverage and finding provenance.
Do not duplicate the full threat model into report.md.
Dedupe preserves all corroborating dynamic source node IDs.
report.json, report.md, and normalized findings agree on finding counts.

Release comparison

Evaluate v0.1.0 as a release-level change rather than adding baseline-versus-treatment tests to the component PRs.

Compare the v0.1.0 release candidate with v0.0.12 (or the latest stable predecessor if it changes) using the same pinned cohort, model profile, judge/scoring identity, and comparable execution policy. Record any intentional policy difference, especially the additional dynamic work, so cost and quality comparisons remain interpretable.

This issue complements existing longitudinal benchmark/model-matrix work such as #138; it does not replace unrelated model coverage.

Failure policy

No fake/offline substitution after a model, target-build, workflow, or artifact failure.
Keep independent target execution (fail-fast: false).
Upload available diagnostics/artifacts under if: always().
A missing threat model, invalid plan, incomplete dynamic expansion, missing provenance, or report inconsistency fails the relevant release gate.
Paid/nondeterministic CI may remain non-required in branch protection as established by Replace synthetic target E2E with real ultrafuzz-bench runs #83, but the v0.1.0 release owner must obtain and review a complete successful cohort before release.

Acceptance criteria

Implementation handoff

Suggested branch/worktree: test/v0.1.0-ultrafuzz-bench, based on the v0.1.0 integration stack once #179, #180 (pinned external database integration), and #182 are available.

Keep benchmark manifests and target revisions unchanged unless a separate approved issue changes the canonical cohort. Store run IDs, candidate commit, database digest, model/reasoning profile, topology digest, and scoring identity with every result.

Tracked by the v0.1.0 meta plan in #185. Treat the contracts and dependency ordering there as part of this implementation handoff.

Started — and the attempt shows this is materially bigger than a lane definition

Draft PR #351 against release/v0.1.0 carries the lane, its manifest wiring and CI plumbing. It is not a working gate and must not be dispatched. Adversarial review rejected it, and the three findings are worth recording here because they change the shape of this issue.

1. A fatal bug that would have cost money. packages/evals/src/public-diagnostics.ts:115 still read lane: z.enum(["smoke", "full"]). That parser runs after runEval() returns, so a threat-model dispatch would have burned the entire cohort's model spend (~$150–300) and then died at the post-eval checkpoint with no diagnostics, no bundle and no gate evidence. Fixed in #351. Ten new lane tests missed it because they stop at the manifest layer.

2. §2 and §3 are not reachable with today's plumbing. Retaining THREAT_MODEL.md, threat-model.json, goal-plan.json and the vulnerability-db manifest cannot be done through reporting.artifacts.include: it feeds uploadsForManifest → this.input.reporters, and the public worker runs eval run --provider none, for which createEvalReporters returns []. Zero reporters, zero uploads. publicBundleSources hardcodes a fixed per-row set, MODAL_COLLECT_RESULT_FILES excludes them, and the run root is rm -rf'd with the sandbox — so the real model outputs are destroyed. The work belongs in publicBundleSources, and it is a prerequisite, not a detail.

3. The dynamic-fanout observability requirement has no substrate. The twelve per-record metrics this issue asks for — expected vs actual child counts, generated/ready/active/terminal/failed/skipped, requested vs effective concurrency, max_dynamic_nodes, source node IDs, tokens/cost/wall-clock — have nowhere to live: EvalRunRecord and EvalWorkflowLifecycle carry no node-level or expansion data. Without it the gate cannot demonstrate that dynamic children are independently visible, which is its stated purpose.

Also: the 15,000s budget is not headroom. PUBLIC_BENCHMARK_SMOKE_MAX_RUNTIME_SECONDS is already 15,000, so this gate would get exactly the smoke lane's wall clock for a graph sized at 55 static agentic nodes plus dynamic expansion.

Suggested sequencing: land the publicBundleSources retention work and the run-record expansion telemetry as two separate changes first; the lane is the last piece, not the first.

Still needed and not suppliable from here: an owner and a compute budget with a number attached.

Tracked in #327.

feat(modal): retain threat-model and goal-plan run artifacts in the p…

feat(evals): make node-level expansion and concurrency observable wit…

Both blocking prerequisites have landed

feat(modal): retain threat-model and goal-plan run artifacts in the public bundle #363 (B1) — threat-model artifacts are retained through publicBundleSources at reports/<row>/artifacts/<node>/<name>, node-namespaced, optional, symlinks refused, >32 per row fails loudly. §2 and §3 are now reachable.
feat(evals): make node-level expansion and concurrency observable without a reporter #365 (B2) — evalRunExpansion derives node counts by status, the static/dynamic split, per-child lineage and requested-vs-effective concurrency from durable state alone, riding on EvalRowScore into scores.jsonl/summary.json. No reporter involved.

Both are on main and now on release/v0.1.0 (8f367b9). PR #351 is rebased onto them, wired, and green across build/test/typecheck/format/lint.

Two defects were caught landing them, both fixed: #365 declared EvalRowScore.expansion required, but it derives from an optional field and scores.jsonl is persisted — that made every historical score unreadable and failed typecheck. And #363 introduced PUBLIC_OPTIONAL_ROW_ARTIFACTS as a literal duplicating BENCHMARK_THREAT_MODEL_RETAINED_ARTIFACTS in another package; it now derives from it.

What this issue still needs, and none of it is code

Decide where the six threat-model-dependent dynamic-fanout metrics in #183 are derived from #364 — six of the twelve required metrics depend on where "expected" child counts come from. Three options, recommendation stated, decision outstanding. Until then the lane reports what a run did but cannot fully assert what it should have done.
A dispatch. No run has been made. The smoke lane has not yet produced scoring_ready: true for any target (dedupe-findings fails its artifact contract on every smoke-lane target #348 is open for the two Foundry-only node failures), so dispatching this gate before that would spend the budget to learn something the cheaper lane can tell us.
An owner and a compute budget with a number on it.
A wall-clock review. PUBLIC_BENCHMARK_SMOKE_MAX_RUNTIME_SECONDS is already 15,000, so this gate would get exactly the smoke lane's budget for 55 static agentic nodes plus dynamic expansion.

#351 stays a draft until at least 1 and 2 are settled.

Scope change from the maintainer: v0.1.0 will not be run in full.

More work lands in v0.1.0 after the current milestone issues close, so release/v0.1.0 is a long-lived accumulation branch, not a release candidate that gets frozen. PR #366 (release/v0.1.0 → main) is the real gate, and it is manual — held on the repository being completely ready to open source.

Consequence for this issue: the full-cohort real-agent gate is deferred. It is no longer a precondition for the v0.1.0 merge, and #351 stays draft. The acceptance path for now is the ultrafuzz-bench smoke lane on gpt-5.6-luna.

The capability work still lands — #363, #365 and the #364 implementation give the gate real artifact retention and real fanout observability, so it exists and is measurable whenever you choose to run it. What is deferred is spending a full cohort's budget on it, not building it.

This also removes the owner-and-budget question as a blocker on v0.1.0: it becomes a question about when to exercise a capability that already works, rather than about a gate the release waits on.

Two findings: the runtime ceiling is correct as-is, and the smoke lane does now score

1. `max_runtime_seconds` — leave `.max(15_000).default(3_600)` alone

I ran three independent analyses of packages/modal/src/config.ts:152, each with a different emphasis (cost/runaway protection; whether work is lost or resumed at the bound; what the lane empirically needs). All three returned the same number, at high confidence: no change. There is no dissent to record.

Nothing comes close to the bound. Measured, not estimated:

source	n	p50	p90	max
`benchmarks/history.json`, `gpt-5.6-luna` `wall_clock_seconds`	90	561 s	741 s	1,098 s
three most recent smoke runs, `result.json` `runtime_ms`	3	—	—	1,322 s

The worst row ever recorded is 7.3% of the 15,000 s bound — a 13.7x margin. No artifact in any run I read contains a row watch timeout. The one 5,174 s outlier in history is manual-kimi-smoke-…-nt7200-…, a manual experiment with node_timeout_seconds raised to 7,200; under CI's PUBLIC_NODE_TIMEOUT_SECONDS = 1800 nothing exceeds 1,862 s.

The bound is pinned from both sides, and 360 minutes is why it is 15,000. The collect job's timeout-minutes: 360 (21,600 s) is not a separate looser bound that was overlooked — it is the binding one. scripts/ci/prepare-modal-benchmarks.mjs:81-88 derives control_timeout_seconds = max_runtime_seconds + 4,800 for the one-wave smoke lane = 19,800 s, which packages/modal/test/ci-config.test.ts:90 pins exactly. That internal deadline sits 1,800 s below GitHub's job kill, and the ordering is load-bearing: the internal deadline fails the step and lets the if: always() artifact uploads run, preserving status.json / result.json / public-eval-diagnostics.json. A GitHub job timeout is a cancellation and takes the diagnostics with it. The largest value that preserves that ordering is roughly 16,200-16,700 s, and 360 minutes cannot itself be raised — it is GitHub's hard cap for a hosted job.

Lowering it would be actively harmful. PUBLIC_BENCHMARK_SMOKE_MAX_RUNTIME_SECONDS = four sequential smoke stages x two 1,800 s attempts + 600 s grace. Below ~14,400 s the watchdog starts firing on rows the topology would have terminated itself, and workflow_terminal: false sets scoring_ready: false for the whole pair.

Work is lost at the bound, which is the argument for not lowering it, not for raising it. PUBLIC_WORKSPACE_ROOT is /tmp/ultrafuzz-public-workspace (public-worker.ts:60), and every generation rm -rfs it (public-worker.ts:215). The only durable resume point is a fully completed bundle. Worth knowing: resume-required does not resume — postModelRecovery is "stop" for all public configs, and run 31220479878's recovery_summary records resumptions: 0, post-model-resume: 0. The name reads as recoverable; the state is terminal and lossy.

So the ceiling is not truncating anything, is not the reason the lane fails, and is already ~25x looser than typical spend ($11 mean, $33 max per luna row, three rows per pair). No diff to the number. The only change I made is #372, a comment recording that derivation at the edit site, because all three analyses had to trace three files to reconstruct it.

The analyses did surface a real spend problem — just not this one. Filed as #371: cleanup_incomplete_run is gated on !cancelled() while cancel-in-progress: true, so a concurrency-cancelled run never terminates its sandboxes. 17 of the last 45 runs were cancelled; 5 ran long enough to have live sandboxes.

2. The smoke lane does now complete and score

scoring_ready: true on 3/3 targets, twice, without the #333 soft-fail hatch:

run	branch	commit	rows scoring	hatch taken?
31220226212	`main`	`5db8c6f`	3/3	no
31220479878	`release/v0.1.0`	`8f367b9`	2/3	yes
31220736791	`test/v0.1.0-ultrafuzz-bench`	`426ff5f`	3/3	no

I confirmed the hatch attribution from the run logs: only 31220479878 emits the ##[warning]… ended as resume-required annotation. On the other two the soft-fail lines appear solely as rendered shell source, never as an emitted annotation.

The dedupe-findings / externalized-state-accounting / lifecycle-view-boundaries artifact-contract failures from #348 do not appear in any of the three. Detail posted there.

The one failure was not the ceiling and not a contract. release/v0.1.0 @ 8f367b9 failed on final-report with failure_category: "agent-failure" on 2 of 3 targets; venus-isolated-pools-hardhat had terminal_report_present: false, reason terminal-report-missing. runtime_ms was 1,322 s — 8.8% of the ceiling. It did not recur in either adjacent run.

What is still outstanding for this gate

release/v0.1.0 has not yet had a hatch-free green of its own. Note that the main green does not transfer: release/v0.1.0 carries 128 files / ~21,600 lines that main does not. test/v0.1.0-ultrafuzz-bench is the closest evidence — it contains all of release/v0.1.0 plus #351's lane wiring — but it is not identical either.

I am pushing the pending main sync to release/v0.1.0, which triggers exactly one smoke run. Result will be posted to #366.

Related: #367 (scoring_ready was vacuously true for a zero-row document, so the acceptance signal itself was untrustworthy), #369 (the hatch now reports whether the pair was scoreable), #370 (whether the hatch should block on it — a maintainer decision). Cross-ref #327, #319, #165.

fix(ci): say whether a soft-failed smoke pair was scoreable (#369)

feat(evals): v0.1.0 threat-model release-gate lane (#351)

Closing as implemented on the intended release branch, consistent with the release-component policy used for #180.

PR #351 merged into release/v0.1.0 as 942e629fd6ad8ac79abb80be919b2af169e1c2b8, and that merge remains an ancestor of the current release head. It ships the opt-in three-target threat-model lane and its checksummed structural gate: exact planner-owned child cardinality, successful dynamic children, independently attributable goal lanes, retained threat-model/goal-plan evidence, and non-serial effective concurrency. The fixture repair, inert reporter predicate removal, and unified dispatch design requested by #413 item 9 are included.

The remaining full paid-cohort execution is deliberately not being represented as completed. The maintainer changed that policy: exercising the full cohort is deferred, routine Luna smoke is the acceptance signal for accumulated branch states, and the manual PR #366 is the eventual open-source readiness gate. That deferred spend is no longer unfinished implementation in this issue, and closing this issue does not merge release/v0.1.0 into main, merge #366, cut a tag, or publish a release.

Release/v0.1.0 (#744)

Navigation Menu

Repository navigation

Gate v0.1.0 on real ultrafuzz-bench threat-model and dynamic-goal runs #183

Summary

Canonical target cohort

Validation layers

1. Deterministic implementation gates

2. Quick real-agent protocol checks

3. Threat-model snapshot examples

Dynamic fanout checks

Report checks

Release comparison

Failure policy

Acceptance criteria

Implementation handoff

Started — and the attempt shows this is materially bigger than a lane definition

Both blocking prerequisites have landed

What this issue still needs, and none of it is code

Two findings: the runtime ceiling is correct as-is, and the smoke lane does now score

1. `max_runtime_seconds` — leave `.max(15_000).default(3_600)` alone

2. The smoke lane does now complete and score

What is still outstanding for this gate

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gate v0.1.0 on real ultrafuzz-bench threat-model and dynamic-goal runs #183

Description

Summary

Canonical target cohort

Validation layers

1. Deterministic implementation gates

2. Quick real-agent protocol checks

3. Threat-model snapshot examples

Dynamic fanout checks

Report checks

Release comparison

Failure policy

Acceptance criteria

Implementation handoff

Activity

aviggiano commented on Aug 4, 2026

aviggiano commented on Aug 7, 2026

Started — and the attempt shows this is materially bigger than a lane definition

aviggiano commented on Aug 7, 2026

Both blocking prerequisites have landed

What this issue still needs, and none of it is code

aviggiano commented on Aug 7, 2026

aviggiano commented on Aug 7, 2026

Two findings: the runtime ceiling is correct as-is, and the smoke lane does now score

1. max_runtime_seconds — leave .max(15_000).default(3_600) alone

2. The smoke lane does now complete and score

What is still outstanding for this gate

aviggiano commented on Aug 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `max_runtime_seconds` — leave `.max(15_000).default(3_600)` alone