Repository navigation
Conversation
aviggiano
commented
Aug 7, 2026
aviggiano
commented
Collaborator
- Do not merge until we have evals showing this is much better!
* feat: add threat model and dynamic goal fanout * feat: add dynamic runtime topology nodes * merge: preserve threat provenance with dynamic runtime nodes * fix: retain packaged prompt assets with dynamic variables * feat(references): integrate external vulnerability database * fix(references): harden vulnerability database sync * feat(runtime): consume pinned vulnerability database reference * fix(topology): bind vulnerability reference semantics * fix(runtime): export Smithers runtime helpers * test(runtime): fixture pinned vulnerability database reference * style: format changed workflow files * test: align topology anchors and lint unused fixtures * test(cli): fixture pinned vulnerability reference for validation * fix(report): allow safe audit-context links * docs: clarify vulnerability database materialization * fix(cloud): harden cloud selected-task handoff boundary Consolidates the reviewed PR #188 remediation work into an intentional commit: the shared cloud selected-task handoff contract, the Modal provider and worker boundary validation, native-v1 vulnerability database materialization, and the threat-model/goal-plan artifact schemas with their tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(cloud): correlate and fully bind the cloud selected-task boundary Closes the five audited cloud-boundary gaps and three prior-review regression gaps: - reject a non-empty outer `tasks` array for a `cloud_worker` dispatch, so an outer entry can neither replace the relocated prompt body nor name a second attempt - reconstruct a runtime-generated dynamic attempt's whole canonical DTO from its group's compiled template, binding the run root and every derived path, the rendered prompt location, the agent and model profile, the execution resources, the artifact output contracts, the pinned planner catalog, the reference trees, and its dynamic provenance; only the expansion key and the two runtime digests remain runtime data, and the generating group is derived from the dispatched attempt ID rather than taken from the handoff - remove the production no-`selected_task` compatibility path and cover its absence for compiled, deferred, and generated attempts - persist a canonical logical-dispatch fingerprint (generation-independent) and check it on durable resume, in checkpoint records and indexes, in published results, and in live sandbox reattachment, keeping the exact same-generation resume check distinct from the allowed reset-generation restore - replace the three independent dependency-array supersets with one correlated runtime-materialization check: each added artifact directory belongs to one added attempt, each verifier derives from one, and each added generated node must be derivable from a declared group Regression gaps: `dynamic-workflow.test.ts` now uses a real non-empty prompt exclusion and asserts transformed bytes for both the dynamic and the deferred snapshot; topology `schema.test.ts` validates a generated graph against the committed expanded-graph JSON Schema document; the stale contract-valid-today comment in the artifacts threat-goal tests is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(preflight): ignore prompt pack test scratch directories * fix(cloud): bind durable recovery to handoff contents * docs(references): state the private-database read prerequisite exactly `monad-developers/web3-vulnerability-database` is a private repository, so a clean scaffold that pins `vulnerability-database.web3` cannot materialize it without an ambient git read credential. Record that prerequisite where the pin is documented, quoting the diagnostic the CLI actually emits. Verified against the pinned revision with no credential available: error: GIT_FAILED: git command failed: git fetch --depth=1 \ --filter=blob:none origin fbf00e990b1316879b674e9903548dba452e40d5: \ remote: Repository not found. fatal: Authentication failed for 'https://github.com/monad-developers/web3-vulnerability-database.git/' The failure surfaces as the `GIT_FAILED` reference diagnostic wrapping git stderr, not as a bare git error, and dropping only the catalog entry leaves `reference-vulnerability-database` dangling, which `expand()` rejects with `INVALID_REFERENCE_NODE`. Both are stated as observed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(references): authenticate the pinned private database read The curated vulnerability database is a private repository, but reference materialization fetched it with a plain anonymous `git fetch`. The detached Modal worker runs `ultrafuzz references sync` in its pre-model phase, so every benchmark pair failed there: a dependency-unreachable error with `model_work_started=false` and `usage=null`, after an image build and a launch had already been paid for. Make the credential an explicit, least-privilege contract instead. References stay anonymous by default and attach a token only when both `ULTRAFUZZ_REFERENCE_GITHUB_TOKEN` and `ULTRAFUZZ_REFERENCE_GITHUB_REPOS` are declared. The allowlist is what carries the security property: a token minted for one private repository is never sent to another remote just because the catalog names it, and a token without an allowlist is inert. The token reaches git only through `GIT_CONFIG_*` carrying an `http.<remote>.extraheader` keyed to the exact remote, so it never enters a remote URL, an argv entry, a config file, or a credential helper; `GIT_FAILED` diagnostics redact both the token and its derived basic-auth encoding. CI mints a short-lived token from the existing eval-history GitHub App, scoped to `contents:read` on `web3-vulnerability-database` alone, and proves access with a new preflight before the image build -- immutable repository metadata and the pinned commit only, so a missing installation or permission costs nothing and reports the exact external action required. The private key never leaves the trusted runner; only the token enters the Modal secret, where it is also a forbidden value for published bundles, diagnostics, and lifecycle logs. `launch` and `collect` each mint their own token with `skip-token-revoke: true`: installation tokens expire after an hour while `collect` can wait longer, and the sandboxes are detached, so a post-job revocation would invalidate a credential a worker still needs. The workflow keeps its push/workflow_dispatch-only trigger, so fork code never reaches the private key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(preflight): report the App installation prerequisite exactly The first CI run proved the credential contract stops before paid compute -- the image build and every sandbox launch were skipped -- but it also exposed a gap in what the failure actually says. `actions/create-github-app-token` fails first, so the preflight that owns the precise diagnosis never ran, and the only evidence was GitHub's bare `Not Found` from `GET /repos/{owner}/{repo}/installation`. That reads like an infrastructure fault rather than the one external action it represents. Let the mint step fail soft and make the preflight the step that fails the job. An absent token in CI means the App has no installation on the target repository, so the preflight now names that: install the eval-history GitHub App with `Contents: Read-only`, an administrator must do it, and the pin must not be replaced, vendored, or removed to work around it. The job still stops before the image build, so this changes only the diagnosis, not the compute that is refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(collect): keep live sandboxes collectable when the token mint fails The recovery token step in `collect` failed hard, and the wait step it precedes is gated on `available == 'true'` without `always()`. A failed mint therefore skipped waiting entirely and abandoned every already-running sandbox uncollected, even though the collection and diagnostic-upload steps that follow do run on `always()` and would then find nothing. Fail that step soft. Waiting and collection stay intact, and a relaunch that genuinely needs the credential still fails on its own pair with a recorded diagnostic instead of taking the whole matrix down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(collect): gate private-reference recovery access * fix(eval): harden private reference recovery * fix(references): use public vulnerability database * fix(runtime): avoid Smithers reserved input field * fix(runtime): normalize persisted workflow input nulls * fix(smoke): require canonical evidence locations * fix(prompts): bind goal IDs to source records * fix(prompts): bind threat-model kinds and goal-plan catalog digest to their validators The Modal smoke benchmark failed deterministically per target at the two new fanout nodes, reproduced in a live sandbox at this head: - threat-model: agents invented invariant kinds (solvency, state-consistency, configuration, availability) because neither threat-model prompt stated the ultrafuzz/threat-model@1 enum. Both prompts now spell out the five valid kinds inline. - goal-plan: the prompt said to copy all database digests from the supplied catalog, so agents set catalog_sha256 to the embedded upstream_catalog_sha256; the validator expects the SHA-256 of the planner catalog file itself. The prompt now says to compute it over the exact bytes at the supplied catalog path and warns that upstream_catalog_sha256 never validates there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): pin remaining fanout artifact fields to validator semantics A second live-sandbox round with the previous fix in place surfaced the rest of the deterministic agent/validator mismatches: - threat-model: string fields with plural names (evidence_needed, impact, security_impact, value_at_risk) were emitted as arrays. Both prompts now state the array-vs-string rule for every leaf field. - goal-plan: agents pre-wrote vulnerability-db-manifest.json, which conflicts with the canonical bytes Ultrafuzz publishes after the node; applicability checks paraphrased threat-model rationale/evidence instead of copying byte-for-byte; class-goal attack_surface_ids carried invented labels instead of threat-model attack-surface IDs. The prompt now forbids writing the manifest or vulnerability-db/ content, requires byte-for-byte capability copies, and pins attack_surface_ids to copied threat-model IDs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(runtime): stop pre-materializing the vulnerability-db snapshot manifest Task preparation backfilled every non-primary declared output with its contract's canonical empty bytes, so goal-plan nodes always started with a 0o600 '{}' placeholder at vulnerability-db-manifest.json. The post-agent snapshot publisher writes that manifest exclusively and fails closed on conflicting content, which made every goal-plan attempt die with ARTIFACT_CONFLICT once its plan validated (reproduced across all three smoke benchmark targets in a live Modal sandbox). Exclude the snapshot manifest from canonical-empty preparation exactly like the runtime-owned workspace patch outputs, and pin the exclusion with a generated-workflow test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): make goal-plan capability copies and placeholder retention mechanical The remaining goal-plan agent failures were content slips on one target: paraphrased capability rationale/evidence in applicability checks and coverage-gap goal_prompts with the {{threat-model:coverage-gap}} placeholder substituted away. Tell the agent to build checks mechanically from threat-model.json and to keep every replacement key's literal {{...}} braces in goal_prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): keep coverage-gap placeholder wording template-safe The previous commit's literal {{<key>}} example was itself parsed as a prompt template variable and failed catalog validation. Reword to describe the double-brace form; the exact {{threat-model:coverage-gap}} text is an item-scoped placeholder the validator accepts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): drop literal coverage-gap braces the topology validator rejects The topology artifact-handoff validator parses double-brace sequences in node prompts as template variables and does not recognize threat-model:coverage-gap outside dynamic-goal contexts. Describe the brace-wrapped key without writing the literal placeholder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(references): assert the public database stays anonymous under a credential The allowlist negative case named a non-existent monad-developers repository, which both carried the pre-transfer owner and asserted nothing about a reference Ultrafuzz actually ships. Use the real pinned public database instead, so the case proves the shipped vulnerability-database reference is fetched anonymously even while a private-reference token is configured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(references): re-pin the vulnerability database to the current corpus release The shipped pin still referenced fbf00e99 (18 records, 21 capabilities, aggregate c0ed7d23). Upstream has since published two further commits, and the corpus now carries 23 records and 28 capabilities under aggregate 62b7fb80, including the new classes/lending category. Re-pin to e46c0e47, the current immutable head. Its database.yml, capabilities.yml and catalog.json are byte-identical to 153e14cc; the intervening commit only changed upstream ingestion tooling, which Ultrafuzz does not consume. Verified by a fresh unauthenticated fetch with GitHub tokens, SSH agent and credential helpers disabled. Regenerate the documented digest table and catalog statistics to match, and correct the test comment that still described the repository as private. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(deps): reconcile the lockfile with Smithers 0.32.0 after rebase The branch carried two lockfile edits against Smithers 0.31.0 that conflicted with #276's 0.32.0 upgrade on main. Resolve by taking main's 0.32.0 lockfile and regenerating the workspace links from the package manifests, which restores the @ultrafuzz/prompts link into packages/artifacts. pnpm install --frozen-lockfile now succeeds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(evals): keep the smoke lane free of dynamic goal fanout The smoke lane declares disable_dynamic_strategies, but that flag only ever pruned dynamic-strategy-generator, which the dedicated smoke graph does not contain. This branch adds threat-goals and class-goals to benchmarks/smoke-benchmark.yml, and those expand one child per planned threat and per applicable class, so the lane would have run unbounded dynamic work while still reporting dynamic strategies as disabled. #277's smoke matrix requires that guarantee to hold. Name the goal-fanout node IDs, add them to the dynamic exclusion set, and apply them on the smoke lane. Production-topology exclusions still cannot be passed wholesale to the smoke graph, because its invariant, differential and dynamic-strategy nodes are absent and would raise unknown-node errors. Add a regression test that reads the smoke topology and asserts the lane prunes every dynamic node it declares, so a future dynamic node added to the smoke graph fails the suite instead of silently running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(modal): keep curated private lanes off the new threat-model nodes A private benchmark lane isolates part of the production topology by naming the nodes it wants pruned. The invariant-only Aave lane does this. Every such list was written before threat-model, goal-plan, goal-roaming, threat-goals, class-goals and reference-vulnerability-database existed, and the first four land in the pre-existing setup and goals groups, so no group-level filter catches them either. Merging this branch would therefore have made the next invariant-only run quietly execute a threat model and unbounded goal hunters, invalidating the comparison it exists to produce. Publish the default-on node IDs from @ultrafuzz/evals and prune them from any private lane that curates an exclusion list, with include_threat_model_goal_fanout to opt back in. An empty list still means "run the whole topology", so uncurated private runs are unchanged. Also prune goal-plan on the smoke lane. The fanout is its only dependent, so excluding just the fanout leaves it terminal and strands the smoke report prompt's artifact_path citation of it; the lane could not plan at all. A new runtime suite expands the real pruned graph to prove both directions. Keep the topology-derived tests honest: the published ID list is checked against the shipped topology, so a future default-on node fails the suite instead of silently widening every curated lane. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(evals): prune goal-plan wherever the goal fanout is pruned Excluding threat-goals and class-goals on their own leaves goal-plan with no dependent, which makes it terminal, and strands the report prompts' artifact_path citation of it, which requires an ancestor. Either way the graph stops validating, so any lane that drops the fanout has to drop goal-plan too. Fold that into BENCHMARK_GOAL_FANOUT_EXCLUDED_NODE_IDS and build the smoke, production-dynamic and curated-lane sets from it, so all three stay valid against the graph they are applied to. Move the pruning coverage into packages/runtime/test/topology-transform.test.ts next to the existing exclusion tests, in the node:test style that package's runner actually compiles and executes, and extend its production-topology case with the same three IDs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(evals): keep the benchmark smoke lane exactly as published Three independent reviews of 7e2687d found that pruning the goal fanout at lane level was not enough to keep the smoke lane's guarantee. Per-run agent concurrency is max_parallel_targets, which is 4, so the pruned 12-node graph still serialized into six waves: 6 x 2 x 1200s = 14,400s against a 15,000s row watchdog that packages/modal/src/config.ts caps at 15,000. The margin fell from 5,400s to 600s and could not be raised without a schema change. Keeping threat-model also kept requiresVulnerabilityDatabase true, so every smoke run gained a plan-time corpus fetch and the exact-cover capability gate in verifyThreatModelVulnerabilityDatabaseCapabilities -- at medium reasoning with two attempts. In a model matrix that gate fails a row outright, so a weaker entrant publishes no observation at all and the comparison is silently biased. Passing any non-empty excluded_node_ids also moves benchmark_execution, and with it the execution-policy and cohort fingerprints that #277's rows must share with the already-published smoke observations. Restore benchmarks/smoke-benchmark.yml byte-for-byte to main, drop the unused smoke threat-model prompt, and take the smoke lane back to an empty exclusion list. The lane keeps its guarantee structurally, the way it always did: the dedicated graph simply does not declare invariant, differential, dynamic-strategy or goal-fanout nodes. A test now asserts that rather than asserting a pruning list. Keep the prompt-only smoke improvements that carry no node dependency: the evidence-path hardening and the source_nodes provenance bullets. The full lane is untouched and still exercises the whole threat-model workstream, and the production topology keeps every default-on node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(runtime): rebuild dedupe provenance from the lifecycle ledger when syncing Three independent reviews of 0d3e5c2 found two real regressions in the new finding-provenance machinery, both introduced by this branch. dependencyFindingProvenanceForTask rebuilt the source expectations without the lifecycle ledger, so buildFindingSourceExpectations returned early and produced one single-source expectation per upstream finding. A dedupe root's source_nodes is the union of everything it merged, which matches none of those, so finalizeTerminalTask re-validated a node the authoritative in-workflow gate had already passed and marked it failed with task-output-validation-failure. That fires on any run with a genuine merge, and it is reachable from ultrafuzz inspect, ultrafuzz status, and the dashboard, so it needs no cloud execution. It also attributed upstream findings to the dependency attempt ID rather than the producing node, so a generated dynamic child could never match its own ledger record. Load the ledger and resolve producer node IDs through tasksByAttempt, mirroring the generated template, which is the authoritative implementation of the same contract. Stop creating artifact directories from this read-only path while here. The smoke prompts described a lifecycle ledger the new gate rejects: entries need node_id and finding_id and must cover every upstream finding exactly once, which only the production prompt said. Bring the smoke prompt up to the same contract and tell the smoke report to carry finding identity through, since provenance is matched on it. Also correct a comment claiming the goal fanout exists in the smoke graph, a reference example naming a node and prompt that do not exist, and the changelog description of the roaming goal, which is a static node rather than an expansion. Document the deliberate asymmetry in the private-reference credential contract and the one-off lineage discontinuity a curated private lane sees. Add a production-topology test proving the exact set every curated lane prunes still plans, and that no dynamic node, goals-group node, or vulnerability database reference survives it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): bind a dedupe root to its lifecycle ledger record Round-three review found the remaining half of the provenance contract was unstated. buildFindingSourceExpectations derives one expectation per upstream finding plus one per ledger record, and matches an emitted finding against whichever expectation shares an identity key with it. Only the ledger-record expectation carries a multi-node union, so a root that merged findings from two different nodes resolves to it solely through a shared dedupe_key. Neither prompt told the model to write that key onto the retained finding, so any genuine cross-lane merge failed with "source_nodes does not preserve the exact dependency discovery-source union", on both the production and smoke lanes. Say so in both prompts, and give the production prompt the exact-coverage rule the smoke prompt already gained: every upstream finding must appear exactly once across all source_artifacts, including ones kept only as a duplicate or family variant. The gate enforces that on both lanes; only the smoke prompt said it. Make the sync-side ledger reader fail the way the template does. Swallowing a parse error re-entered the naive expectation set the previous commit removed and reported a corrupt required artifact as a findings-contract violation. Reject a symlinked ledger or dependency findings file rather than following it out of the run root, matching resolveRegularArtifactFile. Pin the surviving node set in the curated-lane completeness test. Its shape checks could not see a default-on node added to a pre-existing group, which is exactly where threat-model and goal-plan live, so the guarantee it advertised was not the one it enforced. Add the regression test for the expectation set itself, and make the reference example match the shipped threat-goals node rather than a partial copy of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): give dedupe the full producer list its ledger is checked against Round-four review found the last gap in the new provenance gate. Ledger coverage is checked against every dependency findings artifact the runtime read, but the production dedupe prompt named only 5 of the 29 nodes that declare a findings.json output, and told the model never to guess a path. The other 24 -- boundary-tests, the invariant chain, the differential lanes and the rest -- were unreachable from the prompt, so any run in which one of them reported a finding failed dedupe-findings deterministically with "finding lifecycle ledger omitted dependency findings", then cascaded to triage, severity-classification and final-report. The smoke lane was immune only because it enumerates all four of its producers, which is why three earlier rounds and CI missed it. Enumerate every producer. Per-line rendering means a curated lane that prunes a node also drops its line, so an invariant-only lane sees exactly the lanes it still runs rather than an instruction to account for artifacts that no longer exist. State the two rules the gate enforces that the prompt left implicit: node_id is copied out of the finding rather than taken from the group template or the storage directory in its path, and a finding judged unsupported gets its own ledger record instead of being attached to an unrelated root, which would put a non-contributing node in that root's source union. Complete the sync/template mirror: a missing ledger now names itself instead of degrading to the naive expectation set and reporting a missing required artifact as a findings-contract violation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): state the source-union equality rule the review lane is held to Round-five review audited every throw site in the new provenance gate against what the prompts actually instruct, and found the production dedupe prompt still missing the rule the smoke prompt already had: a retained root's source_nodes must equal exactly the node_id set of its own ledger record's source_artifacts. Two of its sentences pushed the other way -- "preserve the relevant source union on family variants" invites a subset, since a variant's artifacts belong to the root's record, and "include all corroborating node IDs" invites a superset from related findings that were deliberately not merged. Either fails the node with the same opaque exact-union error. State the rule, and drop the two sentences that contradict it. Also close the smaller gaps that audit surfaced: every ledger record needs its own dedupe_key including the coverage-only ones, no two records may share one, triage and severity-classification must carry id and dedupe_key through rather than assigning a fresh stable id, and the restart-handling short circuit now lists finding-lifecycle-ledger.json so a resumed attempt cannot declare itself finished without the one output whose absence hard-fails the node. Pin the producer enumeration to the topology with a test, since nothing tied them together and the list had already drifted once. Document the ledger in the review guide, and make the goal-fanout header self-explaining when a curated lane prunes its paths away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): name the finding fields the review contracts now require This branch moves deduped-findings.json, triaged-findings.json and severity-classified-findings.json from ultrafuzz/json-array@1 to ultrafuzz/findings@1. That subjects all three to the full normalized-finding field check, not just the source-union rule, and none of the three prompts listed those fields. severity-classification was the worst case: `summary` is required, the prompt never mentioned it, and its four worked examples omit it, so a model following the only concrete shape in the file fails the node on every attempt. dedupe-findings was missing status, severity_guess and confidence. Name the required fields in all three, and say that the severity examples show only the fields that stage adds rather than the whole object. Pin it with a test, which caught the dedupe-findings gap that reading had missed. Also close the two remaining review items: the restart short circuit now validates the ledger's shape instead of only its existence, so a resumed attempt with a truncated or incomplete ledger rebuilds rather than declaring itself finished; and the producer-enumeration pin now covers the smoke lane too, which had the same drift hazard unguarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(prompts): pin every required finding field, not five of seven The field-contract test listed the five fields normalizeFinding hard-requires but omitted schema_version and id, which the finding schema also requires. All three review prompts already name all seven, so this changes no behaviour; it closes the hole where a later edit could drop one of the two unpinned mentions and leave the test green while the node fails its contract on every attempt -- the exact failure the test was written to prevent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Antonio Viggiano <3029017+aviggiano@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-192-168-1-2.ec2.internal> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
Brings B1 (#363 threat-model artifact retention) and B2 (#365 expansion observability) onto the release branch, plus #362 and the rest of main. workflow-sync.ts: kept #188's accumulating findingsCount AND #350's refreshVerifiedArtifactDigest re-seal — orthogonal, and dropping the latter would have silently reverted the #348 fix on this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
Recording the gate on this PR, so it is not inferred from the draft flag aloneThis PR is the open-source readiness gate. It merges v0.1.0 will not be run in full, and is not expected to be. More work lands in v0.1.0 after the current milestone issues close, so What that means for the open issues. #183's dynamic-fanout half is being completed (#364 decided, implementation in flight) so the capability exists and is measurable — but running it at full cohort scale is deferred, not a precondition for this merge. #351 stays draft accordingly. Current caveat on the smoke signal. The last two Modal runs on So the branch has not yet been validated by a smoke run — it has been unblocked by one. Getting that pair to complete and score is the actual acceptance signal, and it is being worked. Related: #354, where an empty diagnostics row set summarizes as |
This was referenced Aug 7, 2026
…anes at planning time (#373) Implements #364 option (a): the planner writes its dynamic-fanout expectation down, and the eval side compares against the run without ever recomputing the rule. Expected and observed then come from different components, so a planner that under-expands is caught. `goal-plan.json` gains `expected_child_count`, `threat_count`, `applicable_class_count`, `max_dynamic_nodes`, and `goal_lanes` (each lane named, with the node IDs it owns). They are recorded deterministically by the `goal-plan` node's postprocessor from the plan the agent wrote plus the run's resolved `run.max_dynamic_nodes`, before the plan is validated. A value the agent wrote that disagrees with its own plan fails closed rather than being silently corrected, and a plan expecting more children than `max_dynamic_nodes` allows fails as a plan. The cardinality rule -- `threats + applicable classes`, with the fixed roaming node as a static lane rather than a dynamic child -- is computed by exactly one function, `goalPlanExpansionFacts`. That now includes the pre-existing `counts` block, which computed the same arithmetic a second time twenty lines away: `counts.threats`, `counts.applicable_classes` and `counts.dynamic_goals` are checked against those facts instead of restating `threat_goals.length + class_goals.length`, so `threat_count`, `applicable_class_count` and `expected_child_count` are aliases that cannot drift from them. `max_dynamic_nodes` and `goal_lanes` are the only genuinely new information. Lane granularity is recorded rather than left implicit: a lane is one goal, so `lane_id` is a threat or class ID and a run has `threats + classes + 1` lanes. The alternative -- one lane per dynamic group, three per run -- aggregates more cheaply but cannot say which goal was expensive or which one failed. Group totals stay derivable by summing one `kind`; the reverse is not. Per-lane cost is now joined on a key that exists. `UsageLedgerEntry` gains an optional `node_id`, the identity `state.json` keys a node under, resolved by the runtime from its own task table -- a usage event names the workflow task (`node:<attempt>`), not the state node, and only the task table knows that mapping. `attempt_id` could not serve: it is `sha256(workflow run, node, iteration, attempt)` and carries no node field, so the previous join matched nothing on every real ledger while still reporting `lane_cost_evidence: complete`. The field is optional because `usage.jsonl` is durable: ledgers written before it existed still parse, and a run resumed across the upgrade does not fail its replay when a recorded entry gains the key. `evalRunExpansion` now reports `plan`, `expected_vs_actual`, and per-lane `goal_lanes`. It reads the planner's numbers and compares; a mismatch is kept as data with the size and sign of the disagreement, never dropped. Lanes fall out of per-node status for failures, the usage ledger for tokens and cost, and node timestamps for wall-clock, so metrics 5 and 6 need no new telemetry. Each lane carries `cost_evidence` and `usage_matched_node_count`, so a lane that genuinely spent nothing (zero, complete) is distinguishable from one whose join never landed (null, unavailable with a reason: no `node_id` in the ledger, no entry naming this lane's nodes, or no node observed for the lane at all). `EvalRowScore.expansion` stays optional so historical scores still read. The eval test fixture no longer hand-writes `usage.jsonl`. It calls `createUsageLedgerEntry`, the same function the runtime calls; the old fixture invented a `schema_version` and an `attempt_id` shape the real writer cannot produce, which is how a structurally dead join kept a green test. Dynamic child node states carry `provenance.source_node_id`, the one key both sides must agree on and the key #365's reader already looks for. The nested camelCase `dynamic` record stays alongside it for the expansion key, item digest, and manifest path. Verified: `pnpm -r build` 0, `pnpm -r test` 0, `pnpm -r --workspace-concurrency=1 typecheck` 0, `pnpm -w format:check` 0, `pnpm -w lint` 0. No benchmark run was dispatched. Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 7, 2026
Merged
* feat(evals): add the v0.1.0 threat-model release-gate benchmark lane (#183) The three pinned Ultrafuzz-bench targets have never seen the production topology. The smoke lane substitutes the reduced `benchmarks/smoke-benchmark.yml` graph, and the full lane is bound to the EVMBench cohort, so the `threat-model`, `goal-plan`, `threat-goals`, and `class-goals` nodes #188 added run against no real pinned protocol. This adds a third lane rather than editing the smoke graph, whose empty `excluded_node_ids` is what keeps the 99 published observations comparable. The `threat-model` lane pairs the Ultrafuzz-bench cohort with the unmodified production graph: no `topology` override, all three `disable_*` flags off, and an empty exclusion list. Two guards keep that honest. `loadBenchmarkLanesManifest` rejects a checked-in gate lane that turns any production work off, and `adaptBenchmarkManifestToEvalSuite` refuses to compile a suite whose exclusions would remove a node the gate exists to exercise, so a lane edit cannot quietly produce a green run that proves nothing. The gate takes its GPT-5.6 Luna/high runner from `benchmarks/lanes.json` instead of the dispatch form -- the workflow leaves `BENCHMARK_MODELS_JSON` unset for this lane -- so no dispatch-form value can retune its model identity. It runs three sandbox rows with the production graph's eight-way in-workflow concurrency, which is what a large dynamic ready queue needs to be scheduled rather than serialized, and takes the 15,000-second Modal watchdog ceiling. Beyond the report bundle it retains `THREAT_MODEL.md`, `threat-model.json`, `goal-plan.json`, and `vulnerability-db-manifest.json` from every row, because its assertions are deliberately structural and the prose has to stay reviewable. Gate results are not published into `benchmarks/history.json`. Publication is a longitudinal claim about one comparable series, and this lane runs a different topology and execution policy against the same targets. The producer now declares its dispatched lane as a `Benchmark lane <name>` step -- the `workflow_run` payload carries no dispatch inputs, and reading a step name is the introspection the producer's own recovery job already does -- and the qualifier declines any lane that is not the one its trigger is allowed to have produced, instead of assuming every dispatch is the EVMBench matrix. Tests assert the generated dynamic node IDs against the real expansion of the real topology templates: a contract-valid `ultrafuzz/goal-plan@1` document expanded through `planDynamicExpansion` must reproduce the plan's own `node_id` for every threat and class goal. That spelling is `dynamic:threat:<id>` / `dynamic:class:<id>`, which the topology templates, the goal-plan contract, the planner prompt, and the changelog all pin; #185's `dynamic.threat.…` sketch predates that decision and is not what #188 shipped. Nothing dispatches the lane. Selecting `benchmark_lane: threat-model` remains a maintainer decision with a real cost attached. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(evals): let the post-eval checkpoint accept every dispatchable lane (#183) The public eval diagnostics schema still enumerated `["smoke", "full"]` after the `threat-model` release gate became a third lane. The worker builds that document from `public_benchmark.lane` in `runAndCheckpointPublicEvalDiagnostics` only AFTER `runEval()` returns, so a `threat-model` dispatch would spend the whole cohort's model budget and then die at the post-eval-pre-score checkpoint with a Zod `invalid_value` on `lane` -- surfacing as a `PublicEvalDiagnosticsBuildError` and producing no diagnostics, no bundle, and no gate evidence at all. Derive the enum from `BENCHMARK_LANE_NAMES`, which is already the source for the Modal config, the public bundle, and the history observation schemas. The regression test builds the checkpoint for every lane in `BENCHMARK_LANE_NAMES` through the real `createPublicEvalDiagnostics` path rather than asserting the enum, so a future lane that forgets a schema fails here instead of on a paid run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(evals): wire the release-gate lane to the landed retention and observability Merges release/v0.1.0, which now carries B1 (#363) and B2 (#365), and removes the duplicated artifact list: PUBLIC_OPTIONAL_ROW_ARTIFACTS now derives from BENCHMARK_THREAT_MODEL_RETAINED_ARTIFACTS instead of repeating the same four names as a literal in a second package, where the two could drift apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(evals): repair threat-model lane on the release contract Populate the post-#373 goal-plan cardinality and lane fields from the shared authority, using the same 512-node limit exercised by dynamic expansion. Remove the reporter-only artifact include spread; the public worker remains the real retention path. * ci(evals): enforce threat-model structural gate --------- Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # .github/workflows/eval-benchmarks.yml # CHANGELOG.md # packages/artifacts/src/findings-schema.ts # packages/artifacts/src/findings.ts # packages/artifacts/test/artifacts.test.ts # packages/artifacts/test/schema.test.ts # packages/modal/src/node-provider.ts # packages/modal/src/node-worker.ts # packages/modal/test/ci-config.test.ts # packages/runtime/src/index.ts # scripts/ci/qualify-modal-benchmark-publication.mjs # scripts/ci/qualify-modal-benchmark-publication.test.ts
# Conflicts: # packages/modal/test/ci-config.test.ts
* fix(runtime): seal workflow execution controls * fix(runtime): recover stale snapshot publications * fix(runtime): persist detached task control paths * fix(runtime): guide sealed workflow upgrades * fix(runtime): keep sealed control paths private * fix(runtime): publish stock adapter upgrades atomically * test(cli): synchronize status watch samples * fix(runtime): harden stock adapter publication * style(cli): format status watch synchronization * fix(runtime): recover torn adapter marker * Harden init publication and file writes --------- Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
…-sync-ready Sync main fixes into release/v0.1.0 without reverse integration
…0.0.2 # Conflicts: # .ultrafuzz/prompts/review/final-report.md # CHANGELOG.md # packages/runtime/src/templates/smithers/workflows/workflow.tsx
fix(references): bump vulnerability database to v0.0.2
12 of 31 tasks
Collaborator
Author
Exact-head readiness checkpoint (not merge approval)Validated
This clears the previously recorded need for a genuinely completed and scored smoke signal. It is not approval to merge or undraft. Remaining owner decisions/caveats:
Keeping this PR draft until the explicit owner decision. |
…v0.1.0 # Conflicts: # docs/reference/topology-yaml.md # packages/runtime/src/types.ts
Collaborator
Author
Exact-head readiness checkpoint (not merge approval)Validated
This is a readiness checkpoint only. It is not permission to merge or undraft. Remaining owner decisions/caveats:
Keeping this PR draft and unmerged pending the explicit owner decision. |
Collaborator
Author
|
Checkpoint — 2026-08-11 UTC The v0.1.0 candidate is complete and ready for the eventual release decision, but remains intentionally draft and unmerged.
Next steps when the project is ready to cut v0.1.0:
No additional implementation work is currently known for this PR. PR #457 is deliberately separate and is not a prerequisite for this v0.1.0 candidate. |
20 tasks
Preserve the established v0.0.x fuzzing graph as a packaged v0.1.0 profile while excluding the vulnerability database, threat model, and goal fanout.\n\nCloses #559
This was referenced Aug 14, 2026
Collaborator
Author
Superseded integration policy — 2026-08-14Do not merge this PR. The project has moved to the stabilization-first v0.1.0 plan in #184. Shared bug fixes and the explicitly approved OpenRouter integration (#584 / PR #586) will land directly on The replacement flow is: stabilize and review Closing this PR is bookkeeping, not a release cancellation and not a statement that its historical work or smoke evidence is invalid. Preserve this PR and |
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.