Repository navigation
Upgrade the pinned workflow engine to Smithers 0.32.0 - #276
Merged
Merged
Conversation
0.32.0 is the latest release published as `smithers-orchestrator`; 0.33.0 renamed the package to `smthrs`, which is a breaking packaging migration rather than a bugfix and is deliberately out of scope here. None of our four runner source patches became redundant: every anchor is byte-identical to 0.31.0, so the detached admission ceiling, the dedicated supervisor log descriptor, the scheduler terminal-state restore and the resume hydration workarounds all stay. What did change underneath us is the Effect major, so the generated manifest's `overrides.effect` moves from 3.21.4 to 4.0.0-beta.102 to keep a single Effect module identity. Also migrate the outgoing 0.31.0 manifest shape forward. It matched none of the recognised shapes, so every project root carrying a 0.31.0 `.smithers/package.json` -- including in-flight cloud runs resuming against their existing root -- would have hard-failed before the engine started. Report all four workarounds from `doctor`. It previously reported only the two CLI patches, leaving the two resume-durability patches invisible. Adds three regression tests, each verified to fail without its fix: verifying the patch anchors against the actually installed release, the 0.31.0 manifest migration, and the full compatibility-patch posture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Review of the upgrade found the resume-hydration workaround was silently
mis-ordered by 0.32.0. Upstream moved its eager run-startup sequence into a
`startRunRuntime()` closure and, on a resume, calls it from inside the
renderer. The old `const driverRenderer = {` anchor therefore ran our
hydration BEFORE the stale-attempt reset instead of after it, so a node with
a lingering in-progress attempt was restored as finished and the reset then
rewrote the durable row to pending. The scheduler session and the database
disagreed for the whole resume, which is the divergence the workaround
exists to prevent. Re-anchor after `await startRunRuntime();` and assert the
ordering positionally against the pinned release.
Pin the `@effect/*` packages alongside Effect itself. Pinning only `effect`
left `@effect/platform-node-shared` floating on an open caret over
prereleases, and the generated workspace installs with
`--package-lock=false`: npm resolved beta.104 while pnpm resolved beta.103,
so CI never exercised the tree cloud runs install and two containers
resuming one run could disagree.
Stop a half-reified `.smithers/node_modules` from bricking a project root.
Install validation only inspects the top-level runner, so an interrupted
upgrade passed it and then threw from the patcher on a path that returned
instead of reinstalling; every later resume failed identically. A patch
failure there now falls through to the reinstall.
Also: the patcher no longer silently skips the resume patches when only the
CLI package is missing; `doctor` reports an ambiguous package root as
incompatible rather than unknown, and its registry check follows the
upstream package rename instead of asking about a name frozen forever at
0.32.0; the anchor test carries the upstream evidence justifying each
workaround; and the patcher test is seeded from the descriptions so a
described workaround cannot land reported-but-never-applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second review round found three defects in the upgrade itself. The Effect pins only covered the generated runner workspace, so this repository still resolved `@effect/platform-node-shared@4.0.0-beta.103` with an unsatisfied Effect peer while cloud runs installed beta.102. The claim that CI exercised the tree production installs was therefore false. Pin the same set in `pnpm-workspace.yaml`, which is where pnpm 11 reads overrides from, and add a check that keeps the two pin sets identical. `doctor` leaked the upstream engine package name into operator-facing text. The repository de-brands the engine everywhere else, and the successor name slips past the `smithers` scrub by spelling alone, so the guard stayed green while the invariant broke. Describe the rename without naming it, and extend the branding assertions to cover the successor spelling. The registry check also reported "you are on the latest release" whenever the successor lookup alone failed, which is the exact false positive the rename handling was written to remove. Report a partial lookup as a warning instead. Alongside those: preserve the original patch-failure cause through the repair reinstall instead of discarding it, bound that reinstall to once per project root per process so a permanently unpatchable tree fails fast rather than paying a full install on every engine command, surface a renamed-package flag rather than an ambiguous bare version, pin the unconditional resume reset (not just the age-gated sweep) in the ordering assertion and require every marker to be unique, group patcher-test fixtures by source file, and drop a stale version-specific comment the de-versioning pass missed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aviggiano
added a commit
that referenced
this pull request
Aug 6, 2026
Integrates the Smithers 0.32.0 engine upgrade (#276), whose compatibility-patch rework overlaps this branch's own runner patches. Conflict resolution: - Take main's data-driven patch registry, generalized `compatibility_patches` record, and registry-seeded test fixture. They supersede this branch's hardcoded patch list, and the seeded fixture covers newly described patches automatically. - Re-add this branch's twelve runner workarounds as registry entries: the prepare-only fork and replay invocations, and the workflow-path and metadata persistence this branch relies on. Every anchor was re-verified to occur exactly once in the newly pinned 0.32.0 sources, and each is applied again — main's apply path only covers the workarounds main itself carries, so registry entries without it would be reported by `doctor` and never applied. - Drop the resume-activation workflow-path patch. 0.32.0 already passes the resolved workflow path into `activateRunForResume`, and the persisted-path patch requires the resolved and persisted paths to name the same file, so the separate patch is now redundant and its anchor no longer exists. - Bump the directly declared `@smithers-orchestrator/tool-context` pin from 0.31.0 to 0.32.0 to match the newly pinned engine, and regenerate the lockfile. - Keep both sides of the additive test changes: union the imports in the runtime lifecycle test, and assert this branch's prepare-only workarounds alongside main's resume-durability ones in the CLI doctor test. - Rework main's per-patch posture rotation to group by source file. Thirteen workarounds now patch the same CLI source, so a distinct posture per patch is only expressible within a file for the applied and unpatched states; the whole-file incompatible and unknown states are assigned to single-workaround sources, which keeps every posture branch exercised and still catches a swapped mapping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aviggiano
added a commit
that referenced
this pull request
Aug 7, 2026
The branch carried two lockfile edits against Smithers 0.31.0 that conflicted with #276's 0.32.0 upgrade on main. Resolve by taking main's 0.32.0 lockfile and regenerating the workspace links from the package manifests, which restores the @ultrafuzz/prompts link into packages/artifacts. pnpm install --frozen-lockfile now succeeds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aviggiano
added a commit
that referenced
this pull request
Aug 7, 2026
* feat: add threat model and dynamic goal fanout * feat: add dynamic runtime topology nodes * merge: preserve threat provenance with dynamic runtime nodes * fix: retain packaged prompt assets with dynamic variables * feat(references): integrate external vulnerability database * fix(references): harden vulnerability database sync * feat(runtime): consume pinned vulnerability database reference * fix(topology): bind vulnerability reference semantics * fix(runtime): export Smithers runtime helpers * test(runtime): fixture pinned vulnerability database reference * style: format changed workflow files * test: align topology anchors and lint unused fixtures * test(cli): fixture pinned vulnerability reference for validation * fix(report): allow safe audit-context links * docs: clarify vulnerability database materialization * fix(cloud): harden cloud selected-task handoff boundary Consolidates the reviewed PR #188 remediation work into an intentional commit: the shared cloud selected-task handoff contract, the Modal provider and worker boundary validation, native-v1 vulnerability database materialization, and the threat-model/goal-plan artifact schemas with their tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(cloud): correlate and fully bind the cloud selected-task boundary Closes the five audited cloud-boundary gaps and three prior-review regression gaps: - reject a non-empty outer `tasks` array for a `cloud_worker` dispatch, so an outer entry can neither replace the relocated prompt body nor name a second attempt - reconstruct a runtime-generated dynamic attempt's whole canonical DTO from its group's compiled template, binding the run root and every derived path, the rendered prompt location, the agent and model profile, the execution resources, the artifact output contracts, the pinned planner catalog, the reference trees, and its dynamic provenance; only the expansion key and the two runtime digests remain runtime data, and the generating group is derived from the dispatched attempt ID rather than taken from the handoff - remove the production no-`selected_task` compatibility path and cover its absence for compiled, deferred, and generated attempts - persist a canonical logical-dispatch fingerprint (generation-independent) and check it on durable resume, in checkpoint records and indexes, in published results, and in live sandbox reattachment, keeping the exact same-generation resume check distinct from the allowed reset-generation restore - replace the three independent dependency-array supersets with one correlated runtime-materialization check: each added artifact directory belongs to one added attempt, each verifier derives from one, and each added generated node must be derivable from a declared group Regression gaps: `dynamic-workflow.test.ts` now uses a real non-empty prompt exclusion and asserts transformed bytes for both the dynamic and the deferred snapshot; topology `schema.test.ts` validates a generated graph against the committed expanded-graph JSON Schema document; the stale contract-valid-today comment in the artifacts threat-goal tests is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(preflight): ignore prompt pack test scratch directories * fix(cloud): bind durable recovery to handoff contents * docs(references): state the private-database read prerequisite exactly `monad-developers/web3-vulnerability-database` is a private repository, so a clean scaffold that pins `vulnerability-database.web3` cannot materialize it without an ambient git read credential. Record that prerequisite where the pin is documented, quoting the diagnostic the CLI actually emits. Verified against the pinned revision with no credential available: error: GIT_FAILED: git command failed: git fetch --depth=1 \ --filter=blob:none origin fbf00e990b1316879b674e9903548dba452e40d5: \ remote: Repository not found. fatal: Authentication failed for 'https://github.com/monad-developers/web3-vulnerability-database.git/' The failure surfaces as the `GIT_FAILED` reference diagnostic wrapping git stderr, not as a bare git error, and dropping only the catalog entry leaves `reference-vulnerability-database` dangling, which `expand()` rejects with `INVALID_REFERENCE_NODE`. Both are stated as observed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(references): authenticate the pinned private database read The curated vulnerability database is a private repository, but reference materialization fetched it with a plain anonymous `git fetch`. The detached Modal worker runs `ultrafuzz references sync` in its pre-model phase, so every benchmark pair failed there: a dependency-unreachable error with `model_work_started=false` and `usage=null`, after an image build and a launch had already been paid for. Make the credential an explicit, least-privilege contract instead. References stay anonymous by default and attach a token only when both `ULTRAFUZZ_REFERENCE_GITHUB_TOKEN` and `ULTRAFUZZ_REFERENCE_GITHUB_REPOS` are declared. The allowlist is what carries the security property: a token minted for one private repository is never sent to another remote just because the catalog names it, and a token without an allowlist is inert. The token reaches git only through `GIT_CONFIG_*` carrying an `http.<remote>.extraheader` keyed to the exact remote, so it never enters a remote URL, an argv entry, a config file, or a credential helper; `GIT_FAILED` diagnostics redact both the token and its derived basic-auth encoding. CI mints a short-lived token from the existing eval-history GitHub App, scoped to `contents:read` on `web3-vulnerability-database` alone, and proves access with a new preflight before the image build -- immutable repository metadata and the pinned commit only, so a missing installation or permission costs nothing and reports the exact external action required. The private key never leaves the trusted runner; only the token enters the Modal secret, where it is also a forbidden value for published bundles, diagnostics, and lifecycle logs. `launch` and `collect` each mint their own token with `skip-token-revoke: true`: installation tokens expire after an hour while `collect` can wait longer, and the sandboxes are detached, so a post-job revocation would invalidate a credential a worker still needs. The workflow keeps its push/workflow_dispatch-only trigger, so fork code never reaches the private key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(preflight): report the App installation prerequisite exactly The first CI run proved the credential contract stops before paid compute -- the image build and every sandbox launch were skipped -- but it also exposed a gap in what the failure actually says. `actions/create-github-app-token` fails first, so the preflight that owns the precise diagnosis never ran, and the only evidence was GitHub's bare `Not Found` from `GET /repos/{owner}/{repo}/installation`. That reads like an infrastructure fault rather than the one external action it represents. Let the mint step fail soft and make the preflight the step that fails the job. An absent token in CI means the App has no installation on the target repository, so the preflight now names that: install the eval-history GitHub App with `Contents: Read-only`, an administrator must do it, and the pin must not be replaced, vendored, or removed to work around it. The job still stops before the image build, so this changes only the diagnosis, not the compute that is refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(collect): keep live sandboxes collectable when the token mint fails The recovery token step in `collect` failed hard, and the wait step it precedes is gated on `available == 'true'` without `always()`. A failed mint therefore skipped waiting entirely and abandoned every already-running sandbox uncollected, even though the collection and diagnostic-upload steps that follow do run on `always()` and would then find nothing. Fail that step soft. Waiting and collection stay intact, and a relaunch that genuinely needs the credential still fails on its own pair with a recorded diagnostic instead of taking the whole matrix down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(collect): gate private-reference recovery access * fix(eval): harden private reference recovery * fix(references): use public vulnerability database * fix(runtime): avoid Smithers reserved input field * fix(runtime): normalize persisted workflow input nulls * fix(smoke): require canonical evidence locations * fix(prompts): bind goal IDs to source records * fix(prompts): bind threat-model kinds and goal-plan catalog digest to their validators The Modal smoke benchmark failed deterministically per target at the two new fanout nodes, reproduced in a live sandbox at this head: - threat-model: agents invented invariant kinds (solvency, state-consistency, configuration, availability) because neither threat-model prompt stated the ultrafuzz/threat-model@1 enum. Both prompts now spell out the five valid kinds inline. - goal-plan: the prompt said to copy all database digests from the supplied catalog, so agents set catalog_sha256 to the embedded upstream_catalog_sha256; the validator expects the SHA-256 of the planner catalog file itself. The prompt now says to compute it over the exact bytes at the supplied catalog path and warns that upstream_catalog_sha256 never validates there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): pin remaining fanout artifact fields to validator semantics A second live-sandbox round with the previous fix in place surfaced the rest of the deterministic agent/validator mismatches: - threat-model: string fields with plural names (evidence_needed, impact, security_impact, value_at_risk) were emitted as arrays. Both prompts now state the array-vs-string rule for every leaf field. - goal-plan: agents pre-wrote vulnerability-db-manifest.json, which conflicts with the canonical bytes Ultrafuzz publishes after the node; applicability checks paraphrased threat-model rationale/evidence instead of copying byte-for-byte; class-goal attack_surface_ids carried invented labels instead of threat-model attack-surface IDs. The prompt now forbids writing the manifest or vulnerability-db/ content, requires byte-for-byte capability copies, and pins attack_surface_ids to copied threat-model IDs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(runtime): stop pre-materializing the vulnerability-db snapshot manifest Task preparation backfilled every non-primary declared output with its contract's canonical empty bytes, so goal-plan nodes always started with a 0o600 '{}' placeholder at vulnerability-db-manifest.json. The post-agent snapshot publisher writes that manifest exclusively and fails closed on conflicting content, which made every goal-plan attempt die with ARTIFACT_CONFLICT once its plan validated (reproduced across all three smoke benchmark targets in a live Modal sandbox). Exclude the snapshot manifest from canonical-empty preparation exactly like the runtime-owned workspace patch outputs, and pin the exclusion with a generated-workflow test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): make goal-plan capability copies and placeholder retention mechanical The remaining goal-plan agent failures were content slips on one target: paraphrased capability rationale/evidence in applicability checks and coverage-gap goal_prompts with the {{threat-model:coverage-gap}} placeholder substituted away. Tell the agent to build checks mechanically from threat-model.json and to keep every replacement key's literal {{...}} braces in goal_prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): keep coverage-gap placeholder wording template-safe The previous commit's literal {{<key>}} example was itself parsed as a prompt template variable and failed catalog validation. Reword to describe the double-brace form; the exact {{threat-model:coverage-gap}} text is an item-scoped placeholder the validator accepts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prompts): drop literal coverage-gap braces the topology validator rejects The topology artifact-handoff validator parses double-brace sequences in node prompts as template variables and does not recognize threat-model:coverage-gap outside dynamic-goal contexts. Describe the brace-wrapped key without writing the literal placeholder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(references): assert the public database stays anonymous under a credential The allowlist negative case named a non-existent monad-developers repository, which both carried the pre-transfer owner and asserted nothing about a reference Ultrafuzz actually ships. Use the real pinned public database instead, so the case proves the shipped vulnerability-database reference is fetched anonymously even while a private-reference token is configured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(references): re-pin the vulnerability database to the current corpus release The shipped pin still referenced fbf00e99 (18 records, 21 capabilities, aggregate c0ed7d23). Upstream has since published two further commits, and the corpus now carries 23 records and 28 capabilities under aggregate 62b7fb80, including the new classes/lending category. Re-pin to e46c0e47, the current immutable head. Its database.yml, capabilities.yml and catalog.json are byte-identical to 153e14cc; the intervening commit only changed upstream ingestion tooling, which Ultrafuzz does not consume. Verified by a fresh unauthenticated fetch with GitHub tokens, SSH agent and credential helpers disabled. Regenerate the documented digest table and catalog statistics to match, and correct the test comment that still described the repository as private. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(deps): reconcile the lockfile with Smithers 0.32.0 after rebase The branch carried two lockfile edits against Smithers 0.31.0 that conflicted with #276's 0.32.0 upgrade on main. Resolve by taking main's 0.32.0 lockfile and regenerating the workspace links from the package manifests, which restores the @ultrafuzz/prompts link into packages/artifacts. pnpm install --frozen-lockfile now succeeds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(evals): keep the smoke lane free of dynamic goal fanout The smoke lane declares disable_dynamic_strategies, but that flag only ever pruned dynamic-strategy-generator, which the dedicated smoke graph does not contain. This branch adds threat-goals and class-goals to benchmarks/smoke-benchmark.yml, and those expand one child per planned threat and per applicable class, so the lane would have run unbounded dynamic work while still reporting dynamic strategies as disabled. #277's smoke matrix requires that guarantee to hold. Name the goal-fanout node IDs, add them to the dynamic exclusion set, and apply them on the smoke lane. Production-topology exclusions still cannot be passed wholesale to the smoke graph, because its invariant, differential and dynamic-strategy nodes are absent and would raise unknown-node errors. Add a regression test that reads the smoke topology and asserts the lane prunes every dynamic node it declares, so a future dynamic node added to the smoke graph fails the suite instead of silently running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(modal): keep curated private lanes off the new threat-model nodes A private benchmark lane isolates part of the production topology by naming the nodes it wants pruned. The invariant-only Aave lane does this. Every such list was written before threat-model, goal-plan, goal-roaming, threat-goals, class-goals and reference-vulnerability-database existed, and the first four land in the pre-existing setup and goals groups, so no group-level filter catches them either. Merging this branch would therefore have made the next invariant-only run quietly execute a threat model and unbounded goal hunters, invalidating the comparison it exists to produce. Publish the default-on node IDs from @ultrafuzz/evals and prune them from any private lane that curates an exclusion list, with include_threat_model_goal_fanout to opt back in. An empty list still means "run the whole topology", so uncurated private runs are unchanged. Also prune goal-plan on the smoke lane. The fanout is its only dependent, so excluding just the fanout leaves it terminal and strands the smoke report prompt's artifact_path citation of it; the lane could not plan at all. A new runtime suite expands the real pruned graph to prove both directions. Keep the topology-derived tests honest: the published ID list is checked against the shipped topology, so a future default-on node fails the suite instead of silently widening every curated lane. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(evals): prune goal-plan wherever the goal fanout is pruned Excluding threat-goals and class-goals on their own leaves goal-plan with no dependent, which makes it terminal, and strands the report prompts' artifact_path citation of it, which requires an ancestor. Either way the graph stops validating, so any lane that drops the fanout has to drop goal-plan too. Fold that into BENCHMARK_GOAL_FANOUT_EXCLUDED_NODE_IDS and build the smoke, production-dynamic and curated-lane sets from it, so all three stay valid against the graph they are applied to. Move the pruning coverage into packages/runtime/test/topology-transform.test.ts next to the existing exclusion tests, in the node:test style that package's runner actually compiles and executes, and extend its production-topology case with the same three IDs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(evals): keep the benchmark smoke lane exactly as published Three independent reviews of 7e2687d found that pruning the goal fanout at lane level was not enough to keep the smoke lane's guarantee. Per-run agent concurrency is max_parallel_targets, which is 4, so the pruned 12-node graph still serialized into six waves: 6 x 2 x 1200s = 14,400s against a 15,000s row watchdog that packages/modal/src/config.ts caps at 15,000. The margin fell from 5,400s to 600s and could not be raised without a schema change. Keeping threat-model also kept requiresVulnerabilityDatabase true, so every smoke run gained a plan-time corpus fetch and the exact-cover capability gate in verifyThreatModelVulnerabilityDatabaseCapabilities -- at medium reasoning with two attempts. In a model matrix that gate fails a row outright, so a weaker entrant publishes no observation at all and the comparison is silently biased. Passing any non-empty excluded_node_ids also moves benchmark_execution, and with it the execution-policy and cohort fingerprints that #277's rows must share with the already-published smoke observations. Restore benchmarks/smoke-benchmark.yml byte-for-byte to main, drop the unused smoke threat-model prompt, and take the smoke lane back to an empty exclusion list. The lane keeps its guarantee structurally, the way it always did: the dedicated graph simply does not declare invariant, differential, dynamic-strategy or goal-fanout nodes. A test now asserts that rather than asserting a pruning list. Keep the prompt-only smoke improvements that carry no node dependency: the evidence-path hardening and the source_nodes provenance bullets. The full lane is untouched and still exercises the whole threat-model workstream, and the production topology keeps every default-on node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(runtime): rebuild dedupe provenance from the lifecycle ledger when syncing Three independent reviews of 0d3e5c2 found two real regressions in the new finding-provenance machinery, both introduced by this branch. dependencyFindingProvenanceForTask rebuilt the source expectations without the lifecycle ledger, so buildFindingSourceExpectations returned early and produced one single-source expectation per upstream finding. A dedupe root's source_nodes is the union of everything it merged, which matches none of those, so finalizeTerminalTask re-validated a node the authoritative in-workflow gate had already passed and marked it failed with task-output-validation-failure. That fires on any run with a genuine merge, and it is reachable from ultrafuzz inspect, ultrafuzz status, and the dashboard, so it needs no cloud execution. It also attributed upstream findings to the dependency attempt ID rather than the producing node, so a generated dynamic child could never match its own ledger record. Load the ledger and resolve producer node IDs through tasksByAttempt, mirroring the generated template, which is the authoritative implementation of the same contract. Stop creating artifact directories from this read-only path while here. The smoke prompts described a lifecycle ledger the new gate rejects: entries need node_id and finding_id and must cover every upstream finding exactly once, which only the production prompt said. Bring the smoke prompt up to the same contract and tell the smoke report to carry finding identity through, since provenance is matched on it. Also correct a comment claiming the goal fanout exists in the smoke graph, a reference example naming a node and prompt that do not exist, and the changelog description of the roaming goal, which is a static node rather than an expansion. Document the deliberate asymmetry in the private-reference credential contract and the one-off lineage discontinuity a curated private lane sees. Add a production-topology test proving the exact set every curated lane prunes still plans, and that no dynamic node, goals-group node, or vulnerability database reference survives it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): bind a dedupe root to its lifecycle ledger record Round-three review found the remaining half of the provenance contract was unstated. buildFindingSourceExpectations derives one expectation per upstream finding plus one per ledger record, and matches an emitted finding against whichever expectation shares an identity key with it. Only the ledger-record expectation carries a multi-node union, so a root that merged findings from two different nodes resolves to it solely through a shared dedupe_key. Neither prompt told the model to write that key onto the retained finding, so any genuine cross-lane merge failed with "source_nodes does not preserve the exact dependency discovery-source union", on both the production and smoke lanes. Say so in both prompts, and give the production prompt the exact-coverage rule the smoke prompt already gained: every upstream finding must appear exactly once across all source_artifacts, including ones kept only as a duplicate or family variant. The gate enforces that on both lanes; only the smoke prompt said it. Make the sync-side ledger reader fail the way the template does. Swallowing a parse error re-entered the naive expectation set the previous commit removed and reported a corrupt required artifact as a findings-contract violation. Reject a symlinked ledger or dependency findings file rather than following it out of the run root, matching resolveRegularArtifactFile. Pin the surviving node set in the curated-lane completeness test. Its shape checks could not see a default-on node added to a pre-existing group, which is exactly where threat-model and goal-plan live, so the guarantee it advertised was not the one it enforced. Add the regression test for the expectation set itself, and make the reference example match the shipped threat-goals node rather than a partial copy of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): give dedupe the full producer list its ledger is checked against Round-four review found the last gap in the new provenance gate. Ledger coverage is checked against every dependency findings artifact the runtime read, but the production dedupe prompt named only 5 of the 29 nodes that declare a findings.json output, and told the model never to guess a path. The other 24 -- boundary-tests, the invariant chain, the differential lanes and the rest -- were unreachable from the prompt, so any run in which one of them reported a finding failed dedupe-findings deterministically with "finding lifecycle ledger omitted dependency findings", then cascaded to triage, severity-classification and final-report. The smoke lane was immune only because it enumerates all four of its producers, which is why three earlier rounds and CI missed it. Enumerate every producer. Per-line rendering means a curated lane that prunes a node also drops its line, so an invariant-only lane sees exactly the lanes it still runs rather than an instruction to account for artifacts that no longer exist. State the two rules the gate enforces that the prompt left implicit: node_id is copied out of the finding rather than taken from the group template or the storage directory in its path, and a finding judged unsupported gets its own ledger record instead of being attached to an unrelated root, which would put a non-contributing node in that root's source union. Complete the sync/template mirror: a missing ledger now names itself instead of degrading to the naive expectation set and reporting a missing required artifact as a findings-contract violation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): state the source-union equality rule the review lane is held to Round-five review audited every throw site in the new provenance gate against what the prompts actually instruct, and found the production dedupe prompt still missing the rule the smoke prompt already had: a retained root's source_nodes must equal exactly the node_id set of its own ledger record's source_artifacts. Two of its sentences pushed the other way -- "preserve the relevant source union on family variants" invites a subset, since a variant's artifacts belong to the root's record, and "include all corroborating node IDs" invites a superset from related findings that were deliberately not merged. Either fails the node with the same opaque exact-union error. State the rule, and drop the two sentences that contradict it. Also close the smaller gaps that audit surfaced: every ledger record needs its own dedupe_key including the coverage-only ones, no two records may share one, triage and severity-classification must carry id and dedupe_key through rather than assigning a fresh stable id, and the restart-handling short circuit now lists finding-lifecycle-ledger.json so a resumed attempt cannot declare itself finished without the one output whose absence hard-fails the node. Pin the producer enumeration to the topology with a test, since nothing tied them together and the list had already drifted once. Document the ledger in the review guide, and make the goal-fanout header self-explaining when a curated lane prunes its paths away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(prompts): name the finding fields the review contracts now require This branch moves deduped-findings.json, triaged-findings.json and severity-classified-findings.json from ultrafuzz/json-array@1 to ultrafuzz/findings@1. That subjects all three to the full normalized-finding field check, not just the source-union rule, and none of the three prompts listed those fields. severity-classification was the worst case: `summary` is required, the prompt never mentioned it, and its four worked examples omit it, so a model following the only concrete shape in the file fails the node on every attempt. dedupe-findings was missing status, severity_guess and confidence. Name the required fields in all three, and say that the severity examples show only the fields that stage adds rather than the whole object. Pin it with a test, which caught the dedupe-findings gap that reading had missed. Also close the two remaining review items: the restart short circuit now validates the ledger's shape instead of only its existence, so a resumed attempt with a truncated or incomplete ledger rebuilds rather than declaring itself finished; and the producer-enumeration pin now covers the smoke lane too, which had the same drift hazard unguarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(prompts): pin every required finding field, not five of seven The field-contract test listed the five fields normalizeFinding hard-requires but omitted schema_version and id, which the finding schema also requires. All three review prompts already name all seven, so this changes no behaviour; it closes the hole where a later edit could drop one of the two unpinned mentions and leave the test green while the node fails its contract on every attempt -- the exact failure the test was written to prevent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Antonio Viggiano <3029017+aviggiano@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-192-168-1-2.ec2.internal> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
aviggiano
added a commit
that referenced
this pull request
Aug 24, 2026
* feat: add threat-model goal fanout and dynamic topology nodes (#188)
* feat: add threat model and dynamic goal fanout
* feat: add dynamic runtime topology nodes
* merge: preserve threat provenance with dynamic runtime nodes
* fix: retain packaged prompt assets with dynamic variables
* feat(references): integrate external vulnerability database
* fix(references): harden vulnerability database sync
* feat(runtime): consume pinned vulnerability database reference
* fix(topology): bind vulnerability reference semantics
* fix(runtime): export Smithers runtime helpers
* test(runtime): fixture pinned vulnerability database reference
* style: format changed workflow files
* test: align topology anchors and lint unused fixtures
* test(cli): fixture pinned vulnerability reference for validation
* fix(report): allow safe audit-context links
* docs: clarify vulnerability database materialization
* fix(cloud): harden cloud selected-task handoff boundary
Consolidates the reviewed PR #188 remediation work into an intentional
commit: the shared cloud selected-task handoff contract, the Modal
provider and worker boundary validation, native-v1 vulnerability
database materialization, and the threat-model/goal-plan artifact
schemas with their tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(cloud): correlate and fully bind the cloud selected-task boundary
Closes the five audited cloud-boundary gaps and three prior-review
regression gaps:
- reject a non-empty outer `tasks` array for a `cloud_worker` dispatch,
so an outer entry can neither replace the relocated prompt body nor
name a second attempt
- reconstruct a runtime-generated dynamic attempt's whole canonical DTO
from its group's compiled template, binding the run root and every
derived path, the rendered prompt location, the agent and model
profile, the execution resources, the artifact output contracts, the
pinned planner catalog, the reference trees, and its dynamic
provenance; only the expansion key and the two runtime digests remain
runtime data, and the generating group is derived from the dispatched
attempt ID rather than taken from the handoff
- remove the production no-`selected_task` compatibility path and cover
its absence for compiled, deferred, and generated attempts
- persist a canonical logical-dispatch fingerprint (generation-independent)
and check it on durable resume, in checkpoint records and indexes, in
published results, and in live sandbox reattachment, keeping the exact
same-generation resume check distinct from the allowed
reset-generation restore
- replace the three independent dependency-array supersets with one
correlated runtime-materialization check: each added artifact
directory belongs to one added attempt, each verifier derives from
one, and each added generated node must be derivable from a declared
group
Regression gaps: `dynamic-workflow.test.ts` now uses a real non-empty
prompt exclusion and asserts transformed bytes for both the dynamic and
the deferred snapshot; topology `schema.test.ts` validates a generated
graph against the committed expanded-graph JSON Schema document; the
stale contract-valid-today comment in the artifacts threat-goal tests is
corrected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(preflight): ignore prompt pack test scratch directories
* fix(cloud): bind durable recovery to handoff contents
* docs(references): state the private-database read prerequisite exactly
`monad-developers/web3-vulnerability-database` is a private repository, so a
clean scaffold that pins `vulnerability-database.web3` cannot materialize it
without an ambient git read credential. Record that prerequisite where the pin
is documented, quoting the diagnostic the CLI actually emits.
Verified against the pinned revision with no credential available:
error: GIT_FAILED: git command failed: git fetch --depth=1 \
--filter=blob:none origin fbf00e990b1316879b674e9903548dba452e40d5: \
remote: Repository not found.
fatal: Authentication failed for
'https://github.com/monad-developers/web3-vulnerability-database.git/'
The failure surfaces as the `GIT_FAILED` reference diagnostic wrapping git
stderr, not as a bare git error, and dropping only the catalog entry leaves
`reference-vulnerability-database` dangling, which `expand()` rejects with
`INVALID_REFERENCE_NODE`. Both are stated as observed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(references): authenticate the pinned private database read
The curated vulnerability database is a private repository, but reference
materialization fetched it with a plain anonymous `git fetch`. The detached Modal
worker runs `ultrafuzz references sync` in its pre-model phase, so every
benchmark pair failed there: a dependency-unreachable error with
`model_work_started=false` and `usage=null`, after an image build and a launch
had already been paid for.
Make the credential an explicit, least-privilege contract instead.
References stay anonymous by default and attach a token only when both
`ULTRAFUZZ_REFERENCE_GITHUB_TOKEN` and `ULTRAFUZZ_REFERENCE_GITHUB_REPOS` are
declared. The allowlist is what carries the security property: a token minted for
one private repository is never sent to another remote just because the catalog
names it, and a token without an allowlist is inert. The token reaches git only
through `GIT_CONFIG_*` carrying an `http.<remote>.extraheader` keyed to the exact
remote, so it never enters a remote URL, an argv entry, a config file, or a
credential helper; `GIT_FAILED` diagnostics redact both the token and its derived
basic-auth encoding.
CI mints a short-lived token from the existing eval-history GitHub App, scoped to
`contents:read` on `web3-vulnerability-database` alone, and proves access with a
new preflight before the image build -- immutable repository metadata and the
pinned commit only, so a missing installation or permission costs nothing and
reports the exact external action required. The private key never leaves the
trusted runner; only the token enters the Modal secret, where it is also a
forbidden value for published bundles, diagnostics, and lifecycle logs.
`launch` and `collect` each mint their own token with `skip-token-revoke: true`:
installation tokens expire after an hour while `collect` can wait longer, and the
sandboxes are detached, so a post-job revocation would invalidate a credential a
worker still needs. The workflow keeps its push/workflow_dispatch-only trigger, so
fork code never reaches the private key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(preflight): report the App installation prerequisite exactly
The first CI run proved the credential contract stops before paid compute -- the
image build and every sandbox launch were skipped -- but it also exposed a gap in
what the failure actually says. `actions/create-github-app-token` fails first, so
the preflight that owns the precise diagnosis never ran, and the only evidence was
GitHub's bare `Not Found` from `GET /repos/{owner}/{repo}/installation`. That reads
like an infrastructure fault rather than the one external action it represents.
Let the mint step fail soft and make the preflight the step that fails the job. An
absent token in CI means the App has no installation on the target repository, so
the preflight now names that: install the eval-history GitHub App with
`Contents: Read-only`, an administrator must do it, and the pin must not be
replaced, vendored, or removed to work around it.
The job still stops before the image build, so this changes only the diagnosis, not
the compute that is refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(collect): keep live sandboxes collectable when the token mint fails
The recovery token step in `collect` failed hard, and the wait step it precedes is
gated on `available == 'true'` without `always()`. A failed mint therefore skipped
waiting entirely and abandoned every already-running sandbox uncollected, even
though the collection and diagnostic-upload steps that follow do run on `always()`
and would then find nothing.
Fail that step soft. Waiting and collection stay intact, and a relaunch that
genuinely needs the credential still fails on its own pair with a recorded
diagnostic instead of taking the whole matrix down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(collect): gate private-reference recovery access
* fix(eval): harden private reference recovery
* fix(references): use public vulnerability database
* fix(runtime): avoid Smithers reserved input field
* fix(runtime): normalize persisted workflow input nulls
* fix(smoke): require canonical evidence locations
* fix(prompts): bind goal IDs to source records
* fix(prompts): bind threat-model kinds and goal-plan catalog digest to their validators
The Modal smoke benchmark failed deterministically per target at the two new
fanout nodes, reproduced in a live sandbox at this head:
- threat-model: agents invented invariant kinds (solvency, state-consistency,
configuration, availability) because neither threat-model prompt stated the
ultrafuzz/threat-model@1 enum. Both prompts now spell out the five valid
kinds inline.
- goal-plan: the prompt said to copy all database digests from the supplied
catalog, so agents set catalog_sha256 to the embedded
upstream_catalog_sha256; the validator expects the SHA-256 of the planner
catalog file itself. The prompt now says to compute it over the exact bytes
at the supplied catalog path and warns that upstream_catalog_sha256 never
validates there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): pin remaining fanout artifact fields to validator semantics
A second live-sandbox round with the previous fix in place surfaced the rest
of the deterministic agent/validator mismatches:
- threat-model: string fields with plural names (evidence_needed, impact,
security_impact, value_at_risk) were emitted as arrays. Both prompts now
state the array-vs-string rule for every leaf field.
- goal-plan: agents pre-wrote vulnerability-db-manifest.json, which conflicts
with the canonical bytes Ultrafuzz publishes after the node; applicability
checks paraphrased threat-model rationale/evidence instead of copying
byte-for-byte; class-goal attack_surface_ids carried invented labels
instead of threat-model attack-surface IDs. The prompt now forbids writing
the manifest or vulnerability-db/ content, requires byte-for-byte capability
copies, and pins attack_surface_ids to copied threat-model IDs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(runtime): stop pre-materializing the vulnerability-db snapshot manifest
Task preparation backfilled every non-primary declared output with its
contract's canonical empty bytes, so goal-plan nodes always started with a
0o600 '{}' placeholder at vulnerability-db-manifest.json. The post-agent
snapshot publisher writes that manifest exclusively and fails closed on
conflicting content, which made every goal-plan attempt die with
ARTIFACT_CONFLICT once its plan validated (reproduced across all three smoke
benchmark targets in a live Modal sandbox). Exclude the snapshot manifest
from canonical-empty preparation exactly like the runtime-owned workspace
patch outputs, and pin the exclusion with a generated-workflow test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): make goal-plan capability copies and placeholder retention mechanical
The remaining goal-plan agent failures were content slips on one target:
paraphrased capability rationale/evidence in applicability checks and
coverage-gap goal_prompts with the {{threat-model:coverage-gap}} placeholder
substituted away. Tell the agent to build checks mechanically from
threat-model.json and to keep every replacement key's literal {{...}} braces
in goal_prompt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): keep coverage-gap placeholder wording template-safe
The previous commit's literal {{<key>}} example was itself parsed as a
prompt template variable and failed catalog validation. Reword to describe
the double-brace form; the exact {{threat-model:coverage-gap}} text is an
item-scoped placeholder the validator accepts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): drop literal coverage-gap braces the topology validator rejects
The topology artifact-handoff validator parses double-brace sequences in
node prompts as template variables and does not recognize
threat-model:coverage-gap outside dynamic-goal contexts. Describe the
brace-wrapped key without writing the literal placeholder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(references): assert the public database stays anonymous under a credential
The allowlist negative case named a non-existent monad-developers repository,
which both carried the pre-transfer owner and asserted nothing about a
reference Ultrafuzz actually ships. Use the real pinned public database
instead, so the case proves the shipped vulnerability-database reference is
fetched anonymously even while a private-reference token is configured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(references): re-pin the vulnerability database to the current corpus release
The shipped pin still referenced fbf00e99 (18 records, 21 capabilities,
aggregate c0ed7d23). Upstream has since published two further commits, and
the corpus now carries 23 records and 28 capabilities under aggregate
62b7fb80, including the new classes/lending category.
Re-pin to e46c0e47, the current immutable head. Its database.yml,
capabilities.yml and catalog.json are byte-identical to 153e14cc; the
intervening commit only changed upstream ingestion tooling, which Ultrafuzz
does not consume. Verified by a fresh unauthenticated fetch with GitHub
tokens, SSH agent and credential helpers disabled.
Regenerate the documented digest table and catalog statistics to match, and
correct the test comment that still described the repository as private.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(deps): reconcile the lockfile with Smithers 0.32.0 after rebase
The branch carried two lockfile edits against Smithers 0.31.0 that
conflicted with #276's 0.32.0 upgrade on main. Resolve by taking main's
0.32.0 lockfile and regenerating the workspace links from the package
manifests, which restores the @ultrafuzz/prompts link into
packages/artifacts. pnpm install --frozen-lockfile now succeeds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(evals): keep the smoke lane free of dynamic goal fanout
The smoke lane declares disable_dynamic_strategies, but that flag only ever
pruned dynamic-strategy-generator, which the dedicated smoke graph does not
contain. This branch adds threat-goals and class-goals to
benchmarks/smoke-benchmark.yml, and those expand one child per planned threat
and per applicable class, so the lane would have run unbounded dynamic work
while still reporting dynamic strategies as disabled. #277's smoke matrix
requires that guarantee to hold.
Name the goal-fanout node IDs, add them to the dynamic exclusion set, and
apply them on the smoke lane. Production-topology exclusions still cannot be
passed wholesale to the smoke graph, because its invariant, differential and
dynamic-strategy nodes are absent and would raise unknown-node errors.
Add a regression test that reads the smoke topology and asserts the lane
prunes every dynamic node it declares, so a future dynamic node added to the
smoke graph fails the suite instead of silently running.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(modal): keep curated private lanes off the new threat-model nodes
A private benchmark lane isolates part of the production topology by naming
the nodes it wants pruned. The invariant-only Aave lane does this. Every such
list was written before threat-model, goal-plan, goal-roaming, threat-goals,
class-goals and reference-vulnerability-database existed, and the first four
land in the pre-existing setup and goals groups, so no group-level filter
catches them either. Merging this branch would therefore have made the next
invariant-only run quietly execute a threat model and unbounded goal hunters,
invalidating the comparison it exists to produce.
Publish the default-on node IDs from @ultrafuzz/evals and prune them from any
private lane that curates an exclusion list, with
include_threat_model_goal_fanout to opt back in. An empty list still means
"run the whole topology", so uncurated private runs are unchanged.
Also prune goal-plan on the smoke lane. The fanout is its only dependent, so
excluding just the fanout leaves it terminal and strands the smoke report
prompt's artifact_path citation of it; the lane could not plan at all. A new
runtime suite expands the real pruned graph to prove both directions.
Keep the topology-derived tests honest: the published ID list is checked
against the shipped topology, so a future default-on node fails the suite
instead of silently widening every curated lane.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(evals): prune goal-plan wherever the goal fanout is pruned
Excluding threat-goals and class-goals on their own leaves goal-plan with no
dependent, which makes it terminal, and strands the report prompts'
artifact_path citation of it, which requires an ancestor. Either way the graph
stops validating, so any lane that drops the fanout has to drop goal-plan too.
Fold that into BENCHMARK_GOAL_FANOUT_EXCLUDED_NODE_IDS and build the smoke,
production-dynamic and curated-lane sets from it, so all three stay valid
against the graph they are applied to.
Move the pruning coverage into packages/runtime/test/topology-transform.test.ts
next to the existing exclusion tests, in the node:test style that package's
runner actually compiles and executes, and extend its production-topology case
with the same three IDs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(evals): keep the benchmark smoke lane exactly as published
Three independent reviews of 7e2687d found that pruning the goal fanout at
lane level was not enough to keep the smoke lane's guarantee.
Per-run agent concurrency is max_parallel_targets, which is 4, so the pruned
12-node graph still serialized into six waves: 6 x 2 x 1200s = 14,400s against
a 15,000s row watchdog that packages/modal/src/config.ts caps at 15,000. The
margin fell from 5,400s to 600s and could not be raised without a schema
change. Keeping threat-model also kept requiresVulnerabilityDatabase true, so
every smoke run gained a plan-time corpus fetch and the exact-cover capability
gate in verifyThreatModelVulnerabilityDatabaseCapabilities -- at medium
reasoning with two attempts. In a model matrix that gate fails a row outright,
so a weaker entrant publishes no observation at all and the comparison is
silently biased. Passing any non-empty excluded_node_ids also moves
benchmark_execution, and with it the execution-policy and cohort fingerprints
that #277's rows must share with the already-published smoke observations.
Restore benchmarks/smoke-benchmark.yml byte-for-byte to main, drop the unused
smoke threat-model prompt, and take the smoke lane back to an empty exclusion
list. The lane keeps its guarantee structurally, the way it always did: the
dedicated graph simply does not declare invariant, differential,
dynamic-strategy or goal-fanout nodes. A test now asserts that rather than
asserting a pruning list.
Keep the prompt-only smoke improvements that carry no node dependency: the
evidence-path hardening and the source_nodes provenance bullets.
The full lane is untouched and still exercises the whole threat-model
workstream, and the production topology keeps every default-on node.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(runtime): rebuild dedupe provenance from the lifecycle ledger when syncing
Three independent reviews of 0d3e5c2 found two real regressions in the new
finding-provenance machinery, both introduced by this branch.
dependencyFindingProvenanceForTask rebuilt the source expectations without the
lifecycle ledger, so buildFindingSourceExpectations returned early and produced
one single-source expectation per upstream finding. A dedupe root's source_nodes
is the union of everything it merged, which matches none of those, so
finalizeTerminalTask re-validated a node the authoritative in-workflow gate had
already passed and marked it failed with
task-output-validation-failure. That fires on any run with a genuine merge, and
it is reachable from ultrafuzz inspect, ultrafuzz status, and the dashboard, so
it needs no cloud execution. It also attributed upstream findings to the
dependency attempt ID rather than the producing node, so a generated dynamic
child could never match its own ledger record. Load the ledger and resolve
producer node IDs through tasksByAttempt, mirroring the generated template,
which is the authoritative implementation of the same contract. Stop creating
artifact directories from this read-only path while here.
The smoke prompts described a lifecycle ledger the new gate rejects: entries
need node_id and finding_id and must cover every upstream finding exactly once,
which only the production prompt said. Bring the smoke prompt up to the same
contract and tell the smoke report to carry finding identity through, since
provenance is matched on it.
Also correct a comment claiming the goal fanout exists in the smoke graph, a
reference example naming a node and prompt that do not exist, and the changelog
description of the roaming goal, which is a static node rather than an
expansion. Document the deliberate asymmetry in the private-reference credential
contract and the one-off lineage discontinuity a curated private lane sees.
Add a production-topology test proving the exact set every curated lane prunes
still plans, and that no dynamic node, goals-group node, or vulnerability
database reference survives it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): bind a dedupe root to its lifecycle ledger record
Round-three review found the remaining half of the provenance contract was
unstated. buildFindingSourceExpectations derives one expectation per upstream
finding plus one per ledger record, and matches an emitted finding against
whichever expectation shares an identity key with it. Only the ledger-record
expectation carries a multi-node union, so a root that merged findings from two
different nodes resolves to it solely through a shared dedupe_key. Neither
prompt told the model to write that key onto the retained finding, so any
genuine cross-lane merge failed with "source_nodes does not preserve the exact
dependency discovery-source union", on both the production and smoke lanes.
Say so in both prompts, and give the production prompt the exact-coverage rule
the smoke prompt already gained: every upstream finding must appear exactly once
across all source_artifacts, including ones kept only as a duplicate or family
variant. The gate enforces that on both lanes; only the smoke prompt said it.
Make the sync-side ledger reader fail the way the template does. Swallowing a
parse error re-entered the naive expectation set the previous commit removed and
reported a corrupt required artifact as a findings-contract violation. Reject a
symlinked ledger or dependency findings file rather than following it out of the
run root, matching resolveRegularArtifactFile.
Pin the surviving node set in the curated-lane completeness test. Its shape
checks could not see a default-on node added to a pre-existing group, which is
exactly where threat-model and goal-plan live, so the guarantee it advertised
was not the one it enforced.
Add the regression test for the expectation set itself, and make the reference
example match the shipped threat-goals node rather than a partial copy of it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): give dedupe the full producer list its ledger is checked against
Round-four review found the last gap in the new provenance gate. Ledger
coverage is checked against every dependency findings artifact the runtime
read, but the production dedupe prompt named only 5 of the 29 nodes that
declare a findings.json output, and told the model never to guess a path. The
other 24 -- boundary-tests, the invariant chain, the differential lanes and the
rest -- were unreachable from the prompt, so any run in which one of them
reported a finding failed dedupe-findings deterministically with "finding
lifecycle ledger omitted dependency findings", then cascaded to triage,
severity-classification and final-report. The smoke lane was immune only
because it enumerates all four of its producers, which is why three earlier
rounds and CI missed it.
Enumerate every producer. Per-line rendering means a curated lane that prunes a
node also drops its line, so an invariant-only lane sees exactly the lanes it
still runs rather than an instruction to account for artifacts that no longer
exist.
State the two rules the gate enforces that the prompt left implicit: node_id is
copied out of the finding rather than taken from the group template or the
storage directory in its path, and a finding judged unsupported gets its own
ledger record instead of being attached to an unrelated root, which would put a
non-contributing node in that root's source union.
Complete the sync/template mirror: a missing ledger now names itself instead of
degrading to the naive expectation set and reporting a missing required
artifact as a findings-contract violation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): state the source-union equality rule the review lane is held to
Round-five review audited every throw site in the new provenance gate against
what the prompts actually instruct, and found the production dedupe prompt
still missing the rule the smoke prompt already had: a retained root's
source_nodes must equal exactly the node_id set of its own ledger record's
source_artifacts. Two of its sentences pushed the other way -- "preserve the
relevant source union on family variants" invites a subset, since a variant's
artifacts belong to the root's record, and "include all corroborating node IDs"
invites a superset from related findings that were deliberately not merged.
Either fails the node with the same opaque exact-union error.
State the rule, and drop the two sentences that contradict it.
Also close the smaller gaps that audit surfaced: every ledger record needs its
own dedupe_key including the coverage-only ones, no two records may share one,
triage and severity-classification must carry id and dedupe_key through rather
than assigning a fresh stable id, and the restart-handling short circuit now
lists finding-lifecycle-ledger.json so a resumed attempt cannot declare itself
finished without the one output whose absence hard-fails the node.
Pin the producer enumeration to the topology with a test, since nothing tied
them together and the list had already drifted once. Document the ledger in the
review guide, and make the goal-fanout header self-explaining when a curated
lane prunes its paths away.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): name the finding fields the review contracts now require
This branch moves deduped-findings.json, triaged-findings.json and
severity-classified-findings.json from ultrafuzz/json-array@1 to
ultrafuzz/findings@1. That subjects all three to the full normalized-finding
field check, not just the source-union rule, and none of the three prompts
listed those fields. severity-classification was the worst case: `summary` is
required, the prompt never mentioned it, and its four worked examples omit it,
so a model following the only concrete shape in the file fails the node on
every attempt. dedupe-findings was missing status, severity_guess and
confidence.
Name the required fields in all three, and say that the severity examples show
only the fields that stage adds rather than the whole object.
Pin it with a test, which caught the dedupe-findings gap that reading had
missed. Also close the two remaining review items: the restart short circuit
now validates the ledger's shape instead of only its existence, so a resumed
attempt with a truncated or incomplete ledger rebuilds rather than declaring
itself finished; and the producer-enumeration pin now covers the smoke lane
too, which had the same drift hazard unguarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(prompts): pin every required finding field, not five of seven
The field-contract test listed the five fields normalizeFinding hard-requires
but omitted schema_version and id, which the finding schema also requires. All
three review prompts already name all seven, so this changes no behaviour; it
closes the hole where a later edit could drop one of the two unpinned mentions
and leave the test green while the node fails its contract on every attempt --
the exact failure the test was written to prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Antonio Viggiano <3029017+aviggiano@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-192-168-1-2.ec2.internal>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
* feat(goal-plan): record expected dynamic-child cardinality and goal lanes at planning time (#373)
Implements #364 option (a): the planner writes its dynamic-fanout expectation down, and the
eval side compares against the run without ever recomputing the rule. Expected and observed
then come from different components, so a planner that under-expands is caught.
`goal-plan.json` gains `expected_child_count`, `threat_count`, `applicable_class_count`,
`max_dynamic_nodes`, and `goal_lanes` (each lane named, with the node IDs it owns). They are
recorded deterministically by the `goal-plan` node's postprocessor from the plan the agent
wrote plus the run's resolved `run.max_dynamic_nodes`, before the plan is validated. A value
the agent wrote that disagrees with its own plan fails closed rather than being silently
corrected, and a plan expecting more children than `max_dynamic_nodes` allows fails as a plan.
The cardinality rule -- `threats + applicable classes`, with the fixed roaming node as a static
lane rather than a dynamic child -- is computed by exactly one function, `goalPlanExpansionFacts`.
That now includes the pre-existing `counts` block, which computed the same arithmetic a second
time twenty lines away: `counts.threats`, `counts.applicable_classes` and `counts.dynamic_goals`
are checked against those facts instead of restating `threat_goals.length + class_goals.length`,
so `threat_count`, `applicable_class_count` and `expected_child_count` are aliases that cannot
drift from them. `max_dynamic_nodes` and `goal_lanes` are the only genuinely new information.
Lane granularity is recorded rather than left implicit: a lane is one goal, so `lane_id` is a
threat or class ID and a run has `threats + classes + 1` lanes. The alternative -- one lane per
dynamic group, three per run -- aggregates more cheaply but cannot say which goal was expensive
or which one failed. Group totals stay derivable by summing one `kind`; the reverse is not.
Per-lane cost is now joined on a key that exists. `UsageLedgerEntry` gains an optional `node_id`,
the identity `state.json` keys a node under, resolved by the runtime from its own task table --
a usage event names the workflow task (`node:<attempt>`), not the state node, and only the task
table knows that mapping. `attempt_id` could not serve: it is `sha256(workflow run, node,
iteration, attempt)` and carries no node field, so the previous join matched nothing on every
real ledger while still reporting `lane_cost_evidence: complete`. The field is optional because
`usage.jsonl` is durable: ledgers written before it existed still parse, and a run resumed across
the upgrade does not fail its replay when a recorded entry gains the key.
`evalRunExpansion` now reports `plan`, `expected_vs_actual`, and per-lane `goal_lanes`. It reads
the planner's numbers and compares; a mismatch is kept as data with the size and sign of the
disagreement, never dropped. Lanes fall out of per-node status for failures, the usage ledger for
tokens and cost, and node timestamps for wall-clock, so metrics 5 and 6 need no new telemetry.
Each lane carries `cost_evidence` and `usage_matched_node_count`, so a lane that genuinely spent
nothing (zero, complete) is distinguishable from one whose join never landed (null, unavailable
with a reason: no `node_id` in the ledger, no entry naming this lane's nodes, or no node observed
for the lane at all). `EvalRowScore.expansion` stays optional so historical scores still read.
The eval test fixture no longer hand-writes `usage.jsonl`. It calls `createUsageLedgerEntry`, the
same function the runtime calls; the old fixture invented a `schema_version` and an `attempt_id`
shape the real writer cannot produce, which is how a structurally dead join kept a green test.
Dynamic child node states carry `provenance.source_node_id`, the one key both sides must agree on
and the key #365's reader already looks for. The nested camelCase `dynamic` record stays alongside
it for the expansion key, item digest, and manifest path.
Verified: `pnpm -r build` 0, `pnpm -r test` 0, `pnpm -r --workspace-concurrency=1 typecheck` 0,
`pnpm -w format:check` 0, `pnpm -w lint` 0. No benchmark run was dispatched.
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(evals): v0.1.0 threat-model release-gate lane (#351)
* feat(evals): add the v0.1.0 threat-model release-gate benchmark lane (#183)
The three pinned Ultrafuzz-bench targets have never seen the production
topology. The smoke lane substitutes the reduced
`benchmarks/smoke-benchmark.yml` graph, and the full lane is bound to the
EVMBench cohort, so the `threat-model`, `goal-plan`, `threat-goals`, and
`class-goals` nodes #188 added run against no real pinned protocol. This
adds a third lane rather than editing the smoke graph, whose empty
`excluded_node_ids` is what keeps the 99 published observations
comparable.
The `threat-model` lane pairs the Ultrafuzz-bench cohort with the
unmodified production graph: no `topology` override, all three `disable_*`
flags off, and an empty exclusion list. Two guards keep that honest.
`loadBenchmarkLanesManifest` rejects a checked-in gate lane that turns any
production work off, and `adaptBenchmarkManifestToEvalSuite` refuses to
compile a suite whose exclusions would remove a node the gate exists to
exercise, so a lane edit cannot quietly produce a green run that proves
nothing.
The gate takes its GPT-5.6 Luna/high runner from `benchmarks/lanes.json`
instead of the dispatch form -- the workflow leaves `BENCHMARK_MODELS_JSON`
unset for this lane -- so no dispatch-form value can retune its model
identity. It runs three sandbox rows with the production graph's eight-way
in-workflow concurrency, which is what a large dynamic ready queue needs to
be scheduled rather than serialized, and takes the 15,000-second Modal
watchdog ceiling. Beyond the report bundle it retains `THREAT_MODEL.md`,
`threat-model.json`, `goal-plan.json`, and `vulnerability-db-manifest.json`
from every row, because its assertions are deliberately structural and the
prose has to stay reviewable.
Gate results are not published into `benchmarks/history.json`. Publication
is a longitudinal claim about one comparable series, and this lane runs a
different topology and execution policy against the same targets. The
producer now declares its dispatched lane as a `Benchmark lane <name>` step
-- the `workflow_run` payload carries no dispatch inputs, and reading a step
name is the introspection the producer's own recovery job already does --
and the qualifier declines any lane that is not the one its trigger is
allowed to have produced, instead of assuming every dispatch is the
EVMBench matrix.
Tests assert the generated dynamic node IDs against the real expansion of
the real topology templates: a contract-valid `ultrafuzz/goal-plan@1`
document expanded through `planDynamicExpansion` must reproduce the plan's
own `node_id` for every threat and class goal. That spelling is
`dynamic:threat:<id>` / `dynamic:class:<id>`, which the topology templates,
the goal-plan contract, the planner prompt, and the changelog all pin;
#185's `dynamic.threat.…` sketch predates that decision and is not what
#188 shipped.
Nothing dispatches the lane. Selecting `benchmark_lane: threat-model`
remains a maintainer decision with a real cost attached.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(evals): let the post-eval checkpoint accept every dispatchable lane (#183)
The public eval diagnostics schema still enumerated `["smoke", "full"]` after
the `threat-model` release gate became a third lane. The worker builds that
document from `public_benchmark.lane` in `runAndCheckpointPublicEvalDiagnostics`
only AFTER `runEval()` returns, so a `threat-model` dispatch would spend the
whole cohort's model budget and then die at the post-eval-pre-score checkpoint
with a Zod `invalid_value` on `lane` -- surfacing as a
`PublicEvalDiagnosticsBuildError` and producing no diagnostics, no bundle, and
no gate evidence at all.
Derive the enum from `BENCHMARK_LANE_NAMES`, which is already the source for the
Modal config, the public bundle, and the history observation schemas.
The regression test builds the checkpoint for every lane in
`BENCHMARK_LANE_NAMES` through the real `createPublicEvalDiagnostics` path
rather than asserting the enum, so a future lane that forgets a schema fails
here instead of on a paid run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(evals): wire the release-gate lane to the landed retention and observability
Merges release/v0.1.0, which now carries B1 (#363) and B2 (#365), and removes
the duplicated artifact list: PUBLIC_OPTIONAL_ROW_ARTIFACTS now derives from
BENCHMARK_THREAT_MODEL_RETAINED_ARTIFACTS instead of repeating the same four
names as a literal in a second package, where the two could drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(evals): repair threat-model lane on the release contract
Populate the post-#373 goal-plan cardinality and lane fields from the shared authority, using the same 512-node limit exercised by dynamic expansion. Remove the reporter-only artifact include spread; the public worker remains the real retention path.
* ci(evals): enforce threat-model structural gate
---------
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Merge main into release/v0.1.0 after threat-model gate
# Conflicts:
# .github/workflows/eval-benchmarks.yml
# CHANGELOG.md
# packages/artifacts/src/findings-schema.ts
# packages/artifacts/src/findings.ts
# packages/artifacts/test/artifacts.test.ts
# packages/artifacts/test/schema.test.ts
# packages/modal/src/node-provider.ts
# packages/modal/src/node-worker.ts
# packages/modal/test/ci-config.test.ts
# packages/runtime/src/index.ts
# scripts/ci/qualify-modal-benchmark-publication.mjs
# scripts/ci/qualify-modal-benchmark-publication.test.ts
* fix(ci): reconcile release benchmark dispatch lanes
* Merge trusted benchmark recovery prerequisite into release lane
# Conflicts:
# packages/modal/test/ci-config.test.ts
* fix(ci): authenticate threat-model benchmark launches
* Seal workflow execution controls and harden upgrades (#428)
* fix(runtime): seal workflow execution controls
* fix(runtime): recover stale snapshot publications
* fix(runtime): persist detached task control paths
* fix(runtime): guide sealed workflow upgrades
* fix(runtime): keep sealed control paths private
* fix(runtime): publish stock adapter upgrades atomically
* test(cli): synchronize status watch samples
* fix(runtime): harden stock adapter publication
* style(cli): format status watch synchronization
* fix(runtime): recover torn adapter marker
* Harden init publication and file writes
---------
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
* Preserve retry cost, timing, and history provenance (#429)
* fix(evals): retain retry accounting and timing (#427)
* fix(evals): preserve partial history metrics (#427)
* fix(evals): harden history chart provenance (#427)
* test(runtime): use sealed fork evidence for retry history
---------
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
* chore(modal): remove stale release-sync imports
* fix(runtime): admit sealed dynamic control extensions
* fix(references): support vulnerability database v0.0.2
* chore(release): complete v0.1.0 readiness gates
* test(dashboard): make dynamic flow fixture hermetic
* feat(config): add fuzz-only audit profile (#565)
Preserve the established v0.0.x fuzzing graph as a packaged v0.1.0 profile while excluding the vulnerability database, threat model, and goal fanout.\n\nCloses #559
* fix(config): reserve the default audit profile (#580)
* fix(config): reserve default audit profile
* docs: clarify audit profile reporting cohorts
* fix(config): enforce unmodified default profile
* docs: trim audit profile migration note
* refactor(prompts): move agent preamble to MDX (#652)
* fix(runtime): route custom Codex provider preflight (#654)
* fix(runtime): route custom Codex provider preflight
* fix(runtime): harden custom Codex provider routing
* docs: begin provider-agnostic harness research
* docs: separate provider, harness, and model for #653
Publish the provider-agnostic coding harness research and the reviewed
architecture plan, and make both consistent with the decisions recorded on
#653.
Provider, harness, and model stay three independent choices throughout. The
first-party defaults are stated explicitly: OpenAI is reached through the Codex
CLI harness, Anthropic through the Claude Code harness, and DeepSeek V4 through
`deepseek-ai/deepseek-harness` (`dsh`) — final-text-only, and only once it is
qualified. OpenRouter is a gateway with no harness of its own, so it gets a
provider-neutral one: Pi first, OpenCode second. Codex + OpenRouter and
Claude Code + DeepSeek stay supported compatibility pairings and are never the
default.
Narrow two claims the sources do not support:
- Qualification requires a pinned real CLI against the real provider. Fake CLIs
are for simple unit tests only; they never satisfy a gate.
- Drop the proposed `#653-H` session-log reader. `dsh`'s
`session.jsonl.zstd` is undocumented and self-declares `"version": 0` on a
prerelease whose upstream accepts neither issues nor pull requests, so
Ultrafuzz parses nothing from it. The measurement stays as evidence; the
forward-looking parser is replaced by a small real-`dsh` smoke profile that
lives inside the adapter issue (#661) — headless final text plus exit status,
one filesystem/shell artifact check in a disposable worktree, a version pin
asserted at preflight, and an explicit final-text-only fallback that warns
rather than failing the run.
Also record that proposals A-G were filed as #658-#664, and link the follow-up
work from the research page's next-work list.
Refs #653, #658, #659, #660, #661, #662, #663, #664
* docs: apply harness research review and guard both pages
Applies the completed Opus review of the provider/harness documentation for
#653 and makes the two explanation pages consistent with each other and with
the real state of the work.
Plan (docs/explanation/provider-harness-plan.html):
- Masthead and footer now record the real state — #658-#664 filed, PR #665
open, reviewed and published for implementation — instead of claiming no
issues and no PR exist.
- The §1 verdict marks Pi and DeepSeek Harness as proposed defaults pending
real-provider qualification, and states that Codex CLI remains the shipped
OpenRouter default in the interim. The §4 gateway card says the same.
- The §3.2 candidate matrix gains an evidence-provenance column separating
real execution, --help/--version inspection, and upstream documentation, and
its Events/Sessions/Isolation columns now use the capability-contract
spellings.
- The §5 capability boxes use the normative vocabulary
(native-sandbox | external-sandbox-required, filesystem / shell) identical to
the Markdown interface, with a callout binding #658 to that spelling.
- The §7 risk register names five maintained harnesses instead of four, calls
out Kimi separately as an existing binding, and uses one pill class for the
low severity rows.
- §4 subsection headings drop from h4 to h3, and each proposal gains an id so
the #653-D references link to proposal D rather than the section.
Research (docs/explanation/provider-harness-research.md):
- The status block reconciles with the filed issues and the open PR, and points
at the HTML plan as a download-and-open-in-a-browser companion.
- The comparison matrix is now readable on GitHub: a compact matrix plus a
per-candidate subsection carrying the long detail, with an explicit
provenance legend.
- The capability contract is stated as normative and bound to #658.
- Every DeepSeek Harness source URL is pinned to commit
99f6f02fecdb7dff40c3fbc9470f5907c29f74ca rather than master.
- Headings use Title Case and the H1 matches the explanation index.
Index (docs/explanation/index.md) explains how to read the pair: the HTML plan
must be downloaded and opened in a browser, the Markdown page is the
GitHub-readable companion, and §7 and §9 are unique to the plan.
scripts/docs-check.mjs now requires both explanation pages, so the real docs
check fails if either normative document is deleted or renamed.
Part of #653.
* docs: number the gates and align the contract spelling for #653
Apply round-1 review findings on the provider/harness pages.
- Number the Markdown conformance gates G1-G10 to match HTML plan §8, and
add the two generic gates that only existed in the HTML (G7 state
isolation, G8 sandbox boundary), so the G-numbers cited by #659, #661,
#662, #663 resolve on the GitHub-readable page.
- Stop asserting that #658 implements the capability spelling verbatim:
#658's summary block abbreviates `fs` and `native`/`external`. Both
pages now state the normative spelling and record the issue text as
shorthand to be corrected, rather than claiming an alignment that does
not exist.
- Add a "Credential and State Rules" subsection mirroring HTML §5.2, so
the argv rule, run-scoped state root, invocation-cwd rule, and
harness-injected-environment caveat appear on the Markdown page too.
- Use `harness = "dsh"` consistently in the Markdown config example, so
it matches HTML §5.1, "Next Work" item 4, and #661.
- Record that neither dsh nor OpenCode has a measured reasoning surface,
drop `reasoning = "high"` from the dsh example, and state that the
validator rejects `reasoning` on a binding declaring no levels.
- Disclose that PR #654's real-codex assertions are guarded by a
`codex --version` probe and are not exercised in CI.
- Correct the HTML-unique section claim on both the research page and the
explanation index, unify "final-text nodes" phrasing, match the index
link text to the HTML page's H1, and restore alphabetical order in
docs-check.mjs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: correct the dsh reasoning surface and restore page parity for #653
Round-2 review fixes for PR #665.
- dsh does have a reasoning surface: `dsh-llm-deepseek` declares `thinking`
and `reasoningEffort` (off/low/high/max) at 99f6f02, with an omitted effort
resolving to `high`. It is a settings field with no CLI flag and no schema
default, which is why it never appeared in `--dump-default-config`. The
binding declares an empty `reasoningLevels` because nothing measured it,
not because the harness lacks it; #661 owns the measurement.
- Mirror the dsh and OpenCode reasoning findings into the HTML plan (§3.1
row, §3.2 callout, §5 contract box, and the drafted D/E issue bodies) so
the "everything appears on both pages" claim holds again.
- Narrow the matrix-column claim: the columns are drawn from the contract
vocabulary, composite cells name a contract value plus an advertised extra,
and the direct provider-API row is not a harness binding.
- Reconcile the measured Codex 0.147.0 / Claude Code 2.1.233 evidence against
the versions the Modal worker image actually pins (0.146.0 and 2.1.207).
- Grandfather the pairings shipping today out of the G1-G10 default rule and
record the qualification debt explicitly.
- Make `config_dir` a base path the launcher expands per run, so the worked
example stops contradicting the run-scoped state-root rule.
- Use one name for the numbered gates ("qualification gates") across both
pages and the explanation index; reserve "conformance suite" for test code.
- Reframe §2 around what "DeepSeek Code" refers to and anchor the withdrawn
draft to branch commit bb831d61.
- Title-case the HTML h1 to match the index link text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: reconcile the OpenRouter default and page content map for #653
Applies the remaining review findings to the provider/harness documentation
for #653. No runtime change.
- Reconcile Codex + OpenRouter throughout the pairing policy: it is the
grandfathered shipped OpenRouter default today, while a compatibility
pairing is never the recommended default for a new project. "Shipped
default" and "recommended default" are now defined against each other, and
Codex + OpenRouter appears in both grandfathered-pairing lists alongside
Codex + OpenAI, Claude Code + Anthropic, Claude Code + DeepSeek, and Kimi.
- Replace the false cross-page parity claim with an accurate content map in
the Markdown introduction and the explanation index: HTML-only material
(§1, §7, §9, the §2 correction table, and the packaging, retry,
Modal-portability, and maintenance-posture evidence rows) and Markdown-only
material (TOML boundary, TypeScript interfaces, legacy agent mapping,
dsh-specific gate items, and the full pinned source list).
- State that error classification, preflight behavior, and retry semantics
were not measured for any CLI-inspection candidate, with explicit
ownership: #659 for Pi, #662 for OpenCode, and #663 for Codex and Claude
Code re-measured at the versions the shipped Modal image pins.
- Correct the malformed usage JSON in the HTML plan so inputTokens and
outputTokens carry colons, matching the Markdown page.
- Update the substantive revision date to 2026-08-19 in both pages,
including the plan masthead and footer.
* docs: make reasoning levels, preflight, and pin citations consistent for #653
Applies round-1 review findings to both provider/harness pages.
- Reasoning: the contract says `reasoningLevels` records only levels a real
request exercised, but the worked TOML set `reasoning = "high"` on a Pi
binding whose provenance is CLI inspection. Keep the rule and make every
binding obey it: drop the key from the example, record Codex's
`model_reasoning_effort` and Claude Code's `--effort` surfaces with their
in-repo adapters, and replace the "two candidates" claim with a table showing
all five are empty today and which issue measures each.
- Preflight: G2 now owns authenticated preflight behavior including the 401
path, since G5/G6 cover events and resume only.
- Scope the unmeasured dimensions by the real gap (no row hit a real paid
provider) rather than by the CLI-inspection provenance label.
- Cross-page content map: only Modal portability is HTML-exclusive; packaging,
retry, and maintenance posture are on both pages. Same wording in index.md.
- Claude Code + DeepSeek gets the same shipped/recommended vocabulary already
applied to Codex + OpenRouter, and picks up the licensing and installation
detail every other candidate records.
- Cite `CODEX_CLI_VERSION` and `modalSecurityToolchainCommands()` instead of
raw line numbers, and guard both pins in docs-check so moving one in the
Modal image without updating the docs fails `pnpm docs:check`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: resolve the #663 ownership pointer and contract gaps for #653
Round-2 review fixes for the provider/harness doc pair:
- #663's body scopes only the Modal image capability checks, so the pages
no longer assert it already owns the version reconciliation, the Codex and
Claude Code error-classification/preflight/retry measurements, or their
reasoning levels. Both pages now state the issue must be extended and that
the doc spelling governs, matching the device already used for #658.
- Rescope the Constraints bullet from "every CLI-inspection candidate" to
every candidate including the two real-run rows.
- Stop overloading "pinned": the provenance definition now says the CLI was
installed at an exact recorded version, leaving "pinned" to mean the
shipped Modal image version defined in the gates section.
- Fix the conformance-suite cross-reference from §6 to §7.
- Give `cloudPortable` a stated owner and record local/Modal portability for
Pi and OpenCode, so the contract field resolves for every candidate.
- Title Case the new index.md heading, match the two TOC labels to their
headings, give all five candidate subsections a parallel Reasoning bullet,
and note that #659's enumerated G2 criterion predates G2's widening.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: correct the OpenCode reasoning surface and gate statuses for #653
Applies the outstanding review findings on the provider/harness pages.
- OpenCode does expose a reasoning surface: real `opencode-ai@1.18.18
run --help` advertises `--variant` as a provider-specific reasoning
effort. Record it as CLI inspection, keep `reasoningLevels` empty
pending #662, and drop every claim that OpenCode has no such surface —
in the Markdown detail, its reasoning table, the mirrored HTML
callout, and #653-E's drafted scope.
- Stop claiming ownership is explicit for Codex and Claude Code while
also calling the work unowned. Ownership is explicit only for Pi
(#659) and OpenCode (#662); #663 is nominated, not assigned, until its
acceptance criteria are extended. Both pages now say that identically.
- Downgrade the HTML G2 dsh status from `passed locally` to a warning:
the canary passed locally but authenticated preflight is unmeasured,
and G2 now covers both.
- Title-case the Markdown `Credential And State Rules` heading.
- Narrow the dsh default claim: dsh's shipped composition sets
`agent-default-model` to `deepseek-official` / `deepseek-v4-flash`,
which is not a `shipped default` in the sense the pairing policy
reserves for Ultrafuzz provider/harness selection.
- Repair the HTML conformance cross-reference so it names the
`conformance suite` of §9 and the `conformance test` of §7.
- Replace `shipped default by inertia` with `by grandfathering` in
Phase 5.
- Correct the `scripts/docs-check.mjs` pin-guard comment to
`[pin, ...filesThatMustContainIt]` and explain why the interpolated
Codex pin needs two entries while the Claude Code pin needs one.
- Add `Local and Modal results agree for every approved harness` to the
HTML §9-F acceptance list, matching filed issue #663.
Refs #653.
* docs: fix the reasoning-surface count and make cloudPortable expressible for #653
Apply round-1 review findings on the provider/harness pages.
Majors:
- The §3.2 reasoning callout in the HTML plan still opened with "Four
candidates do have a known surface" while enumerating five and closing
with "All five bindings" — surviving residue of the pre-correction claim
that OpenCode had no surface. It now reads "All five candidates", agreeing
with the enumeration, with the paragraph's own conclusion, and with
provider-harness-research.md.
- `HarnessCapabilities.cloudPortable` was a required `boolean` while the page
states three times that no candidate declares a value. A required boolean
cannot express "unmeasured", so #658 could not implement the contract
verbatim as instructed. It is now optional, with the absent case defined
the way an empty `reasoningLevels` already was: the validator refuses cloud
scheduling rather than reading absence as `false`. `cloudPortable` also
joins the §5 layer-3 capability box list, which the adjacent callout calls
out as matching the interface.
Minors:
- "the two converge at Phase 5" was unresolvable from the Markdown page,
which never defines the phases. Next Work entries now carry their phase
numbers and the reference points at #664.
- The cross-page content map claimed the legacy `agent = "…"` forward mapping
was Markdown-only; the plan carries the `DeepSeekAgent` half in its §2 table
and §9-C scope. Narrowed in both index.md and the research page.
- Drafted §9-A and §9-C acceptance lists were missing criteria their filed
counterparts (#658, #660) carry: the event-class-below-node-requirement
failure case, and the compatibility-pairing migration.
- The reasoning table's "Owner" column asserted #663 ownership that the prose
two paragraphs below denies; the two rows are now marked "(nominated)".
- Issue-tracker errata across both pages were phrased as standing
instructions ("#663's body must be extended", "#658's body should be
corrected") that go stale silently the moment an issue is edited. They are
now durable conditionals and precedence rules — ownership attaches when an
issue's acceptance criteria name the work, and the docs govern spelling
wherever the two disagree — so the pages stay correct either way.
Nit: "Codex and Pi, whose surfaces are known but were only inspected" misused
the provenance vocabulary; Codex has no recorded CLI inspection. The two
provenances are now distinguished.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: apply the page's own ownership rule to #661 and #662 for #653
Round-2 review fixes for the provider/harness pages:
- The reasoning-measurement rows for dsh (#661) and OpenCode (#662) are
marked `(nominated)` like the two #663 rows. Neither filed issue names a
reasoning, variant, effort, or thinking measurement, so the page's own rule
— ownership counts only when the acceptance criteria name it — leaves #659
as the single tracked row.
- dsh joins the error-classification, authenticated-preflight, and retry
ownership accounting: #661 is nominated there too, so all five harness
candidates are accounted for instead of four.
- `reasoningLevels` becomes optional, matching `cloudPortable`: absent means
unmeasured, `[]` means measured and no level accepted. That is what §9-E's
"or the binding documents that it accepts none" needs in order to pass, and
the acceptance wording now names the spelling.
- §9's "Filed as #658–#664" callout records which §9 items post-date the
filing and are therefore drafts rather than filed text.
- The masthead and footer no longer assert that PR #665 is open, since the
page only exists on a release branch once it merges.
- The harness-pin guard in `scripts/docs-check.mjs` derives both versions from
`packages/modal/src/runner.ts` instead of hard-coding them, guards the read
with `existsSync`, and fails with a named diagnostic if either regex stops
matching — so a pin bump is a one-file edit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: apply the nomination rule to the Codex and Claude Code bullets for #653
The §3.2 reasoning bullets for Codex and Claude Code still read "until #663
measures the levels", which asserts ownership the page elsewhere denies: #663's
filed acceptance criteria name no reasoning surface, so the Owner table marks
both rows "(nominated)" and the callout says the nomination becomes ownership
only once the criteria name it. The dsh and OpenCode bullets already carry that
softening; these two now match, so every candidate bullet applies the same rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: consolidate measurement ownership and restore issue parity for #653
Applies the four findings from the review round at 4ddaff34.
- Bring the HTML §9 drafted acceptance lists back into parity with the filed
issues: §9-B gains #659's G7/G8 item (a run-scoped state root writes nothing
to the operator's real home, and the enforced sandbox boundary is named and
proven) and §9-E gains #662's credential-isolation item (the provider
credential reaches only the OpenRouter endpoint and never appears in argv).
- Consolidate the nomination-versus-ownership rule in the research page. The
normative rule and the current per-issue snapshot are now stated once in a
Measurement Ownership subsection under Qualification Gates, and the repeated
per-candidate, contract, and constraint prose is reduced to pointers at it.
Every substantive distinction is preserved: #659 owns Pi, #662 owns
OpenCode's gate measurements but is only nominated for its reasoning surface,
#663 owns the cloudPortable declared value but is only nominated for the
Codex and Claude Code shipped-pin measurements, and #661 is nominated for
dsh.
- Reflow the long Markdown prose lines to the file's ~80-column style.
Part of #653.
* docs: restore #664/#662/#659 acceptance parity and fix the ownership pointer for #653
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: correct the developer-preview source and spell the node requirement for #653
DeepSeek's API docs do call the harness a developer preview — the Quick
Start's "Integrate with Agent Tools" section says so — so the §2 cell no
longer attributes that characterization to the README alone.
The §7 risk table is grouped thematically, not ordered by severity, so both
content maps drop the "severity-ranked" claim rather than promise an ordering
the table does not have.
The Markdown page is the normative spelling of the contract, so it now spells
the node side of the event-class match that #658 and #661 both depend on:
a NodeRequirements interface and the rule that the validator matches a node's
requirement against the binding's capability before launch. The validator
rejection list gains the event-class check.
The pin guard asserted existence, so a bump applied to only one of the HTML
plan's two sites still passed while the other kept asserting the old version.
It now checks every printed literal and names the stale line. The block
comment no longer claims a bump is a one-file edit, which is the opposite of
what the check enforces.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: source the developer preview from the README and define the event-class relation for #653
Applies the six findings from the capped review at 6c14e1d4: the HTML §2
obsolete-claims cell attributes the developer-preview phrase to the DeepSeek
Harness README alone; both pages define the event-class comparison as a
partial order (final-text-only below jsonl, rpc incomparable and exact-match)
in identical normative wording for #658; the reasoning table and mirrored HTML
callout use only the Measurement Ownership terms owns/nominated; and the
Markdown page cedes the full HTML-versus-Markdown content map to the
explanation index, keeping a concise pointer plus the download-and-open note
while the problem statement leads the page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: scope the pin definition to preflight and fix config_dir composition for #653
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: carve out persisted-credential auth and de-collide config_dir for #653
The credential/state rule claimed every harness gets a run-scoped state root so
no run reads an operator's real home, but Ultrafuzz's default Codex mode is
`auth = "subscription"`, which reads `CODEX_HOME/auth.json` and falls back to
`~/.codex`. Add invariant 7: a harness whose credential lives inside its own
state root keeps a stable, writable root outside the run scope and declares that
exemption, rather than being rejected as unsupported. Mirror it in plan §5.2 and
the layer-4 box, and map the legacy `[agents.CodexAgent]` keys in the migration
paragraph.
Rename the new seed key to `config_seed_dir`. `[agents.<id>].config_dir` is
already shipped and is exported verbatim as `CODEX_HOME`, so reusing the
spelling for seed-copy semantics would silently change what `CODEX_HOME` means
for every existing Codex profile.
Scope the normative claim to #658 until SPECS.md absorbs the capability
contract, and record the same carve-out in the explanation index so its
"not normative" preamble no longer contradicts both pages.
State the nomination-vs-ownership rule once, in Measurement Ownership, and leave
bare pointers in the comparison matrix, the five per-candidate subsections, the
reasoning table, and Next Work.
Co-Authored-By: Claude Opus 5 <noreply@anthropi…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #274
Upgrades the pinned workflow engine from
smithers-orchestrator@0.31.0to0.32.0, the latest release published under that package name, and adjusts our own code where the upgrade actually changes behaviour. Full analysis, including the review-round corrections below, is appended to #274.Why 0.32.0 and not 0.33.x
GitHub carries
v0.33.0/v0.33.1tags, but0.33.0isrefactor!: rename npm package smithers-orchestrator to smthrs; the renamedsmthrspackage has only0.33.0, and0.33.1is unpublished. The rename is a breaking packaging migration touching every scoped subpath we patch, the bin resolution, the doctor registry probe and the generated manifests. It is not a bugfix, and neither of our two CLI workarounds is fixed there either (verified against@smthrs/cli@0.33.0).Did upstream fix anything we worked around?
No — all four workarounds stay. Verified against the real published 0.32.0 tarballs rather than the changelog prose:
DETACHED_ADMISSION_TIMEOUT_MS = 30_000is still hard-coded, the detached-engine logfdis still closed and then reused for the supervisor spawn, andrestoreTerminalTaskStatesdoes not exist anywhere in 0.32.0.But one workaround was affected, and my first pass got this wrong (see the correction in #274). 0.32.0 moved the eager run-startup sequence into a
startRunRuntime()closure and, on resume, calls it from insidedriverRenderer.render:cancelStaleAttemptsand the stale-attempt→pendingreset ran before our hydration anchor.So hydrating at the old anchor restored a node as
finishedand then let the reset rewrite the durable row topending, leaving the scheduler session and the database disagreeing for the whole resume — the exact divergence this workaround prevents. The patch is re-anchored to run immediately afterawait startRunRuntime();, restoring the 0.31.0 ordering, and the ordering is now asserted positionally against the pinned release instead of assumed.Changes
SMITHERS_ORCHESTRATOR_VERSION→0.32.0.overrides.effectmoves3.21.4→4.0.0-beta.102, and the@effect/*packages are pinned alongside it. Pinning Effect alone left@effect/platform-node-sharedfloating on an open caret over prereleases with--package-lock=false: measured, a fresh install gotbeta.104(whoseeffectpeerbeta.102does not satisfy) while pnpm gotbeta.103, so CI was not exercising what cloud runs install and two containers resuming the same run could install different Effect trees. Verified after the fix: one Effect copy, zero nested duplicates, every@effect/*atbeta.102with peers satisfied.0.31.0shape matched no recognised shape, so every project root with a0.31.0.smithers/package.json— including in-flight cloud runs resuming against their existing root — would have hard-failed before the engine started. The ad-hocPREVIOUS/OLDER/OLDEST/LEGACYconstants collapse into oneSUPERSEDED_SMITHERS_DEPENDENCIESlist, and the migration now refreshes any stale override rather than only a missingeffectone..smithers/node_modulesno longer bricks the project root.installedSmithersValidationErroronly inspects the top-level runner, so an interrupted upgrade install passed validation and then threw from the patcher on a path that returns instead of reinstalling — every later resume failed identically with no recovery short of deleting the tree by hand. A patch failure there now falls through to the reinstall.doctorsaid the patches "apply on the next run". Only a tree with none of the three is tolerated now.doctorreports all four workarounds, not just the two CLI ones — the two resume-durability patches, whose absence silently costs durable resume progress, were invisible. An ambiguous package root now reportsincompatible(the next run hard-fails) rather thanunknown.smithers-orchestrator, a name frozen at0.32.0forever, so it would have reported "you are on the latest release" for every future release. It now reports the newer of the two names and says that upgrading past0.32.0means migrating to the renamed package.applySmithers031CompatibilityPatches→applySmithersCompatibilityPatches, and comments asserting "Smithers 0.31" behaviour re-verified against0.32.0rather than renumbered.Regression tests
every runner compatibility patch still anchors in the pinned Smithers release— reads the actually installed release. Per workaround it asserts the anchor is unique, the replacement is absent, and the upstream evidence justifying the workaround still holds (restoreTerminalTaskStatesabsent, no configurable admission timeout). It asserts the resume reset/hydration ordering positionally, and that the Effect override matches what the pinned runner declares. Negative-verified by mutating the installed scheduler source.startRun patches every described runner compatibility workaround— seeded from the descriptions, so a newly described workaround cannot land reported-but-never-applied.startRun migrates the Smithers 0.31.0 manifest and its Effect 3 override forward— negative-verified with the migration entry removed.diagnoseProject reports a posture for every tracked compatibility patch— distinct posture per patch so a swapped id→source mapping cannot pass, and assertsincompatiblefailsdoctor.@effect/*packages, with the accepted shape derived fromrenderSmithersPackageJson().Validation
pnpm -w run ci(format check, lint, build, release validation — which runs every package's test suite plus the workspace typecheck) — green.pnpm -w typecheck,pnpm -w run test:ci-scripts,git diff --check— green.smithers up, task dispatch, worktree, durable output, worktree reap.finished, thenup --resumeit. The hydration log line appears with no errors.Accepted without change
Forward-only manifest migration means an older Ultrafuzz meeting a manifest a newer one rewrote fails until it is updated. That is the established design for every past runner bump (0.27, 0.28, 0.29, 0.31), so the rollout requirement is unchanged: the controller and the Modal image roll together.
Greptile Summary
The PR upgrades the pinned Smithers workflow engine to 0.32.0 and updates the surrounding compatibility, dependency, migration, repair, and diagnostic behavior.
Confidence Score: 5/5
The PR appears safe to merge.
No blocking failure remains.
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD Manifest["Generated .smithers/package.json"] --> Migrate["Migrate superseded manifest"] Migrate --> Install["Install pinned Smithers 0.32.0 + Effect beta.102 set"] Install --> Patch["Apply four compatibility patches"] Patch --> Start["Start or resume workflow"] Start --> Reset["Smithers resets stale attempts"] Reset --> Hydrate["Restore durable terminal task state"] Hydrate --> Schedule["Scheduler continues workflow"] Patch --> Doctor["Doctor reports patch posture"] Registry["Legacy and successor registry lookups"] --> DoctorReviews (3): Last reviewed commit: "fix(runtime): align the workspace Effect..." | Re-trigger Greptile
Context used (3)
@ultrafuzz/cli)