Repository navigation
Seal workflow execution controls and harden upgrades - #428
Merged
Merged
Conversation
added 11 commits
August 8, 2026 22:15
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ) { | ||
| throw new Error("generated agent adapter directory changed before atomic publication"); | ||
| } | ||
| const directoryAccessPath = agentAdapterDirectoryDescriptorPath(directoryDescriptor, openedDirectory); |
There was a problem hiding this comment.
If init runs where neither /proc/self/fd nor /dev/fd exposes a verifiable directory descriptor, stock-adapter publication calls agentAdapterDirectoryDescriptorPath without the fallback used for ordinary scaffold writes, causing INIT_PATH_UNSAFE and preventing both upgrades and interrupted-publication recovery.
Knowledge Base Used:
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/runtime/src/init.ts
Line: 978
Comment:
**Descriptor-only adapter migration**
If `init` runs where neither `/proc/self/fd` nor `/dev/fd` exposes a verifiable directory descriptor, stock-adapter publication calls `agentAdapterDirectoryDescriptorPath` without the fallback used for ordinary scaffold writes, causing `INIT_PATH_UNSAFE` and preventing both upgrades and interrupted-publication recovery.
**Knowledge Base Used:**
- [Runtime: Run Orchestration Lifecycle](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime.md)
- [CLI (`@ultrafuzz/cli`)](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/cli.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
aviggiano
added a commit
that referenced
this pull request
Aug 24, 2026
* feat: add threat-model goal fanout and dynamic topology nodes (#188)
* feat: add threat model and dynamic goal fanout
* feat: add dynamic runtime topology nodes
* merge: preserve threat provenance with dynamic runtime nodes
* fix: retain packaged prompt assets with dynamic variables
* feat(references): integrate external vulnerability database
* fix(references): harden vulnerability database sync
* feat(runtime): consume pinned vulnerability database reference
* fix(topology): bind vulnerability reference semantics
* fix(runtime): export Smithers runtime helpers
* test(runtime): fixture pinned vulnerability database reference
* style: format changed workflow files
* test: align topology anchors and lint unused fixtures
* test(cli): fixture pinned vulnerability reference for validation
* fix(report): allow safe audit-context links
* docs: clarify vulnerability database materialization
* fix(cloud): harden cloud selected-task handoff boundary
Consolidates the reviewed PR #188 remediation work into an intentional
commit: the shared cloud selected-task handoff contract, the Modal
provider and worker boundary validation, native-v1 vulnerability
database materialization, and the threat-model/goal-plan artifact
schemas with their tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(cloud): correlate and fully bind the cloud selected-task boundary
Closes the five audited cloud-boundary gaps and three prior-review
regression gaps:
- reject a non-empty outer `tasks` array for a `cloud_worker` dispatch,
so an outer entry can neither replace the relocated prompt body nor
name a second attempt
- reconstruct a runtime-generated dynamic attempt's whole canonical DTO
from its group's compiled template, binding the run root and every
derived path, the rendered prompt location, the agent and model
profile, the execution resources, the artifact output contracts, the
pinned planner catalog, the reference trees, and its dynamic
provenance; only the expansion key and the two runtime digests remain
runtime data, and the generating group is derived from the dispatched
attempt ID rather than taken from the handoff
- remove the production no-`selected_task` compatibility path and cover
its absence for compiled, deferred, and generated attempts
- persist a canonical logical-dispatch fingerprint (generation-independent)
and check it on durable resume, in checkpoint records and indexes, in
published results, and in live sandbox reattachment, keeping the exact
same-generation resume check distinct from the allowed
reset-generation restore
- replace the three independent dependency-array supersets with one
correlated runtime-materialization check: each added artifact
directory belongs to one added attempt, each verifier derives from
one, and each added generated node must be derivable from a declared
group
Regression gaps: `dynamic-workflow.test.ts` now uses a real non-empty
prompt exclusion and asserts transformed bytes for both the dynamic and
the deferred snapshot; topology `schema.test.ts` validates a generated
graph against the committed expanded-graph JSON Schema document; the
stale contract-valid-today comment in the artifacts threat-goal tests is
corrected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(preflight): ignore prompt pack test scratch directories
* fix(cloud): bind durable recovery to handoff contents
* docs(references): state the private-database read prerequisite exactly
`monad-developers/web3-vulnerability-database` is a private repository, so a
clean scaffold that pins `vulnerability-database.web3` cannot materialize it
without an ambient git read credential. Record that prerequisite where the pin
is documented, quoting the diagnostic the CLI actually emits.
Verified against the pinned revision with no credential available:
error: GIT_FAILED: git command failed: git fetch --depth=1 \
--filter=blob:none origin fbf00e990b1316879b674e9903548dba452e40d5: \
remote: Repository not found.
fatal: Authentication failed for
'https://github.com/monad-developers/web3-vulnerability-database.git/'
The failure surfaces as the `GIT_FAILED` reference diagnostic wrapping git
stderr, not as a bare git error, and dropping only the catalog entry leaves
`reference-vulnerability-database` dangling, which `expand()` rejects with
`INVALID_REFERENCE_NODE`. Both are stated as observed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(references): authenticate the pinned private database read
The curated vulnerability database is a private repository, but reference
materialization fetched it with a plain anonymous `git fetch`. The detached Modal
worker runs `ultrafuzz references sync` in its pre-model phase, so every
benchmark pair failed there: a dependency-unreachable error with
`model_work_started=false` and `usage=null`, after an image build and a launch
had already been paid for.
Make the credential an explicit, least-privilege contract instead.
References stay anonymous by default and attach a token only when both
`ULTRAFUZZ_REFERENCE_GITHUB_TOKEN` and `ULTRAFUZZ_REFERENCE_GITHUB_REPOS` are
declared. The allowlist is what carries the security property: a token minted for
one private repository is never sent to another remote just because the catalog
names it, and a token without an allowlist is inert. The token reaches git only
through `GIT_CONFIG_*` carrying an `http.<remote>.extraheader` keyed to the exact
remote, so it never enters a remote URL, an argv entry, a config file, or a
credential helper; `GIT_FAILED` diagnostics redact both the token and its derived
basic-auth encoding.
CI mints a short-lived token from the existing eval-history GitHub App, scoped to
`contents:read` on `web3-vulnerability-database` alone, and proves access with a
new preflight before the image build -- immutable repository metadata and the
pinned commit only, so a missing installation or permission costs nothing and
reports the exact external action required. The private key never leaves the
trusted runner; only the token enters the Modal secret, where it is also a
forbidden value for published bundles, diagnostics, and lifecycle logs.
`launch` and `collect` each mint their own token with `skip-token-revoke: true`:
installation tokens expire after an hour while `collect` can wait longer, and the
sandboxes are detached, so a post-job revocation would invalidate a credential a
worker still needs. The workflow keeps its push/workflow_dispatch-only trigger, so
fork code never reaches the private key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(preflight): report the App installation prerequisite exactly
The first CI run proved the credential contract stops before paid compute -- the
image build and every sandbox launch were skipped -- but it also exposed a gap in
what the failure actually says. `actions/create-github-app-token` fails first, so
the preflight that owns the precise diagnosis never ran, and the only evidence was
GitHub's bare `Not Found` from `GET /repos/{owner}/{repo}/installation`. That reads
like an infrastructure fault rather than the one external action it represents.
Let the mint step fail soft and make the preflight the step that fails the job. An
absent token in CI means the App has no installation on the target repository, so
the preflight now names that: install the eval-history GitHub App with
`Contents: Read-only`, an administrator must do it, and the pin must not be
replaced, vendored, or removed to work around it.
The job still stops before the image build, so this changes only the diagnosis, not
the compute that is refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(collect): keep live sandboxes collectable when the token mint fails
The recovery token step in `collect` failed hard, and the wait step it precedes is
gated on `available == 'true'` without `always()`. A failed mint therefore skipped
waiting entirely and abandoned every already-running sandbox uncollected, even
though the collection and diagnostic-upload steps that follow do run on `always()`
and would then find nothing.
Fail that step soft. Waiting and collection stay intact, and a relaunch that
genuinely needs the credential still fails on its own pair with a recorded
diagnostic instead of taking the whole matrix down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(collect): gate private-reference recovery access
* fix(eval): harden private reference recovery
* fix(references): use public vulnerability database
* fix(runtime): avoid Smithers reserved input field
* fix(runtime): normalize persisted workflow input nulls
* fix(smoke): require canonical evidence locations
* fix(prompts): bind goal IDs to source records
* fix(prompts): bind threat-model kinds and goal-plan catalog digest to their validators
The Modal smoke benchmark failed deterministically per target at the two new
fanout nodes, reproduced in a live sandbox at this head:
- threat-model: agents invented invariant kinds (solvency, state-consistency,
configuration, availability) because neither threat-model prompt stated the
ultrafuzz/threat-model@1 enum. Both prompts now spell out the five valid
kinds inline.
- goal-plan: the prompt said to copy all database digests from the supplied
catalog, so agents set catalog_sha256 to the embedded
upstream_catalog_sha256; the validator expects the SHA-256 of the planner
catalog file itself. The prompt now says to compute it over the exact bytes
at the supplied catalog path and warns that upstream_catalog_sha256 never
validates there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): pin remaining fanout artifact fields to validator semantics
A second live-sandbox round with the previous fix in place surfaced the rest
of the deterministic agent/validator mismatches:
- threat-model: string fields with plural names (evidence_needed, impact,
security_impact, value_at_risk) were emitted as arrays. Both prompts now
state the array-vs-string rule for every leaf field.
- goal-plan: agents pre-wrote vulnerability-db-manifest.json, which conflicts
with the canonical bytes Ultrafuzz publishes after the node; applicability
checks paraphrased threat-model rationale/evidence instead of copying
byte-for-byte; class-goal attack_surface_ids carried invented labels
instead of threat-model attack-surface IDs. The prompt now forbids writing
the manifest or vulnerability-db/ content, requires byte-for-byte capability
copies, and pins attack_surface_ids to copied threat-model IDs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(runtime): stop pre-materializing the vulnerability-db snapshot manifest
Task preparation backfilled every non-primary declared output with its
contract's canonical empty bytes, so goal-plan nodes always started with a
0o600 '{}' placeholder at vulnerability-db-manifest.json. The post-agent
snapshot publisher writes that manifest exclusively and fails closed on
conflicting content, which made every goal-plan attempt die with
ARTIFACT_CONFLICT once its plan validated (reproduced across all three smoke
benchmark targets in a live Modal sandbox). Exclude the snapshot manifest
from canonical-empty preparation exactly like the runtime-owned workspace
patch outputs, and pin the exclusion with a generated-workflow test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): make goal-plan capability copies and placeholder retention mechanical
The remaining goal-plan agent failures were content slips on one target:
paraphrased capability rationale/evidence in applicability checks and
coverage-gap goal_prompts with the {{threat-model:coverage-gap}} placeholder
substituted away. Tell the agent to build checks mechanically from
threat-model.json and to keep every replacement key's literal {{...}} braces
in goal_prompt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): keep coverage-gap placeholder wording template-safe
The previous commit's literal {{<key>}} example was itself parsed as a
prompt template variable and failed catalog validation. Reword to describe
the double-brace form; the exact {{threat-model:coverage-gap}} text is an
item-scoped placeholder the validator accepts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(prompts): drop literal coverage-gap braces the topology validator rejects
The topology artifact-handoff validator parses double-brace sequences in
node prompts as template variables and does not recognize
threat-model:coverage-gap outside dynamic-goal contexts. Describe the
brace-wrapped key without writing the literal placeholder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(references): assert the public database stays anonymous under a credential
The allowlist negative case named a non-existent monad-developers repository,
which both carried the pre-transfer owner and asserted nothing about a
reference Ultrafuzz actually ships. Use the real pinned public database
instead, so the case proves the shipped vulnerability-database reference is
fetched anonymously even while a private-reference token is configured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(references): re-pin the vulnerability database to the current corpus release
The shipped pin still referenced fbf00e99 (18 records, 21 capabilities,
aggregate c0ed7d23). Upstream has since published two further commits, and
the corpus now carries 23 records and 28 capabilities under aggregate
62b7fb80, including the new classes/lending category.
Re-pin to e46c0e47, the current immutable head. Its database.yml,
capabilities.yml and catalog.json are byte-identical to 153e14cc; the
intervening commit only changed upstream ingestion tooling, which Ultrafuzz
does not consume. Verified by a fresh unauthenticated fetch with GitHub
tokens, SSH agent and credential helpers disabled.
Regenerate the documented digest table and catalog statistics to match, and
correct the test comment that still described the repository as private.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(deps): reconcile the lockfile with Smithers 0.32.0 after rebase
The branch carried two lockfile edits against Smithers 0.31.0 that
conflicted with #276's 0.32.0 upgrade on main. Resolve by taking main's
0.32.0 lockfile and regenerating the workspace links from the package
manifests, which restores the @ultrafuzz/prompts link into
packages/artifacts. pnpm install --frozen-lockfile now succeeds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(evals): keep the smoke lane free of dynamic goal fanout
The smoke lane declares disable_dynamic_strategies, but that flag only ever
pruned dynamic-strategy-generator, which the dedicated smoke graph does not
contain. This branch adds threat-goals and class-goals to
benchmarks/smoke-benchmark.yml, and those expand one child per planned threat
and per applicable class, so the lane would have run unbounded dynamic work
while still reporting dynamic strategies as disabled. #277's smoke matrix
requires that guarantee to hold.
Name the goal-fanout node IDs, add them to the dynamic exclusion set, and
apply them on the smoke lane. Production-topology exclusions still cannot be
passed wholesale to the smoke graph, because its invariant, differential and
dynamic-strategy nodes are absent and would raise unknown-node errors.
Add a regression test that reads the smoke topology and asserts the lane
prunes every dynamic node it declares, so a future dynamic node added to the
smoke graph fails the suite instead of silently running.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(modal): keep curated private lanes off the new threat-model nodes
A private benchmark lane isolates part of the production topology by naming
the nodes it wants pruned. The invariant-only Aave lane does this. Every such
list was written before threat-model, goal-plan, goal-roaming, threat-goals,
class-goals and reference-vulnerability-database existed, and the first four
land in the pre-existing setup and goals groups, so no group-level filter
catches them either. Merging this branch would therefore have made the next
invariant-only run quietly execute a threat model and unbounded goal hunters,
invalidating the comparison it exists to produce.
Publish the default-on node IDs from @ultrafuzz/evals and prune them from any
private lane that curates an exclusion list, with
include_threat_model_goal_fanout to opt back in. An empty list still means
"run the whole topology", so uncurated private runs are unchanged.
Also prune goal-plan on the smoke lane. The fanout is its only dependent, so
excluding just the fanout leaves it terminal and strands the smoke report
prompt's artifact_path citation of it; the lane could not plan at all. A new
runtime suite expands the real pruned graph to prove both directions.
Keep the topology-derived tests honest: the published ID list is checked
against the shipped topology, so a future default-on node fails the suite
instead of silently widening every curated lane.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(evals): prune goal-plan wherever the goal fanout is pruned
Excluding threat-goals and class-goals on their own leaves goal-plan with no
dependent, which makes it terminal, and strands the report prompts'
artifact_path citation of it, which requires an ancestor. Either way the graph
stops validating, so any lane that drops the fanout has to drop goal-plan too.
Fold that into BENCHMARK_GOAL_FANOUT_EXCLUDED_NODE_IDS and build the smoke,
production-dynamic and curated-lane sets from it, so all three stay valid
against the graph they are applied to.
Move the pruning coverage into packages/runtime/test/topology-transform.test.ts
next to the existing exclusion tests, in the node:test style that package's
runner actually compiles and executes, and extend its production-topology case
with the same three IDs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(evals): keep the benchmark smoke lane exactly as published
Three independent reviews of 7e2687d found that pruning the goal fanout at
lane level was not enough to keep the smoke lane's guarantee.
Per-run agent concurrency is max_parallel_targets, which is 4, so the pruned
12-node graph still serialized into six waves: 6 x 2 x 1200s = 14,400s against
a 15,000s row watchdog that packages/modal/src/config.ts caps at 15,000. The
margin fell from 5,400s to 600s and could not be raised without a schema
change. Keeping threat-model also kept requiresVulnerabilityDatabase true, so
every smoke run gained a plan-time corpus fetch and the exact-cover capability
gate in verifyThreatModelVulnerabilityDatabaseCapabilities -- at medium
reasoning with two attempts. In a model matrix that gate fails a row outright,
so a weaker entrant publishes no observation at all and the comparison is
silently biased. Passing any non-empty excluded_node_ids also moves
benchmark_execution, and with it the execution-policy and cohort fingerprints
that #277's rows must share with the already-published smoke observations.
Restore benchmarks/smoke-benchmark.yml byte-for-byte to main, drop the unused
smoke threat-model prompt, and take the smoke lane back to an empty exclusion
list. The lane keeps its guarantee structurally, the way it always did: the
dedicated graph simply does not declare invariant, differential,
dynamic-strategy or goal-fanout nodes. A test now asserts that rather than
asserting a pruning list.
Keep the prompt-only smoke improvements that carry no node dependency: the
evidence-path hardening and the source_nodes provenance bullets.
The full lane is untouched and still exercises the whole threat-model
workstream, and the production topology keeps every default-on node.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(runtime): rebuild dedupe provenance from the lifecycle ledger when syncing
Three independent reviews of 0d3e5c2 found two real regressions in the new
finding-provenance machinery, both introduced by this branch.
dependencyFindingProvenanceForTask rebuilt the source expectations without the
lifecycle ledger, so buildFindingSourceExpectations returned early and produced
one single-source expectation per upstream finding. A dedupe root's source_nodes
is the union of everything it merged, which matches none of those, so
finalizeTerminalTask re-validated a node the authoritative in-workflow gate had
already passed and marked it failed with
task-output-validation-failure. That fires on any run with a genuine merge, and
it is reachable from ultrafuzz inspect, ultrafuzz status, and the dashboard, so
it needs no cloud execution. It also attributed upstream findings to the
dependency attempt ID rather than the producing node, so a generated dynamic
child could never match its own ledger record. Load the ledger and resolve
producer node IDs through tasksByAttempt, mirroring the generated template,
which is the authoritative implementation of the same contract. Stop creating
artifact directories from this read-only path while here.
The smoke prompts described a lifecycle ledger the new gate rejects: entries
need node_id and finding_id and must cover every upstream finding exactly once,
which only the production prompt said. Bring the smoke prompt up to the same
contract and tell the smoke report to carry finding identity through, since
provenance is matched on it.
Also correct a comment claiming the goal fanout exists in the smoke graph, a
reference example naming a node and prompt that do not exist, and the changelog
description of the roaming goal, which is a static node rather than an
expansion. Document the deliberate asymmetry in the private-reference credential
contract and the one-off lineage discontinuity a curated private lane sees.
Add a production-topology test proving the exact set every curated lane prunes
still plans, and that no dynamic node, goals-group node, or vulnerability
database reference survives it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): bind a dedupe root to its lifecycle ledger record
Round-three review found the remaining half of the provenance contract was
unstated. buildFindingSourceExpectations derives one expectation per upstream
finding plus one per ledger record, and matches an emitted finding against
whichever expectation shares an identity key with it. Only the ledger-record
expectation carries a multi-node union, so a root that merged findings from two
different nodes resolves to it solely through a shared dedupe_key. Neither
prompt told the model to write that key onto the retained finding, so any
genuine cross-lane merge failed with "source_nodes does not preserve the exact
dependency discovery-source union", on both the production and smoke lanes.
Say so in both prompts, and give the production prompt the exact-coverage rule
the smoke prompt already gained: every upstream finding must appear exactly once
across all source_artifacts, including ones kept only as a duplicate or family
variant. The gate enforces that on both lanes; only the smoke prompt said it.
Make the sync-side ledger reader fail the way the template does. Swallowing a
parse error re-entered the naive expectation set the previous commit removed and
reported a corrupt required artifact as a findings-contract violation. Reject a
symlinked ledger or dependency findings file rather than following it out of the
run root, matching resolveRegularArtifactFile.
Pin the surviving node set in the curated-lane completeness test. Its shape
checks could not see a default-on node added to a pre-existing group, which is
exactly where threat-model and goal-plan live, so the guarantee it advertised
was not the one it enforced.
Add the regression test for the expectation set itself, and make the reference
example match the shipped threat-goals node rather than a partial copy of it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): give dedupe the full producer list its ledger is checked against
Round-four review found the last gap in the new provenance gate. Ledger
coverage is checked against every dependency findings artifact the runtime
read, but the production dedupe prompt named only 5 of the 29 nodes that
declare a findings.json output, and told the model never to guess a path. The
other 24 -- boundary-tests, the invariant chain, the differential lanes and the
rest -- were unreachable from the prompt, so any run in which one of them
reported a finding failed dedupe-findings deterministically with "finding
lifecycle ledger omitted dependency findings", then cascaded to triage,
severity-classification and final-report. The smoke lane was immune only
because it enumerates all four of its producers, which is why three earlier
rounds and CI missed it.
Enumerate every producer. Per-line rendering means a curated lane that prunes a
node also drops its line, so an invariant-only lane sees exactly the lanes it
still runs rather than an instruction to account for artifacts that no longer
exist.
State the two rules the gate enforces that the prompt left implicit: node_id is
copied out of the finding rather than taken from the group template or the
storage directory in its path, and a finding judged unsupported gets its own
ledger record instead of being attached to an unrelated root, which would put a
non-contributing node in that root's source union.
Complete the sync/template mirror: a missing ledger now names itself instead of
degrading to the naive expectation set and reporting a missing required
artifact as a findings-contract violation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): state the source-union equality rule the review lane is held to
Round-five review audited every throw site in the new provenance gate against
what the prompts actually instruct, and found the production dedupe prompt
still missing the rule the smoke prompt already had: a retained root's
source_nodes must equal exactly the node_id set of its own ledger record's
source_artifacts. Two of its sentences pushed the other way -- "preserve the
relevant source union on family variants" invites a subset, since a variant's
artifacts belong to the root's record, and "include all corroborating node IDs"
invites a superset from related findings that were deliberately not merged.
Either fails the node with the same opaque exact-union error.
State the rule, and drop the two sentences that contradict it.
Also close the smaller gaps that audit surfaced: every ledger record needs its
own dedupe_key including the coverage-only ones, no two records may share one,
triage and severity-classification must carry id and dedupe_key through rather
than assigning a fresh stable id, and the restart-handling short circuit now
lists finding-lifecycle-ledger.json so a resumed attempt cannot declare itself
finished without the one output whose absence hard-fails the node.
Pin the producer enumeration to the topology with a test, since nothing tied
them together and the list had already drifted once. Document the ledger in the
review guide, and make the goal-fanout header self-explaining when a curated
lane prunes its paths away.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(prompts): name the finding fields the review contracts now require
This branch moves deduped-findings.json, triaged-findings.json and
severity-classified-findings.json from ultrafuzz/json-array@1 to
ultrafuzz/findings@1. That subjects all three to the full normalized-finding
field check, not just the source-union rule, and none of the three prompts
listed those fields. severity-classification was the worst case: `summary` is
required, the prompt never mentioned it, and its four worked examples omit it,
so a model following the only concrete shape in the file fails the node on
every attempt. dedupe-findings was missing status, severity_guess and
confidence.
Name the required fields in all three, and say that the severity examples show
only the fields that stage adds rather than the whole object.
Pin it with a test, which caught the dedupe-findings gap that reading had
missed. Also close the two remaining review items: the restart short circuit
now validates the ledger's shape instead of only its existence, so a resumed
attempt with a truncated or incomplete ledger rebuilds rather than declaring
itself finished; and the producer-enumeration pin now covers the smoke lane
too, which had the same drift hazard unguarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(prompts): pin every required finding field, not five of seven
The field-contract test listed the five fields normalizeFinding hard-requires
but omitted schema_version and id, which the finding schema also requires. All
three review prompts already name all seven, so this changes no behaviour; it
closes the hole where a later edit could drop one of the two unpinned mentions
and leave the test green while the node fails its contract on every attempt --
the exact failure the test was written to prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Antonio Viggiano <3029017+aviggiano@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-192-168-1-2.ec2.internal>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
* feat(goal-plan): record expected dynamic-child cardinality and goal lanes at planning time (#373)
Implements #364 option (a): the planner writes its dynamic-fanout expectation down, and the
eval side compares against the run without ever recomputing the rule. Expected and observed
then come from different components, so a planner that under-expands is caught.
`goal-plan.json` gains `expected_child_count`, `threat_count`, `applicable_class_count`,
`max_dynamic_nodes`, and `goal_lanes` (each lane named, with the node IDs it owns). They are
recorded deterministically by the `goal-plan` node's postprocessor from the plan the agent
wrote plus the run's resolved `run.max_dynamic_nodes`, before the plan is validated. A value
the agent wrote that disagrees with its own plan fails closed rather than being silently
corrected, and a plan expecting more children than `max_dynamic_nodes` allows fails as a plan.
The cardinality rule -- `threats + applicable classes`, with the fixed roaming node as a static
lane rather than a dynamic child -- is computed by exactly one function, `goalPlanExpansionFacts`.
That now includes the pre-existing `counts` block, which computed the same arithmetic a second
time twenty lines away: `counts.threats`, `counts.applicable_classes` and `counts.dynamic_goals`
are checked against those facts instead of restating `threat_goals.length + class_goals.length`,
so `threat_count`, `applicable_class_count` and `expected_child_count` are aliases that cannot
drift from them. `max_dynamic_nodes` and `goal_lanes` are the only genuinely new information.
Lane granularity is recorded rather than left implicit: a lane is one goal, so `lane_id` is a
threat or class ID and a run has `threats + classes + 1` lanes. The alternative -- one lane per
dynamic group, three per run -- aggregates more cheaply but cannot say which goal was expensive
or which one failed. Group totals stay derivable by summing one `kind`; the reverse is not.
Per-lane cost is now joined on a key that exists. `UsageLedgerEntry` gains an optional `node_id`,
the identity `state.json` keys a node under, resolved by the runtime from its own task table --
a usage event names the workflow task (`node:<attempt>`), not the state node, and only the task
table knows that mapping. `attempt_id` could not serve: it is `sha256(workflow run, node,
iteration, attempt)` and carries no node field, so the previous join matched nothing on every
real ledger while still reporting `lane_cost_evidence: complete`. The field is optional because
`usage.jsonl` is durable: ledgers written before it existed still parse, and a run resumed across
the upgrade does not fail its replay when a recorded entry gains the key.
`evalRunExpansion` now reports `plan`, `expected_vs_actual`, and per-lane `goal_lanes`. It reads
the planner's numbers and compares; a mismatch is kept as data with the size and sign of the
disagreement, never dropped. Lanes fall out of per-node status for failures, the usage ledger for
tokens and cost, and node timestamps for wall-clock, so metrics 5 and 6 need no new telemetry.
Each lane carries `cost_evidence` and `usage_matched_node_count`, so a lane that genuinely spent
nothing (zero, complete) is distinguishable from one whose join never landed (null, unavailable
with a reason: no `node_id` in the ledger, no entry naming this lane's nodes, or no node observed
for the lane at all). `EvalRowScore.expansion` stays optional so historical scores still read.
The eval test fixture no longer hand-writes `usage.jsonl`. It calls `createUsageLedgerEntry`, the
same function the runtime calls; the old fixture invented a `schema_version` and an `attempt_id`
shape the real writer cannot produce, which is how a structurally dead join kept a green test.
Dynamic child node states carry `provenance.source_node_id`, the one key both sides must agree on
and the key #365's reader already looks for. The nested camelCase `dynamic` record stays alongside
it for the expansion key, item digest, and manifest path.
Verified: `pnpm -r build` 0, `pnpm -r test` 0, `pnpm -r --workspace-concurrency=1 typecheck` 0,
`pnpm -w format:check` 0, `pnpm -w lint` 0. No benchmark run was dispatched.
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(evals): v0.1.0 threat-model release-gate lane (#351)
* feat(evals): add the v0.1.0 threat-model release-gate benchmark lane (#183)
The three pinned Ultrafuzz-bench targets have never seen the production
topology. The smoke lane substitutes the reduced
`benchmarks/smoke-benchmark.yml` graph, and the full lane is bound to the
EVMBench cohort, so the `threat-model`, `goal-plan`, `threat-goals`, and
`class-goals` nodes #188 added run against no real pinned protocol. This
adds a third lane rather than editing the smoke graph, whose empty
`excluded_node_ids` is what keeps the 99 published observations
comparable.
The `threat-model` lane pairs the Ultrafuzz-bench cohort with the
unmodified production graph: no `topology` override, all three `disable_*`
flags off, and an empty exclusion list. Two guards keep that honest.
`loadBenchmarkLanesManifest` rejects a checked-in gate lane that turns any
production work off, and `adaptBenchmarkManifestToEvalSuite` refuses to
compile a suite whose exclusions would remove a node the gate exists to
exercise, so a lane edit cannot quietly produce a green run that proves
nothing.
The gate takes its GPT-5.6 Luna/high runner from `benchmarks/lanes.json`
instead of the dispatch form -- the workflow leaves `BENCHMARK_MODELS_JSON`
unset for this lane -- so no dispatch-form value can retune its model
identity. It runs three sandbox rows with the production graph's eight-way
in-workflow concurrency, which is what a large dynamic ready queue needs to
be scheduled rather than serialized, and takes the 15,000-second Modal
watchdog ceiling. Beyond the report bundle it retains `THREAT_MODEL.md`,
`threat-model.json`, `goal-plan.json`, and `vulnerability-db-manifest.json`
from every row, because its assertions are deliberately structural and the
prose has to stay reviewable.
Gate results are not published into `benchmarks/history.json`. Publication
is a longitudinal claim about one comparable series, and this lane runs a
different topology and execution policy against the same targets. The
producer now declares its dispatched lane as a `Benchmark lane <name>` step
-- the `workflow_run` payload carries no dispatch inputs, and reading a step
name is the introspection the producer's own recovery job already does --
and the qualifier declines any lane that is not the one its trigger is
allowed to have produced, instead of assuming every dispatch is the
EVMBench matrix.
Tests assert the generated dynamic node IDs against the real expansion of
the real topology templates: a contract-valid `ultrafuzz/goal-plan@1`
document expanded through `planDynamicExpansion` must reproduce the plan's
own `node_id` for every threat and class goal. That spelling is
`dynamic:threat:<id>` / `dynamic:class:<id>`, which the topology templates,
the goal-plan contract, the planner prompt, and the changelog all pin;
#185's `dynamic.threat.…` sketch predates that decision and is not what
#188 shipped.
Nothing dispatches the lane. Selecting `benchmark_lane: threat-model`
remains a maintainer decision with a real cost attached.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(evals): let the post-eval checkpoint accept every dispatchable lane (#183)
The public eval diagnostics schema still enumerated `["smoke", "full"]` after
the `threat-model` release gate became a third lane. The worker builds that
document from `public_benchmark.lane` in `runAndCheckpointPublicEvalDiagnostics`
only AFTER `runEval()` returns, so a `threat-model` dispatch would spend the
whole cohort's model budget and then die at the post-eval-pre-score checkpoint
with a Zod `invalid_value` on `lane` -- surfacing as a
`PublicEvalDiagnosticsBuildError` and producing no diagnostics, no bundle, and
no gate evidence at all.
Derive the enum from `BENCHMARK_LANE_NAMES`, which is already the source for the
Modal config, the public bundle, and the history observation schemas.
The regression test builds the checkpoint for every lane in
`BENCHMARK_LANE_NAMES` through the real `createPublicEvalDiagnostics` path
rather than asserting the enum, so a future lane that forgets a schema fails
here instead of on a paid run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(evals): wire the release-gate lane to the landed retention and observability
Merges release/v0.1.0, which now carries B1 (#363) and B2 (#365), and removes
the duplicated artifact list: PUBLIC_OPTIONAL_ROW_ARTIFACTS now derives from
BENCHMARK_THREAT_MODEL_RETAINED_ARTIFACTS instead of repeating the same four
names as a literal in a second package, where the two could drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(evals): repair threat-model lane on the release contract
Populate the post-#373 goal-plan cardinality and lane fields from the shared authority, using the same 512-node limit exercised by dynamic expansion. Remove the reporter-only artifact include spread; the public worker remains the real retention path.
* ci(evals): enforce threat-model structural gate
---------
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Merge main into release/v0.1.0 after threat-model gate
# Conflicts:
# .github/workflows/eval-benchmarks.yml
# CHANGELOG.md
# packages/artifacts/src/findings-schema.ts
# packages/artifacts/src/findings.ts
# packages/artifacts/test/artifacts.test.ts
# packages/artifacts/test/schema.test.ts
# packages/modal/src/node-provider.ts
# packages/modal/src/node-worker.ts
# packages/modal/test/ci-config.test.ts
# packages/runtime/src/index.ts
# scripts/ci/qualify-modal-benchmark-publication.mjs
# scripts/ci/qualify-modal-benchmark-publication.test.ts
* fix(ci): reconcile release benchmark dispatch lanes
* Merge trusted benchmark recovery prerequisite into release lane
# Conflicts:
# packages/modal/test/ci-config.test.ts
* fix(ci): authenticate threat-model benchmark launches
* Seal workflow execution controls and harden upgrades (#428)
* fix(runtime): seal workflow execution controls
* fix(runtime): recover stale snapshot publications
* fix(runtime): persist detached task control paths
* fix(runtime): guide sealed workflow upgrades
* fix(runtime): keep sealed control paths private
* fix(runtime): publish stock adapter upgrades atomically
* test(cli): synchronize status watch samples
* fix(runtime): harden stock adapter publication
* style(cli): format status watch synchronization
* fix(runtime): recover torn adapter marker
* Harden init publication and file writes
---------
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
* Preserve retry cost, timing, and history provenance (#429)
* fix(evals): retain retry accounting and timing (#427)
* fix(evals): preserve partial history metrics (#427)
* fix(evals): harden history chart provenance (#427)
* test(runtime): use sealed fork evidence for retry history
---------
Co-authored-by: Ubuntu <ubuntu@ip-192-168-2-198.ec2.internal>
* chore(modal): remove stale release-sync imports
* fix(runtime): admit sealed dynamic control extensions
* fix(references): support vulnerability database v0.0.2
* chore(release): complete v0.1.0 readiness gates
* test(dashboard): make dynamic flow fixture hermetic
* feat(config): add fuzz-only audit profile (#565)
Preserve the established v0.0.x fuzzing graph as a packaged v0.1.0 profile while excluding the vulnerability database, threat model, and goal fanout.\n\nCloses #559
* fix(config): reserve the default audit profile (#580)
* fix(config): reserve default audit profile
* docs: clarify audit profile reporting cohorts
* fix(config): enforce unmodified default profile
* docs: trim audit profile migration note
* refactor(prompts): move agent preamble to MDX (#652)
* fix(runtime): route custom Codex provider preflight (#654)
* fix(runtime): route custom Codex provider preflight
* fix(runtime): harden custom Codex provider routing
* docs: begin provider-agnostic harness research
* docs: separate provider, harness, and model for #653
Publish the provider-agnostic coding harness research and the reviewed
architecture plan, and make both consistent with the decisions recorded on
#653.
Provider, harness, and model stay three independent choices throughout. The
first-party defaults are stated explicitly: OpenAI is reached through the Codex
CLI harness, Anthropic through the Claude Code harness, and DeepSeek V4 through
`deepseek-ai/deepseek-harness` (`dsh`) — final-text-only, and only once it is
qualified. OpenRouter is a gateway with no harness of its own, so it gets a
provider-neutral one: Pi first, OpenCode second. Codex + OpenRouter and
Claude Code + DeepSeek stay supported compatibility pairings and are never the
default.
Narrow two claims the sources do not support:
- Qualification requires a pinned real CLI against the real provider. Fake CLIs
are for simple unit tests only; they never satisfy a gate.
- Drop the proposed `#653-H` session-log reader. `dsh`'s
`session.jsonl.zstd` is undocumented and self-declares `"version": 0` on a
prerelease whose upstream accepts neither issues nor pull requests, so
Ultrafuzz parses nothing from it. The measurement stays as evidence; the
forward-looking parser is replaced by a small real-`dsh` smoke profile that
lives inside the adapter issue (#661) — headless final text plus exit status,
one filesystem/shell artifact check in a disposable worktree, a version pin
asserted at preflight, and an explicit final-text-only fallback that warns
rather than failing the run.
Also record that proposals A-G were filed as #658-#664, and link the follow-up
work from the research page's next-work list.
Refs #653, #658, #659, #660, #661, #662, #663, #664
* docs: apply harness research review and guard both pages
Applies the completed Opus review of the provider/harness documentation for
#653 and makes the two explanation pages consistent with each other and with
the real state of the work.
Plan (docs/explanation/provider-harness-plan.html):
- Masthead and footer now record the real state — #658-#664 filed, PR #665
open, reviewed and published for implementation — instead of claiming no
issues and no PR exist.
- The §1 verdict marks Pi and DeepSeek Harness as proposed defaults pending
real-provider qualification, and states that Codex CLI remains the shipped
OpenRouter default in the interim. The §4 gateway card says the same.
- The §3.2 candidate matrix gains an evidence-provenance column separating
real execution, --help/--version inspection, and upstream documentation, and
its Events/Sessions/Isolation columns now use the capability-contract
spellings.
- The §5 capability boxes use the normative vocabulary
(native-sandbox | external-sandbox-required, filesystem / shell) identical to
the Markdown interface, with a callout binding #658 to that spelling.
- The §7 risk register names five maintained harnesses instead of four, calls
out Kimi separately as an existing binding, and uses one pill class for the
low severity rows.
- §4 subsection headings drop from h4 to h3, and each proposal gains an id so
the #653-D references link to proposal D rather than the section.
Research (docs/explanation/provider-harness-research.md):
- The status block reconciles with the filed issues and the open PR, and points
at the HTML plan as a download-and-open-in-a-browser companion.
- The comparison matrix is now readable on GitHub: a compact matrix plus a
per-candidate subsection carrying the long detail, with an explicit
provenance legend.
- The capability contract is stated as normative and bound to #658.
- Every DeepSeek Harness source URL is pinned to commit
99f6f02fecdb7dff40c3fbc9470f5907c29f74ca rather than master.
- Headings use Title Case and the H1 matches the explanation index.
Index (docs/explanation/index.md) explains how to read the pair: the HTML plan
must be downloaded and opened in a browser, the Markdown page is the
GitHub-readable companion, and §7 and §9 are unique to the plan.
scripts/docs-check.mjs now requires both explanation pages, so the real docs
check fails if either normative document is deleted or renamed.
Part of #653.
* docs: number the gates and align the contract spelling for #653
Apply round-1 review findings on the provider/harness pages.
- Number the Markdown conformance gates G1-G10 to match HTML plan §8, and
add the two generic gates that only existed in the HTML (G7 state
isolation, G8 sandbox boundary), so the G-numbers cited by #659, #661,
#662, #663 resolve on the GitHub-readable page.
- Stop asserting that #658 implements the capability spelling verbatim:
#658's summary block abbreviates `fs` and `native`/`external`. Both
pages now state the normative spelling and record the issue text as
shorthand to be corrected, rather than claiming an alignment that does
not exist.
- Add a "Credential and State Rules" subsection mirroring HTML §5.2, so
the argv rule, run-scoped state root, invocation-cwd rule, and
harness-injected-environment caveat appear on the Markdown page too.
- Use `harness = "dsh"` consistently in the Markdown config example, so
it matches HTML §5.1, "Next Work" item 4, and #661.
- Record that neither dsh nor OpenCode has a measured reasoning surface,
drop `reasoning = "high"` from the dsh example, and state that the
validator rejects `reasoning` on a binding declaring no levels.
- Disclose that PR #654's real-codex assertions are guarded by a
`codex --version` probe and are not exercised in CI.
- Correct the HTML-unique section claim on both the research page and the
explanation index, unify "final-text nodes" phrasing, match the index
link text to the HTML page's H1, and restore alphabetical order in
docs-check.mjs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: correct the dsh reasoning surface and restore page parity for #653
Round-2 review fixes for PR #665.
- dsh does have a reasoning surface: `dsh-llm-deepseek` declares `thinking`
and `reasoningEffort` (off/low/high/max) at 99f6f02, with an omitted effort
resolving to `high`. It is a settings field with no CLI flag and no schema
default, which is why it never appeared in `--dump-default-config`. The
binding declares an empty `reasoningLevels` because nothing measured it,
not because the harness lacks it; #661 owns the measurement.
- Mirror the dsh and OpenCode reasoning findings into the HTML plan (§3.1
row, §3.2 callout, §5 contract box, and the drafted D/E issue bodies) so
the "everything appears on both pages" claim holds again.
- Narrow the matrix-column claim: the columns are drawn from the contract
vocabulary, composite cells name a contract value plus an advertised extra,
and the direct provider-API row is not a harness binding.
- Reconcile the measured Codex 0.147.0 / Claude Code 2.1.233 evidence against
the versions the Modal worker image actually pins (0.146.0 and 2.1.207).
- Grandfather the pairings shipping today out of the G1-G10 default rule and
record the qualification debt explicitly.
- Make `config_dir` a base path the launcher expands per run, so the worked
example stops contradicting the run-scoped state-root rule.
- Use one name for the numbered gates ("qualification gates") across both
pages and the explanation index; reserve "conformance suite" for test code.
- Reframe §2 around what "DeepSeek Code" refers to and anchor the withdrawn
draft to branch commit bb831d61.
- Title-case the HTML h1 to match the index link text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: reconcile the OpenRouter default and page content map for #653
Applies the remaining review findings to the provider/harness documentation
for #653. No runtime change.
- Reconcile Codex + OpenRouter throughout the pairing policy: it is the
grandfathered shipped OpenRouter default today, while a compatibility
pairing is never the recommended default for a new project. "Shipped
default" and "recommended default" are now defined against each other, and
Codex + OpenRouter appears in both grandfathered-pairing lists alongside
Codex + OpenAI, Claude Code + Anthropic, Claude Code + DeepSeek, and Kimi.
- Replace the false cross-page parity claim with an accurate content map in
the Markdown introduction and the explanation index: HTML-only material
(§1, §7, §9, the §2 correction table, and the packaging, retry,
Modal-portability, and maintenance-posture evidence rows) and Markdown-only
material (TOML boundary, TypeScript interfaces, legacy agent mapping,
dsh-specific gate items, and the full pinned source list).
- State that error classification, preflight behavior, and retry semantics
were not measured for any CLI-inspection candidate, with explicit
ownership: #659 for Pi, #662 for OpenCode, and #663 for Codex and Claude
Code re-measured at the versions the shipped Modal image pins.
- Correct the malformed usage JSON in the HTML plan so inputTokens and
outputTokens carry colons, matching the Markdown page.
- Update the substantive revision date to 2026-08-19 in both pages,
including the plan masthead and footer.
* docs: make reasoning levels, preflight, and pin citations consistent for #653
Applies round-1 review findings to both provider/harness pages.
- Reasoning: the contract says `reasoningLevels` records only levels a real
request exercised, but the worked TOML set `reasoning = "high"` on a Pi
binding whose provenance is CLI inspection. Keep the rule and make every
binding obey it: drop the key from the example, record Codex's
`model_reasoning_effort` and Claude Code's `--effort` surfaces with their
in-repo adapters, and replace the "two candidates" claim with a table showing
all five are empty today and which issue measures each.
- Preflight: G2 now owns authenticated preflight behavior including the 401
path, since G5/G6 cover events and resume only.
- Scope the unmeasured dimensions by the real gap (no row hit a real paid
provider) rather than by the CLI-inspection provenance label.
- Cross-page content map: only Modal portability is HTML-exclusive; packaging,
retry, and maintenance posture are on both pages. Same wording in index.md.
- Claude Code + DeepSeek gets the same shipped/recommended vocabulary already
applied to Codex + OpenRouter, and picks up the licensing and installation
detail every other candidate records.
- Cite `CODEX_CLI_VERSION` and `modalSecurityToolchainCommands()` instead of
raw line numbers, and guard both pins in docs-check so moving one in the
Modal image without updating the docs fails `pnpm docs:check`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: resolve the #663 ownership pointer and contract gaps for #653
Round-2 review fixes for the provider/harness doc pair:
- #663's body scopes only the Modal image capability checks, so the pages
no longer assert it already owns the version reconciliation, the Codex and
Claude Code error-classification/preflight/retry measurements, or their
reasoning levels. Both pages now state the issue must be extended and that
the doc spelling governs, matching the device already used for #658.
- Rescope the Constraints bullet from "every CLI-inspection candidate" to
every candidate including the two real-run rows.
- Stop overloading "pinned": the provenance definition now says the CLI was
installed at an exact recorded version, leaving "pinned" to mean the
shipped Modal image version defined in the gates section.
- Fix the conformance-suite cross-reference from §6 to §7.
- Give `cloudPortable` a stated owner and record local/Modal portability for
Pi and OpenCode, so the contract field resolves for every candidate.
- Title Case the new index.md heading, match the two TOC labels to their
headings, give all five candidate subsections a parallel Reasoning bullet,
and note that #659's enumerated G2 criterion predates G2's widening.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: correct the OpenCode reasoning surface and gate statuses for #653
Applies the outstanding review findings on the provider/harness pages.
- OpenCode does expose a reasoning surface: real `opencode-ai@1.18.18
run --help` advertises `--variant` as a provider-specific reasoning
effort. Record it as CLI inspection, keep `reasoningLevels` empty
pending #662, and drop every claim that OpenCode has no such surface —
in the Markdown detail, its reasoning table, the mirrored HTML
callout, and #653-E's drafted scope.
- Stop claiming ownership is explicit for Codex and Claude Code while
also calling the work unowned. Ownership is explicit only for Pi
(#659) and OpenCode (#662); #663 is nominated, not assigned, until its
acceptance criteria are extended. Both pages now say that identically.
- Downgrade the HTML G2 dsh status from `passed locally` to a warning:
the canary passed locally but authenticated preflight is unmeasured,
and G2 now covers both.
- Title-case the Markdown `Credential And State Rules` heading.
- Narrow the dsh default claim: dsh's shipped composition sets
`agent-default-model` to `deepseek-official` / `deepseek-v4-flash`,
which is not a `shipped default` in the sense the pairing policy
reserves for Ultrafuzz provider/harness selection.
- Repair the HTML conformance cross-reference so it names the
`conformance suite` of §9 and the `conformance test` of §7.
- Replace `shipped default by inertia` with `by grandfathering` in
Phase 5.
- Correct the `scripts/docs-check.mjs` pin-guard comment to
`[pin, ...filesThatMustContainIt]` and explain why the interpolated
Codex pin needs two entries while the Claude Code pin needs one.
- Add `Local and Modal results agree for every approved harness` to the
HTML §9-F acceptance list, matching filed issue #663.
Refs #653.
* docs: fix the reasoning-surface count and make cloudPortable expressible for #653
Apply round-1 review findings on the provider/harness pages.
Majors:
- The §3.2 reasoning callout in the HTML plan still opened with "Four
candidates do have a known surface" while enumerating five and closing
with "All five bindings" — surviving residue of the pre-correction claim
that OpenCode had no surface. It now reads "All five candidates", agreeing
with the enumeration, with the paragraph's own conclusion, and with
provider-harness-research.md.
- `HarnessCapabilities.cloudPortable` was a required `boolean` while the page
states three times that no candidate declares a value. A required boolean
cannot express "unmeasured", so #658 could not implement the contract
verbatim as instructed. It is now optional, with the absent case defined
the way an empty `reasoningLevels` already was: the validator refuses cloud
scheduling rather than reading absence as `false`. `cloudPortable` also
joins the §5 layer-3 capability box list, which the adjacent callout calls
out as matching the interface.
Minors:
- "the two converge at Phase 5" was unresolvable from the Markdown page,
which never defines the phases. Next Work entries now carry their phase
numbers and the reference points at #664.
- The cross-page content map claimed the legacy `agent = "…"` forward mapping
was Markdown-only; the plan carries the `DeepSeekAgent` half in its §2 table
and §9-C scope. Narrowed in both index.md and the research page.
- Drafted §9-A and §9-C acceptance lists were missing criteria their filed
counterparts (#658, #660) carry: the event-class-below-node-requirement
failure case, and the compatibility-pairing migration.
- The reasoning table's "Owner" column asserted #663 ownership that the prose
two paragraphs below denies; the two rows are now marked "(nominated)".
- Issue-tracker errata across both pages were phrased as standing
instructions ("#663's body must be extended", "#658's body should be
corrected") that go stale silently the moment an issue is edited. They are
now durable conditionals and precedence rules — ownership attaches when an
issue's acceptance criteria name the work, and the docs govern spelling
wherever the two disagree — so the pages stay correct either way.
Nit: "Codex and Pi, whose surfaces are known but were only inspected" misused
the provenance vocabulary; Codex has no recorded CLI inspection. The two
provenances are now distinguished.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: apply the page's own ownership rule to #661 and #662 for #653
Round-2 review fixes for the provider/harness pages:
- The reasoning-measurement rows for dsh (#661) and OpenCode (#662) are
marked `(nominated)` like the two #663 rows. Neither filed issue names a
reasoning, variant, effort, or thinking measurement, so the page's own rule
— ownership counts only when the acceptance criteria name it — leaves #659
as the single tracked row.
- dsh joins the error-classification, authenticated-preflight, and retry
ownership accounting: #661 is nominated there too, so all five harness
candidates are accounted for instead of four.
- `reasoningLevels` becomes optional, matching `cloudPortable`: absent means
unmeasured, `[]` means measured and no level accepted. That is what §9-E's
"or the binding documents that it accepts none" needs in order to pass, and
the acceptance wording now names the spelling.
- §9's "Filed as #658–#664" callout records which §9 items post-date the
filing and are therefore drafts rather than filed text.
- The masthead and footer no longer assert that PR #665 is open, since the
page only exists on a release branch once it merges.
- The harness-pin guard in `scripts/docs-check.mjs` derives both versions from
`packages/modal/src/runner.ts` instead of hard-coding them, guards the read
with `existsSync`, and fails with a named diagnostic if either regex stops
matching — so a pin bump is a one-file edit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: apply the nomination rule to the Codex and Claude Code bullets for #653
The §3.2 reasoning bullets for Codex and Claude Code still read "until #663
measures the levels", which asserts ownership the page elsewhere denies: #663's
filed acceptance criteria name no reasoning surface, so the Owner table marks
both rows "(nominated)" and the callout says the nomination becomes ownership
only once the criteria name it. The dsh and OpenCode bullets already carry that
softening; these two now match, so every candidate bullet applies the same rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: consolidate measurement ownership and restore issue parity for #653
Applies the four findings from the review round at 4ddaff34.
- Bring the HTML §9 drafted acceptance lists back into parity with the filed
issues: §9-B gains #659's G7/G8 item (a run-scoped state root writes nothing
to the operator's real home, and the enforced sandbox boundary is named and
proven) and §9-E gains #662's credential-isolation item (the provider
credential reaches only the OpenRouter endpoint and never appears in argv).
- Consolidate the nomination-versus-ownership rule in the research page. The
normative rule and the current per-issue snapshot are now stated once in a
Measurement Ownership subsection under Qualification Gates, and the repeated
per-candidate, contract, and constraint prose is reduced to pointers at it.
Every substantive distinction is preserved: #659 owns Pi, #662 owns
OpenCode's gate measurements but is only nominated for its reasoning surface,
#663 owns the cloudPortable declared value but is only nominated for the
Codex and Claude Code shipped-pin measurements, and #661 is nominated for
dsh.
- Reflow the long Markdown prose lines to the file's ~80-column style.
Part of #653.
* docs: restore #664/#662/#659 acceptance parity and fix the ownership pointer for #653
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: correct the developer-preview source and spell the node requirement for #653
DeepSeek's API docs do call the harness a developer preview — the Quick
Start's "Integrate with Agent Tools" section says so — so the §2 cell no
longer attributes that characterization to the README alone.
The §7 risk table is grouped thematically, not ordered by severity, so both
content maps drop the "severity-ranked" claim rather than promise an ordering
the table does not have.
The Markdown page is the normative spelling of the contract, so it now spells
the node side of the event-class match that #658 and #661 both depend on:
a NodeRequirements interface and the rule that the validator matches a node's
requirement against the binding's capability before launch. The validator
rejection list gains the event-class check.
The pin guard asserted existence, so a bump applied to only one of the HTML
plan's two sites still passed while the other kept asserting the old version.
It now checks every printed literal and names the stale line. The block
comment no longer claims a bump is a one-file edit, which is the opposite of
what the check enforces.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: source the developer preview from the README and define the event-class relation for #653
Applies the six findings from the capped review at 6c14e1d4: the HTML §2
obsolete-claims cell attributes the developer-preview phrase to the DeepSeek
Harness README alone; both pages define the event-class comparison as a
partial order (final-text-only below jsonl, rpc incomparable and exact-match)
in identical normative wording for #658; the reasoning table and mirrored HTML
callout use only the Measurement Ownership terms owns/nominated; and the
Markdown page cedes the full HTML-versus-Markdown content map to the
explanation index, keeping a concise pointer plus the download-and-open note
while the problem statement leads the page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: scope the pin definition to preflight and fix config_dir composition for #653
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: carve out persisted-credential auth and de-collide config_dir for #653
The credential/state rule claimed every harness gets a run-scoped state root so
no run reads an operator's real home, but Ultrafuzz's default Codex mode is
`auth = "subscription"`, which reads `CODEX_HOME/auth.json` and falls back to
`~/.codex`. Add invariant 7: a harness whose credential lives inside its own
state root keeps a stable, writable root outside the run scope and declares that
exemption, rather than being rejected as unsupported. Mirror it in plan §5.2 and
the layer-4 box, and map the legacy `[agents.CodexAgent]` keys in the migration
paragraph.
Rename the new seed key to `config_seed_dir`. `[agents.<id>].config_dir` is
already shipped and is exported verbatim as `CODEX_HOME`, so reusing the
spelling for seed-copy semantics would silently change what `CODEX_HOME` means
for every existing Codex profile.
Scope the normative claim to #658 until SPECS.md absorbs the capability
contract, and record the same carve-out in the explanation index so its
"not normative" preamble no longer contradicts both pages.
State the nomination-vs-ownership rule once, in Measurement Ownership, and leave
bare pointers in the comparison matrix, the five per-candidate subsections, the
reasoning table, and Next Work.
Co-Authored-By: Claude Opus 5 <noreply@anthropi…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Queue
Implements priority 8 of #413. This supersedes the remaining control-file work in #165 without merging that obsolete branch. The prerequisite artifact-tail and executable-capability slices already landed as #420 and #421.
Refs #413.
Greptile Summary
The PR seals workflow controls and dependency closures into content-addressed execution snapshots, binds lifecycle operations to authenticated workflow links, hardens Modal handoff and recovery, and migrates recognized stock adapters.
Confidence Score: 4/5
The adapter migration portability failure should be fixed before merging because rerunning init can fail on non-procfs or restricted hosts even though initialization itself is not documented as Linux-only.
The execution sealing and lifecycle recovery paths are extensively guarded, but stock-adapter publication and recovery unconditionally require descriptor pseudo-paths while the adjacent scaffold-write path explicitly supports environments without them.
Files Needing Attention: packages/runtime/src/init.ts
Important Files Changed
Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "Harden init publication and file writes" | Re-trigger Greptile
Context used (4)
@ultrafuzz/cli)