[aviggiano]

Problem

The controller re-runs the artifact gates on the host after the in-workflow verifier has already accepted an attempt. Whatever the host rejects at that point cannot be retried inside the attempt. The node is recorded as task-output-validation-failure, and its dependents then fail finalization because their prerequisite never got a controller manifest. This PR fixes five cases where valid campaign output was rejected that way:

1. Failed optional generated-tests producer. aggregate-test-files threw artifact verification marker does not exist: .../strategy-b.json when any failure_policy: continue strategy or goal had failed. The verifier had admitted the aggregation without that producer.
2. Dynamic goal producer. The host looked up a materialized goal under its node ID and threw contains unsafe segment "dynamic:threat:<id>".
3. Stale sealed closure below a dynamic group. Every contextual host gate that consults the sealed attempt authority failed on nodes below a dynamic group's direct dependent, with artifact ancestor closure does not match the exact planned set; missing: ".../artifacts/dynamic-...". In the default topology dedupe-findings is the only direct dependent of the goal groups, so by code reading this hits triage, severity-classification, aggregate-test-files and final-report whenever the goal plan expands to at least one goal. I found this while fixing case 2: once aggregation reads through the sealed authority, it hits this check too.
4. Severity matrix. severity-classification failed with 3× SEVERITY_MATRIX_FIELD_MISSING for a schema-valid false-positive record. The schema requires severity, impact and likelihood only for true positives.
5. Coverage prose. Final-report and coverage nodes failed on ordinary sentences such as Recon reached 85% line coverage on Vault.sol. (in a report.json note) or Handlers reachable: 7/9 (in coverage-report.md), which the scanner reported as UNSCOPED_COVERAGE_PERCENTAGE/FRACTION/SCORE. When no coverage producer was planned or admitted, the same sentence in report.md also failed as REPORT_COVERAGE_EVIDENCE_UNPLANNED. That covers every smoke run, which plans no coverage producer, and default runs whose failure_policy: continue stateful-invariant-coverage failed.

Root cause

Each case is host-side logic that disagrees with the verifier's authority or with the schema:

• Cases 1, 2 and the fan-out attribution. aggregation-semantic-context.ts walked every planned transitive ancestor, keyed by planned node ID, with its own marker, manifest and prerequisite-chain authentication. The verifier instead admits producers through the attempt's sealed dependencyArtifactDirs and its persisted optional-dependency admission, keyed by attempt ID. The same walk also attributed bundles from multi-model fan-out producers to the model attempt index. The verifier uses the loop attempt index (workflow.tsx dependencyTask.metadata.loop.attemptIndex), so for model indexes above 0 any manifest one gate accepted the other rejected.
• Case 3. lowerTaskDynamicDependencies (dynamic-runtime.ts) adds a group's generated attempts only to its direct dependents' sealed tasks. The lowered planned graph, however, reaches those attempts from every deeper descendant. assertExactSealedAttemptAuthority required the two closures to be equal.
• Case 4. severity-matrix.ts ran only on the host and required all three fields on every record.
• Case 5. The natural-language score scanner runs only on the host, and agents cannot self-check against it. Without an admitted coverage producer, verifyFinalReportCoverageEvidence also counted any coverage-context score in report.md as invented coverage evidence.

Change

• fix(runtime): build host aggregation sources from the sealed attempt authority
  • Aggregation now gets its producers from finalizedDeclaredContractProducers(layout, "ultrafuzz/generated-tests@3", node, attemptAuthority). That walks the sealed closure, skips optional producers that the persisted admission omitted, maps each attempt to its planned node, and reads each producer through loadFinalizedNodeOutputSnapshot.
  • aggregation-semantic-context.ts shrinks from 483 to 92 lines. It now only projects bundles, attributed to node.loop.attempt_index, which the task-manifest validator binds to the verifier's value.
  • The bespoke authentication chain and its 725-line test file are deleted. The canonical reader has its own coverage in verified-output.test.ts.
  • The test fixture sealer no longer gives deferred dynamic templates workflow bindings; real graphs never carry them and the task-manifest validator rejects them.
• fix(runtime): accept the sealed closure dynamic lowering gives deeper descendants
  • A generated ancestor's attempts are now expected only if the consumer's sealed closure names them. This applies to any consumer, not only deeper descendants. By code reading, lowering adds them to a direct dependent's closure, so in practice only deeper descendants' closures omit them.
  • Static ancestors must still all be present, and nothing outside the planned closure is accepted. The existing fan-in omitted-ancestor and unrelated-ancestor checks still fail as before.
• fix(runtime): delete the host-only severity matrix gate
  • Removes severity-matrix.ts, its call site, its export and its tests.
  • Everything it checked is still enforced, and each check below runs in the in-workflow verifier too, so a mismatch can be retried:
    • Both schemas are additionalProperties: false with High/Medium/Low enums.
    • The severity-classification-matrix registry gate checks the matrix on every classified record that carries all three fields.
    • Final-report issue rows are matrix-checked in every classification mode, smoke's bounded mode included. projectCanonicalFinalReport → assertCanonicalIssuePresentation (final-report-markdown.ts) requires severity, impact and likelihood on each issue and throws when severity differs from the impact × likelihood result. The verifier runs that projection on every task that declares a report@3 output (verifyFinalReportCanonicalProjection in workflow.tsx). The test canonical final-report validation rejects presentation drift instead of repairing it pins the rejection.
    • In strict severity-handoff mode, report rows must copy those fields exactly from the classified findings.
• fix(runtime): report unscoped coverage prose scores as warnings
  • UNSCOPED_COVERAGE_SCORE, _PERCENTAGE and _FRACTION are now warnings. This is the stop-gap from the spec.
  • These stay errors: the typed coverage_evidence deep-equality check, the canonical scoped-section projection (including a duplicated or contradicting scoped score anywhere in the document), the stylesheet and select-control checks, and the 2,048-candidate scan limit.
• fix(runtime): keep unscoped report prose advisory when no coverage producer was admitted (review round)
  • Without an admitted coverage producer, REPORT_COVERAGE_EVIDENCE_UNPLANNED now counts only report.md scores bound to an exact declaration-completeness scope. A typed report.json.coverage_evidence still fails, and so does a scoped score such as recon-selected-declaration-completeness: `1/1` . The final-report prompt already tells agents to omit the section when no producer is selected.
• fix(prompts): keep the coverage projection partial identical to its reference docs (review round)
  • The earlier docs commit edited only docs/reference/artifacts-reports.md, and packages/prompts/test/semantic-anchors.test.ts fails when that page does not contain coverage-evidence-markdown.mdx verbatim. It failed on the previous head (expect(docs).toContain(partial)).
  • The agent-facing partial also still said bare scores are rejected. It now says publication rejects missing, duplicated, reordered, or contradicting scoped scores and warns about scores that name no exact scope.
  • The docs page carries the partial verbatim again, followed by a docs-only paragraph on the scan limit and on reports without an admitted producer.
• test(runtime): pin host aggregation attribution for model-fanout producers adds the discriminating test for the attempt-index change.
• test(runtime): pin aggregation finalization and implicit-visibility gate results (review round)
  • Adds a regression guard: an admitted generated-tests producer without its verification marker fails aggregation (REQUIRED_ARTIFACT_INVALID). Before, the deleted test file covered this.
  • Restores a result assertion in the implicit-visibility coverage loop: only its two stylesheet cases fail.
• CHANGELOG entry, corrected in the review round.

Net: packages/runtime/src +90/−649, tests +601/−888.

Deliberately not built (and why)

• Fixing the dynamic lowering itself, so deeper descendants' sealed closures include generated attempts. That would rewrite in-flight runs' sealed task plans, which verifyDynamicRuntimeMaterialization re-derives and fingerprints. It would also widen what descendants must admit and change their prompt authority.
  • Consequence, unchanged by this PR: in the default topology, aggregate-test-files does not aggregate the dynamic goal hunters' generated tests. Neither the verifier nor the agent's prompt sees them either.
  • Follow-up.
• The same planned-closure equality on Modal cloud workers. packages/modal/src/node-provider.ts ~2776-2807 still has it; I did not touch the modal package. Code reading suggests a cloud-executed deeper descendant fails there the same way; I did not run it.
• A replacement severity-matrix check for bounded-mode reports. None is missing. The canonical report projection already rejects a mismatched final-report row inside the verifier (see Change). An earlier revision of this PR and its CHANGELOG entry said bounded-mode rows were no longer matrix-checked. That was wrong, and both are corrected.
• Requiring generated attempts in a direct dependent's closure. That would keep the stricter check for the one consumer shape lowering does extend. It adds a condition to a fail-closed check for no stability gain, so I left it out.
• Deleting the prose scanner (~1,300 lines) or downgrading its scoped-score errors. That is the owner decision the spec reserves. Scores that name an exact scope stay errors in two cases:
  • Outside the canonical section, they must match the typed evidence.
  • Without an admitted producer, any such score in report.md fails as REPORT_COVERAGE_EVIDENCE_UNPLANNED.
• Un-exporting assertGeneratedTestManifestSemantics (packages/artifacts/src/generated-tests.ts). Its only caller outside the artifacts package was the deleted aggregation code. That file belongs to the artifacts dead-code item, so I left it there.

Verification

Discriminating evidence. For each test I ran the test file against the origin/main runtime sources (git checkout origin/main -- packages/runtime/src, recompile, run, restore). Every test below fails there with the production message and passes on this branch:

Test (artifact-gates.test.ts)	Result on origin/main
host aggregation intake skips a failed optional generated-tests producer the verifier did not admit	artifact verification marker does not exist: .../strategy-b.json
host aggregation intake keys a directly consumed dynamic producer by its storage attempt ID	generated-test aggregation source artifact directory contains unsafe segment "dynamic:threat:t1"
host aggregation intake follows the sealed closure below a dynamic group's direct dependent	same unsafe-segment error; with only the first commit applied it fails with artifact ancestor closure does not match the exact planned set; missing: ".../dynamic-goal-template-…"
host aggregation intake attributes model-fanout bundles to the loop attempt index the verifier uses	Aggregation source bundle attribution does not match authority ["strategy-a__model_1__attempt_1",…]
severity classification keeps a false-positive record without severity, impact, or likelihood	3× SEVERITY_MATRIX_FIELD_MISSING
coverage gate binds selected and unselected ranges… (rewritten prose cases, plus the new Handlers reachable: 7/9, Target: 90% of Recon-selected declarations. and After iteration 2 we covered 41 of 57 functions.)	UNSCOPED_COVERAGE_* with severity error
final report preserves typed coverage evidence… (rewritten prose cases, plus the new report.json notes Recon reached 85% line coverage on Vault.sol. and Coverage of withdraw() was 3/4 branches in the replay.)	UNSCOPED_COVERAGE_* with severity error
producer-free final reports require the exact not-planned implementation coverage (new unscoped report.md sentences)	UNSCOPED_COVERAGE_* errors plus REPORT_COVERAGE_EVIDENCE_UNPLANNED
final reports disclose planned but omitted implementation coverage without consuming failed output (new unscoped report.md sentence, coverage producer planned but not admitted)	UNSCOPED_COVERAGE_* errors plus REPORT_COVERAGE_EVIDENCE_UNPLANNED

• Review-round tests against the previous head. The last two rows also fail on the previous head d0091aba, where the only error left is REPORT_COVERAGE_EVIDENCE_UNPLANNED. They pass with 99c2f7ca. The existing scoped recon-selected-declaration-completeness: `1/1` case in the producer-free test still fails with REPORT_COVERAGE_EVIDENCE_UNPLANNED.
• The finalization guard is not a behaviour change. On origin/main sources, a probe with the same fixture minus the failed producer rejects an admitted producer without a marker with artifact verification marker does not exist: .../strategy-a.json. On this branch it fails with REQUIRED_ARTIFACT_INVALID.
• The transitive-dynamic test uses the sealed-closure shape the real compiler produces. I checked that shape with a script that calls startRun and then materializeDynamicRuntime on a planner → dynamic fan-out → join → triage-shaped grandchild topology:
  • The grandchild's sealed closure was [planner, strict-join]; the lowered graph's was [dynamic:threat:…, planner, strict-join].
  • assertSmithersTaskManifestMatchesPlannedGraph accepted the materialized state.
  • On main, verifyRequiredArtifactsForAttempt failed on the grandchild with the closure mismatch.
  • With this branch it gets past that check and stops at the script's own unfinalized join. I did not extend the script to finalize the whole chain.
• Rewritten coverage tests. The scanner still reports every case, now as a warning. Cases whose documents also carry a real canonical-section or stylesheet error still fail.

Commands run on the final head (266dafa0, on b6dd1da9):

• pnpm -w build.
• node --test dist-test/test/artifact-gates.test.js dist-test/test/verified-output.test.js dist-test/test/terminal-report-completion.test.js dist-test/test/final-report-markdown.test.js: 275/275.
• packages/cli: --test-name-pattern '^agent-owned bytes stay identical across validation, sync, aggregation'. This runs sync, the host aggregation gate and the verified reads end to end; it passes.
• packages/prompts: npx vitest run: 104/104, including semantic-anchors.test.ts, which fails on d0091aba.
• CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci: clean.
• npx eslint and npx prettier --check on the changed files, and git diff --check: clean.
• pnpm --filter @ultrafuzz/runtime typecheck and pnpm --filter @ultrafuzz/prompts typecheck: clean.
• pnpm -w knip: clean.
• node scripts/docs-check.mjs, node scripts/prompt-catalog-docs.mjs --check and node scripts/audit-profile-docs.mjs --check: clean.

Not run: the runtime.test.ts shards, the full CLI suite, generated-workflow-verifier.test.ts, the Bun adapter contracts, the modal package, and any end-to-end campaign.

Risk / compatibility

• No change to schemas, contract descriptions, or the validator-build inputs, so validator_build does not rotate and in-flight runs keep validating.
• Prompt text changes. coverage-evidence-markdown.mdx is rendered into the stateful-invariant coverage and final-report prompts, so runs planned after this change see the new sentence. It is not a validator-build input.
• Public exports: @ultrafuzz/runtime no longer exports expectedSeverityFromMatrix, validateSeverityMatrixArtifact, SeverityLevel or SeverityArtifactKind. authenticatedAggregationSemanticContext now takes { layout, attemptId, producers } and a new AggregationSourceProducer type is exported. Nothing in the repo besides artifact-gates.ts used them, and the generated workflow.tsx imports none of them.
• The host is more permissive in three places:
  • Unscoped prose scores no longer block, with or without an admitted coverage producer.
  • Non-true-positive severity records that omit severity, impact and likelihood no longer fail on the host. The schema already allows this.
  • A consumer's sealed closure may omit a generated ancestor's attempts. Static ancestors are still required and unplanned directories still rejected.
• The aggregation host gate now also runs assertExactSealedAttemptAuthority and requires admitted producers to be controller-finalized. Other contextual gates already required both, and host finalization already requires admitted direct prerequisites to be finalized before the gate runs, so I expect no new rejection in real runs. That expectation comes from reading the code, not from running a campaign.
• Shared-file overlap:
  • Hunks in artifact-gates.ts sit near w05b's areas but do not overlap its listed ranges.
  • The artifacts-reports.md hunk stays within the scoped-coverage section, which w11 also edits elsewhere.
  • coverage-evidence-markdown.mdx is not in w12's prompt list.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge, though its release-note entry should be restored.

Findings

1.  Bounded report severities go unchecked ▶

Fix with agent prompt

### Issue 1
packages/runtime/src/artifact-gates.ts:undefined-421
In a bounded topology such as `smoke`, the final report assigns each promoted finding’s severity, impact, and likelihood without an upstream severity-classification artifact. Removing this host check leaves no matrix check for those rows. A report can therefore publish `High` severity with `High` impact and `Low` likelihood, even though the matrix yields `Medium`. Keep the check in the verifier-accessible bounded-report path so contradictory severity cannot be published.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR aligns host artifact gates with sealed attempt admission, relaxes host-only checks that rejected valid severity and coverage output, and updates coverage guidance.

• Aggregation now reads finalized producers admitted through the sealed attempt authority.
• Dynamic descendants may omit generated attempts absent from their sealed closure.
• The Unreleased changelog entry for these changes was removed.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Sealed attempt closure] --> B[Admitted producer attempts]
  B --> C[Finalized output snapshots]
  C --> D[Aggregation semantic context]
  D --> E[Host artifact gate]

Loading

_{Reviews (3) · Last reviewed commit: "chore: move the changelog entry to the c..."}

[greptile-apps]

Bounded report severities go unchecked

In a bounded topology such as smoke, the final report assigns each promoted finding’s severity, impact, and likelihood without an upstream severity-classification artifact. Removing this host check leaves no matrix check for those rows. A report can therefore publish High severity with High impact and Low likelihood, even though the matrix yields Medium. Keep the check in the verifier-accessible bounded-report path so contradictory severity cannot be published.

Knowledge Base Used: Artifact management

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/artifact-gates.ts
Line: 421

Comment:
**Bounded report severities go unchecked**

In a bounded topology such as `smoke`, the final report assigns each promoted finding’s severity, impact, and likelihood without an upstream severity-classification artifact. Removing this host check leaves no matrix check for those rows. A report can therefore publish `High` severity with `High` impact and `Low` likelihood, even though the matrix yields `Medium`. Keep the check in the verifier-accessible bounded-report path so contradictory severity cannot be published.

**Knowledge Base Used:** [Artifact management](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/artifact-management.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.