dedupe-findings fails its artifact contract on every smoke-lane target #348

Activity

1. aviggiano

   added

   bugSomething isn't workingSomething isn't working

   on Aug 7, 2026
2. aviggiano

   added a commit that references this issue on Aug 7, 2026

   fix(runtime): re-seal the verification marker after normalizing a pro…

   ...

   94d3291
3. aviggiano

   closed this as completedin #350on Aug 7, 2026
4. aviggiano commented on Aug 7, 2026

   

   aviggiano

   on Aug 7, 2026

   CollaboratorAuthor

   More actions

   Fixed by PR #350 for the dedupe-findings cause. Root cause was not the node's output at all: normalizeFindings rewrites a producer's findings.json in place after the verification marker has already sealed the pre-normalization bytes, so every dependent re-hashes against a stale digest and assertVerifiedDependency fails closed. Measured drift on a contract-valid finding: 02826071… → 32741e5f….

   It is dedupe-findings and only dedupe-findings because it is the sole smoke node whose dependencies declare findings.json. An empty array normalizes back to byte-identical [], which is why the empty-artifact fixtures never caught it.

   Reopening scope for the two Foundry-only failures. externalized-state-accounting and lifecycle-view-boundaries do not share this cause — their dependency is smoke-context, which publishes only Markdown and is never rewritten. Their category implies agent-succeeded-then-verifier-failed; the likeliest candidate is the generated-test companion path (verifyGeneratedTestFiles / materializeGeneratedTestCompanions), which a Foundry lane exercises and the other two toolchains largely do not. Not investigated to a conclusion.

   The third acceptance criterion — scoring_ready: true for at least one target — still needs a real smoke run.
5. aviggiano

   reopened this on Aug 7, 2026
6. aviggiano commented on Aug 7, 2026

   

   aviggiano

   on Aug 7, 2026

   CollaboratorAuthor

   More actions

   Your first acceptance criterion — "The specific contract assertion `dedupe-findings` violates is
   identified and quoted" — cannot currently be met from CI artifacts. I tried, and filed #361 for the
   reason.

   I pulled the bundle from run 31199189053 and every entry of `rows[].failed_nodes[]` has exactly four
   keys: `node_id`, `status`, `timed_out`, `failure_category`. Searching the whole published bundle,
   including `analysis/analysis-bundle.json`, returns zero occurrences of any error text. So the
   published evidence is:

   very-liquid-vaults-foundry   dedupe-findings  artifact-contract
   venus-isolated-pools-hardhat dedupe-findings  artifact-contract
   stableswap-ng-vyper          dedupe-findings  artifact-contract
   

   and nothing more. The message does exist — state.json on the run's volume carries a per-node
   last_error — but it is dropped by both the attempt ledger and the diagnostics collector. Whoever takes
   this will need volume access to those three smoke runs, or #361 fixed first.

   One structural note that may help when the text is in hand. materializeMissingDedupeArtifact in the
   generated workflow treats an existing deduped-findings.json as acceptable only when it validates and
   value.length > 0; an empty-but-valid [] falls through to the dependency-recovery path, which then
   writes to the mirror root. That is a plausible place for a contract failure to originate on targets
   that legitimately produce no findings, and all three smoke targets are exactly that case. I have not
   confirmed it — I am flagging it as where I would look first, not as a diagnosis. The node also declares
   its primary as either ultrafuzz/json-array@1 or ultrafuzz/findings@1 while the function validates
   with ultrafuzz/findings@1 unconditionally, which is worth checking against what the topology actually
   declares for the smoke-lane targets.
7. aviggiano

   added a commit that references this issue on Aug 7, 2026

   Merge origin/main into bench/deepseek-v4-flash-results

   ...

   43a32b4
8. aviggiano commented on Aug 7, 2026

   

   aviggiano

   on Aug 7, 2026

   CollaboratorAuthor

   More actions

   Residual diagnosis for the two Foundry-only failures. I could not identify a single cause, and I am not going to name one. Here is what I ruled out, what I found instead, and what would settle it.

   Correction to the framing

   The comment above says the category "implies agent-succeeded-then-verifier-failed". That is not what artifact-contract means. finalizeTerminalTask (packages/runtime/src/workflow-sync.ts:2035) stamps it whenever evidenceSource is "verifier" or "preparation" — the wrapper on either side of the agent. So the category alone does not tell us the agent ran. That widening is what #350 relied on for dedupe-findings, and it applies here too.

   Ruled out

   1. The fix(runtime): re-seal the verification marker after normalizing a producer's findings #350 cause. Confirmed not shared. smoke-context publishes one Markdown output and nothing rewrites it, so assertVerifiedDependency cannot go stale for these two the way it did for dedupe-findings.
   2. A per-node topology difference. There is none. All four smoke strategy nodes — time-warp-sequences, external-dependency-boundaries, externalized-state-accounting, lifecycle-view-boundaries — are byte-identical in benchmarks/smoke-benchmark.yml: same prompt (smoke/smoke-strategy.md), same depends_on: [smoke-context], same two outputs (findings.json primary, generated-tests.json). Two failed, two did not. The discriminator is therefore what the agent wrote, not how the node is configured.
   3. The concrete/logical node-id mismatch in the companion path. This is a real defect and I have fixed it in fix(runtime): accept the logical node directory generated tests are mandated to use #362 — materializeGeneratedTestCompanion searched test/foundry/<concreteNodeId>/ while the prompt mandates test/foundry/<logicalId>/. It affects 57 of the 65 generated-test nodes in the production topology (strategies group has loops: 3). It cannot be the smoke cause: the smoke lane pins strategy_loops: 1, so all four concrete ids equal their logical ids and the mandated directory was searched. I verified this both by expanding both manifests and by running the fixture both ways.
   4. A cross-sandbox artifact boundary. Not applicable. The smoke lane is one Modal worker per target running eval run --provider none locally (public-worker.ts:192); nodes take the Worktree branch, not the Sandbox branch, so there is no sandbox→orchestrator artifact round-trip that could drop a companion file.
   5. The materializeMissingDedupeArtifact value.length > 0 path flagged above. It is scoped to deduped-findings.json, which only dedupe-findings declares. Neither of these two nodes reaches it.

   What I did instead of guessing

   I built a harness that runs the real workflow.tsx — placeholders substituted, JSX tail removed, smithers-orchestrator resolved — over a temporary run root with a git-backed workspace, and drove the actual post-agent chain (prepareArtifactMirror → normalizeLegacyGeneratedTestManifests → materializeGeneratedTestCompanions → verifyArtifacts twice, matching the inline call and the separate verifier Task). Then I enumerated what an agent could plausibly have written for a node with exactly these two outputs.

   Passes: test mirrored into the artifact directory; test only at test/foundry/<node>/; nested subdirectory under it; legacy string-list manifest; manifest absent entirely (preparation's empty mirror manifest is used); artifact copy differing from the workspace copy; entries carrying agent-written size_bytes/sha256.

   Fails, each with artifact-contract:

   what the agent wrote	error
   test in the repo's conventional test/, not test/foundry/…	generated test file is missing generated-tests/Esa.t.sol
   same file in two accepted roots with different bytes	generated test sources conflict generated-tests/Esa.t.sol
   manifest entry as an absolute path	path must use the generated-tests/<file> prefix…
   manifest entry as ./generated-tests/…	same
   schema_version: "1"	Invalid input: expected "1.0"
   run_id/node_id omitted	expected string, received undefined

   Every one of these is Foundry-shaped in practice: on venus-isolated-pools-hardhat and stableswap-ng-vyper the prompt forbids installing dependencies, so an agent has no runnable native harness and writes the empty manifest — which is trivially valid and cannot trip any of these. On a Foundry target forge is present, so agents actually author .t.sol files and populate the manifest. That is the toolchain asymmetry, and it explains "Foundry-only" without selecting between the rows above.

   One asymmetry worth noting on its own: verifyArtifacts picks the first artifact root where the file exists and then validates, while normalizeLegacyGeneratedTestManifests and materializeGeneratedTestCompanions iterate roots until the contract validates. So a malformed generated-tests.json in the canonical directory fails hard even though preparation's valid empty manifest is sitting in the mirror, whereas an absent one falls back and passes. I have not changed this — it is a deliberate strictness question for a maintainer, not a bug I can assert.

   What would settle it

   Any one of these, in order of cost:

   1. A failed node's error message never leaves the volume: the ledger and the eval diagnostics keep only failure_category #361. One last_error string per failed node in the diagnostics bundle picks the row out of the table above immediately. Every candidate above produces a distinct, quotable message. This is the right fix and it is already filed.
   2. Volume access to the three smoke runs' state.json, which already carries nodes[<id>].last_error. Same information, available today, no code change — but the volume is rm -rf'd with the sandbox, so it must be captured from a new run.
   3. The row's generated-tests.json and generated-tests/ directory retained per row in the published bundle. fix(runtime): accept the logical node directory generated tests are mandated to use #362's sibling PR does exactly this for the threat-model artifacts; the same mechanism would cover these.

   Without one of those, naming a cause here would be a guess dressed as a diagnosis. The acceptance criterion "the specific contract assertion is identified and quoted" is, as you found, not currently satisfiable from CI artifacts — and that is true for these two nodes as much as for dedupe-findings.
9. aviggiano

   added 2 commits that reference this issue on Aug 7, 2026

   fix(runtime): accept the logical node directory generated tests are m…

   ...

   fcf3573

   Merge origin/main into release/v0.1.0

   ...

   8f367b9
10. aviggiano commented on Aug 7, 2026

    

    aviggiano

    on Aug 7, 2026

    CollaboratorAuthor

    More actions

    No longer reproducing — all three smoke targets now pass their artifact contracts

    I pulled public-eval-diagnostics.json from the three most recent Modal Eval Benchmarks runs. The failure table in the issue body is gone.

    run	branch	commit	targets with failed nodes	scoring_ready
    31220226212	main	5db8c6f	none — 0 failed nodes across 3/3	true
    31220479878	release/v0.1.0	8f367b9	venus, stableswap (final-report, agent-failure) — not the nodes in this issue	false
    31220736791	test/v0.1.0-ultrafuzz-bench	426ff5f	none — 0 failed nodes across 3/3	true

    Specifically, in all three runs there is no dedupe-findings failure on any target, and no externalized-state-accounting or lifecycle-view-boundaries failure on very-liquid-vaults-foundry — that target reports workflow_status: "succeeded", failed_nodes: [], scoring_ready: true in every one of them, including the run that failed overall.

    No row in any of the three carries failure_category: "artifact-contract".

    Acceptance criteria

    • The specific contract assertion dedupe-findings violates is identified and quoted — addressed by fix(runtime): re-seal the verification marker after normalizing a producer's findings #350 (re-seal the verification marker after normalizing a producer's findings).
    • A regression covers it — fix(runtime): re-seal the verification marker after normalizing a producer's findings #350 and fix(runtime): accept the logical node directory generated tests are mandated to use #362 each shipped one.
    • The smoke lane produces scoring_ready: true for at least one target — it produced true for all three, in two independent runs 8 minutes apart.

    The two Foundry-only nodes track to #362 (accept the logical node directory generated tests are mandated to use), which landed at fcf3573 and is present in all three commits above.

    The one remaining failure is a different thing

    release/v0.1.0 @ 8f367b9 failed on final-report with failure_category: "agent-failure" on 2 of 3 targets, and terminal_report_present: false on venus-isolated-pools-hardhat — reason code terminal-report-missing. That is a model/agent-level outcome at the report node, not an artifact-contract violation, and it did not recur in either adjacent run on the same tree. Details in #183.

    Recommend closing. Refs #319, #327, #183.
11. aviggiano commented on Aug 8, 2026

    

    aviggiano

    on Aug 8, 2026

    CollaboratorAuthor

    More actions

    The discriminator run was cancelled and is unusable

    I rechecked run 31264665386, the main push at 70646d2 that first contained #407.

    • launch succeeded and persisted the one Luna smoke sandbox.
    • During collect, status inspection repeatedly failed with Error: incompatible Modal checkpoint: image fingerprint mismatch; the preserved control artifact records only status-query-failed and leaves the launch in phase launched with an active/unknown recovery lifecycle.
    • The next main push run, 31269522928, started at 17:28:53 UTC. With branch concurrency cancellation enabled, run 31264665386 reported The operation was canceled at 17:29:06 and concluded cancelled.
    • Its final artifacts contain the plan, launch state, and control diagnostics only. There is no public-results artifact, outcome document, public-eval-diagnostics.json, or failed_nodes evidence.

    Therefore this run cannot answer the discriminator question: it provides no evidence that lifecycle-view-boundaries either passed or recurred. The absence of a failure row is an absence of collected evidence, not a pass. Keep #348 open until a non-cancelled main smoke run produces usable diagnostics for all three rows.

    The independent recurring final-report / agent-failure / terminal-report-missing signature from the two preceding completed runs is now tracked separately in #416.
12. aviggiano commented on Aug 9, 2026

    

    aviggiano

    on Aug 9, 2026

    CollaboratorAuthor

    More actions

    Acceptance verified on a non-cancelled main smoke run

    Run 31304412920 executed attempt 1 at exact main SHA cb672c67bbe1255a0c96dc2e89653e7422f2246b and completed through the ordinary launch/collect matrix gate. Immutable evidence: results artifact 9035716983 / sha256:e2b1394ec826588edb1916d5c237b6e20b2293ae89e20151f3c722f68f2d65cd; control artifact 9035717118 / sha256:bdaa5a84078775d03247c2263b006c11417e26b1af28f8aaa4cba8b6c7d88a9c.

    The original dedupe-findings cause and regression are in #350 (94d3291): runtime findings normalization changed bytes after sealing, so assertVerifiedDependency hit verified dependency artifact changed findings.json; the fix re-seals only that runtime-authored rewrite and tests the downstream marker boundary.

    The fresh retained v2 diagnostics contain exactly the Foundry, Hardhat, and Vyper smoke rows: workflow_succeeded: 3, workflow_failed: 0, three terminal reports, summary.scoring_ready: true, and 0 failed nodes. Thus neither dedupe-findings nor the residual Foundry artifact-contract signatures reproduce. The exact evidence-selector failures exposed after bounded messages shipped were fixed with regressions in #446.

    Historical bound: the pre-#415 residual rows did not retain messages, so this does not retroactively assign them a byte-level cause; it establishes the quoted dedupe cause and a clean, fully collected non-recurrence on current main.

    Closing as accepted. #319 remains open for its separate three-adjacent-run stability condition.
13. aviggiano

    closed this as completedon Aug 9, 2026