Skip to content

Closeout audit blocks on harness handoffs and declared Node/BAML outputs #1139

Closed

Description

@ViperJuice

Observed

In Consiliency/jev-assembler#1, the STATE gate and CI pass, but phase-loop-closeout-audit --repo . exits 1 on ignored outputs generated by the phase verification workflow. The audit reports .dev-skills/ (harness handoffs), .baml/, .cache/, baml_sdk/, dist/, and generated/baml/ as unknown_ignored. The prior STATE runner ended with dirty_worktree_conflict while its verification had an unrelated BAML gate failure; that BAML failure has since been fixed and the public gate now passes.

Repro: run npm ci, npm run bootstrap:baml, npm test, npm run verify:phase -- STATE, then phase-loop-closeout-audit --repo . in the STATE worktree. The audit exits 1 despite a clean tracked tree. .dev-skills/handoffs is written by the installed execute-phase workflow itself.

Expected

The closeout audit should recognize harness-owned handoffs and support a provenance-checked way to classify declared, deterministic tool outputs without broadly trusting every ignored .cache or dist path. Unknown ignored data must remain fail-closed.

Acceptance

  • .dev-skills/handoffs/** is classified as runner-owned when generated by the harness.
  • Project build outputs can be classified from explicit producer/plan evidence without making arbitrary ignored files safe.
  • Regression covers a passing Node/BAML phase verification that leaves ignored generated SDK, dist, and cache outputs, followed by a successful closeout audit.
  • Unrecognized ignored outputs still block.

Activity

  1. ViperJuice commented on Sep 28, 2026

    @ViperJuice
    CollaboratorAuthor

    Local recovery for Consiliency/jev-assembler#1: moved the nine ignored output groups to a private workspace backup without deleting them, reran phase-loop-closeout-audit --repo . (exit 0), committed the refreshed tracked evidence, and ran phase-loop reconcile --verification-status passed with the passing runner-owned verification.json from the relaunch. STATE is now recorded complete and Consiliency/jev-assembler#1 is ready for review. The classifier gap remains reproducible on a normal Node/BAML verification run, which recreates those outputs.

  2. ViperJuice commented on Sep 28, 2026

    @ViperJuice
    CollaboratorAuthor

    Consiliency/jev-assembler#1 merged with STATE gate and CI passing. For local closeout, we preserved generated BAML/Node outputs and harness handoffs outside the STATE worktree before the audit and worktree prune; this was a workaround, not a classifier fix. The requested provenance-aware classification and regression in this issue remain open.

  3. ViperJuice commented on Sep 28, 2026

    @ViperJuice
    CollaboratorAuthor

    Reproduced again while executing TX for Consiliency/jev-assembler#4. The phase-loop required closeout audit exited 1 and classified ignored generated outputs (.baml/, .cache/, .dev-skills/, dist/, and generated/baml/) as unknown, reporting dirty_worktree_conflict despite phase-owned source changes and captured test results. This is the same classifier behavior seen in STATE and CORE. I preserved the generated outputs outside the worktree. TX has separate substantive acceptance blockers, so this comment is limited to the repeat audit defect; it is not a request to mark TX complete.

  4. ViperJuice commented on Sep 30, 2026

    @ViperJuice
    CollaboratorAuthor

    Picked up on 2026-09-30, and an implementation lane has started. The design is provenance-based and general:

    • Harness handoffs are identified by the harness's own writer record, not by directory name.
    • Project build outputs are accepted only when a declaration in the plan or config covers them AND the runner's evidence ties them to the declared producer run.
    • Anything else stays unknown_ignored and fails closed.

    It includes a Node/BAML-shaped regression test. A draft PR will follow through the governed board. Once it lands, Consiliency/jev-assembler phases shouldn't need the manual move-aside workaround.

  5. ViperJuice commented on Oct 2, 2026

    @ViperJuice
    CollaboratorAuthor

    Observed gap

    While closing Consiliency/jev-assembler#10 in a newly created, initially clean owned worktree, the installed phase-loop-closeout-audit returned exit 1 for actual outputs of frozen npm installation, TypeScript build, the pinned BAML compatibility check and resolver-generated handoffs. Runtime source is release 0.7.21; the same classifier on merged Consiliency/agent-harness main recognizes Python caches and node_modules but has no measured producer mechanism for these outputs.

    Actual buckets: runner_owned .phase-loop/; tool_cache node_modules/; unknown_ignored .baml/.gitignore, .baml/cache/, .baml/profiles-v1.lock, .baml/profiles-v1/, .cache/, .dev-skills/, dist/, generated/baml/. No contents, profile values or credentials were inspected. The target plan and actual sealed commands own/generated these outputs. All current source hashes and runner validation pass.

    The module also ignores --help and audits the current directory instead, returning a failure without usage text.

    Desired bounded fix

    Add supported producer attribution for measured Node/BAML build outputs and resolver-owned handoffs, with deny-by-default path containment. Do not broadly trust arbitrary .cache contents or files/symlinks borrowing a cache name. Keep unknown user/private outputs blocking and add negative cases alongside the real tooling cases. Give --help an actual read-only usage response.

    Current workaround

    Preserve only this run's generated outputs in its declared .phase-loop/runs/... archive, recording original paths and producers; rerun the unchanged classifier. No classifier verdict, evidence seal, credential value or source code is changed. This is a separate runtime usability gap; it does not add source scope to the fixture repair.

  6. ViperJuice commented on Oct 3, 2026

    @ViperJuice
    CollaboratorAuthor

    Fixed by agent-harness#1189, merged as 9135190. Follow-ups are tracked separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions