Skip to content

Closeout audit cannot attribute preserved review and operational evidence under .dev-skills #1772

Open

Description

@ViperJuice

Problem and impact

SAFEDEBUG in EzBidPro is implemented, independently reviewed, tested, and merged, but phase-loop-closeout-audit --phase SAFEDEBUG cannot classify preserved workflow review/evidence artifacts under .dev-skills/. They are legitimate workflow outputs, but there is no supported provenance path for them. The executor correctly keeps the phase blocked with dirty_worktree_conflict.

Please add a bounded, provenance-checked way to retain these artifacts without broadly trusting .dev-skills, deleting evidence, moving it into .phase-loop to borrow runner trust, or rerunning historical model reviews merely to manufacture new outputs.

Confirmed implementation gap

Observed installed release: 2026.10.16-grok.1-f9c18ab78c0b. Also checked the relevant files directly from GitHub main at 0fcc004bf105e023e9376c4c2ad5e31052cab4f2 on 2026-10-19; the relevant behavior is still present.

  • closeout_classifier.py::_grade_by_provenance enumerates unknown ignored directories, then tries path classification, is_harness_handoff, and verify_declared_output. There is no review/evidence attribution route.
  • generated_outputs.py::_RESERVED_ROOTS includes .dev-skills; _safe_pattern rejects producer declarations covering it. Keep that broad prohibition intact.
  • generated_outputs.py::is_harness_handoff (line 786 at that revision) accepts validated handoff contracts, not review bundles, model outputs, or operational receipts.
  • load_declaration (line 214) reads declarations from HEAD; adding an uncommitted declaration is not a workaround, and even a committed declaration cannot cover the reserved root.
  • Tests already enforce important negative behavior in phase-loop-runtime/tests/test_closeout_generated_outputs.py: test_other_files_under_dev_skills_block, test_unbounded_or_reserved_globs_are_rejected, test_a_generated_file_edited_after_the_producer_ran_blocks, and handoff repo/commit identity tests. Extend these protections rather than relaxing them.

This is not merely a collapsed-parent ignore problem: the actual audit enumerates all 95 leaf files and still cannot attribute them.

Measured reproduction

Read-only commands used on 2026-10-17:

phase-loop-closeout-audit --repo "$UMBRELLA_ROOT" --phase SAFEDEBUG
phase-loop-closeout-audit --repo "$EXTENSION_ROOT" --phase SAFEDEBUG
Tree Exit Results
EzBidPro/EZBidPro umbrella 1 95 unknown_ignored, plus recognized runner output and tool cache
EzBidPro/navblue-extension component 0 15 declared_output, 0 unknown

Unknown umbrella files are exactly:

  • 3 JSON files under .dev-skills/evidence/SAFEDEBUG/: interactive-receipt.json, safedebug-ablation-matrix.json, safedebug-prod-behavior.json.
  • 92 historical artifacts under .dev-skills/reviews/roadmap-v7-20261014T063025Z/, including review input bundles, result JSON/logs, retries, failed/empty outputs, and explicitly labelled stand-in reviews.

The independent Sol audit hashed all 95 files. Thirteen explicit internal digest bindings matched; zero mismatched. Seventeen files had full-digest references in inspected metadata; 78 did not. No symlinks or unexpected output families were found. Subsequent verification confirmed the original 95 file hashes remained unchanged.

These facts prove consistency, not authenticated historical authorship. There is no immutable original baseline for every historical file. A solution must not pretend that hashing old files today proves who produced them.

Local maintainer reproduction paths (metadata only, no capture data):

  • Umbrella: /mnt/workspace/worktrees/viperjuice/EZBidPro-debug-release-roadmap-v7
  • Component: /mnt/workspace/worktrees/viperjuice/navblue-extension-safedebug-v7
  • Per-file inventory: umbrella .phase-loop/runs/safedebug-sol-audit-20261017/readonly-audit.json
  • Latest repair report: umbrella .phase-loop/runs/safedebug-ec5-repair-20261017205131Z/repair-report.json

These local artifacts are not available from a fresh clone. Tests should reproduce the failure using synthetic files, not copy private captures or local review transcripts into fixtures.

Minimal synthetic regression shape

  1. Create a temporary Git repo with .gitignore containing /.dev-skills/ and .phase-loop/; commit normal fixture inputs.
  2. Write a valid harness handoff and synthetic review/evidence artifacts through the supported producer path being added.
  3. Include a known failed review artifact and an empty output: retention must not imply successful review.
  4. Run the actual audit, including Git's collapsed-directory reporting and leaf enumeration.
  5. Verified artifacts should be attributable; an arbitrary sibling .dev-skills/notes.txt must still block.

Today steps 2–4 have no supported review/evidence producer record to use, and review/evidence leaves remain unknown even when referenced by a plan/evidence document.

Suggested implementation direction (design proposal, not a mandated schema)

Introduce a narrowly scoped artifact-provenance record for harness review/evidence producers and an explicit attribution branch in the classifier. Reuse existing sealing/path/identity helpers where appropriate. Bind exact paths and SHA-256 values to a repository identity, producing run/task and artifact kind, and the relevant phase or explicitly consumed upstream review. Record producer identity at creation, rather than accepting a hand-written claim merely because it contains matching hashes.

Important design distinctions:

  • Retention provenance versus acceptance: a failed, degraded, empty, or stand-in review can be valid retained history while providing no successful review vote. Classification must never upgrade its verdict.
  • Historical adoption versus original authorship: define an explicit, auditable adoption route for existing artifacts lacking producer receipts, or continue rejecting those files with an actionable reason. Any authorized adoption records who accepted custody, when, exact digests, and known limitations; it must not backdate or invent producer identity. A self-authored digest manifest alone must not confer trust.
  • Cross-phase evidence: roadmap reviews predate SAFEDEBUG. Allow an explicit verified dependency relation if appropriate; do not require rewriting history to label every artifact as produced by the consuming phase.
  • Commit changes: reviewed artifacts can legitimately outlive a code commit. Bind them to their actual reviewed inputs and consumption record. Do not weaken the existing HEAD/phase binding of deterministic build-output records.
  • Privacy: audit metadata and diagnostics should identify paths/digests/reasons without echoing evidence contents or secrets. No blanket recursive content import.

Likely touch points: closeout_classifier.py, a dedicated or shared provenance helper near generated_outputs.py, actual review/evidence producer write paths, corresponding runtime tests, and operator/skill documentation for existing artifacts. Locate the real producer paths before selecting final ownership; do not patch only the consumer and leave producers unable to emit accepted records.

Acceptance and negative tests

  • Actual CLI audit accepts correctly recorded synthetic review/evidence artifacts under parent-directory ignore rules, with a specific explanatory attribution reason.
  • Modified bytes, missing receipt/artifact, malformed/unsupported record, wrong repo, wrong producing task/run, unapproved historical adoption, or absent phase/dependency linkage fail closed with useful diagnostics.
  • Reject path traversal, absolute/outside-repo paths, symlink escape, and inappropriate cross-repo receipt reuse. Cover relevant case-insensitive reserved-path behavior.
  • Unregistered siblings under .dev-skills still block; .dev-skills/** remains forbidden as a generic producer declaration.
  • Failed/empty review artifacts may be retained without becoming successful reviews or acceptance evidence.
  • Existing handoff, deterministic build-output, stale-record, and unknown-output tests stay green.
  • Artifact capture/recording failures are surfaced; no partial registration silently blesses unmatched files.
  • A read-only audit does not move, rewrite, delete, rerun, or relabel historical evidence.
  • Provide a documented migration/adoption procedure and clearly report any historical files that still cannot be attributed. Passing all 95 without adequate evidence is NOT an acceptance requirement.
  • After a reviewed release is installed, rerun the actual EzBidPro audits; close SAFEDEBUG only if its remaining requirements genuinely pass. Runtime classification is not a substitute for phase acceptance.

Evidence and work already completed

  • Product fixes: https://github.com/EzBidPro/navblue-extension/pull/24 (merged).
  • Roadmap, regression tests, and truthful incomplete evidence: https://github.com/EzBidPro/EZBidPro/pull/26 (merged).
  • Documentation-review operational recording repair: https://github.com/EzBidPro/EZBidPro/pull/27 (merged).
  • Latest sealed verification: run safedebug-ec5-repair-20261017205131Z; artifact SHA-256 7b9927412ffeeb4501ed0341486eee98635e3d8b6132b43b6169985b9110310d; log a52c5985d91f5a491b806a0af4e15ce09cba8c4a2fb7b28e4098e0ea8596ce74.
  • Verification passed: focused extension 120/120, selected umbrella 15/15, full extension 533/533, typecheck, production/development builds. A separate full umbrella run passed 46/46.
  • EC5 is already operationally recorded: original completed T3 review identities, verdicts, and reviewed hashes were embedded through public verification_evidence.run_verification, with original artifacts preserved. This issue does not require a new president.ruling.v1 authorization or another model review merely to record an existing one.

Related issues / scope boundaries

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions