Skip to content

fix(artifacts): stop demanding byte-equality for enriched bounded report fields - #919

Merged
aviggiano merged 1 commit into
mainfrom
fix/issue-918-bounded-report-enrichment
Aug 25, 2026
Merged

aviggiano merged 1 commit into
mainfrom
fix/issue-918-bounded-report-enrichment

Conversation

@aviggiano

@aviggiano aviggiano commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #918. Fifth and — per the run evidence — likely final blocker in the chain (#886 → #892 → #901 → #909 → this) between the re-enabled eval lane and a publishable smoke observation.

Problem

Smoke run 32898410734 (candidate a695b0a) ran the full pipeline for the first time — 2.5 h of real agent work, all workflows launched, all review stages green — and all three targets failed at the last node, final-report, on report-severity-classification-preservation.

The smoke lane omits severity classification, so the gate takes the bounded-classification branch. The final-report prompt for that mode says: enrich the dedupe lifecycle record in the report object and choose the row's triage_classification. The gate's dedicated checks honor exactly that (lifecycleRecordPreservationIssues with BOUNDED_REPORT_LIFECYCLE_OWNED_FIELDS; row classification must equal the enriched record's) — but the generic loop then deep-equals every dedupe-finding field against the report row. Whenever a dedupe row itself carries lifecycle or a preliminary triage_classification (schema-legal; the venus and stableswap artifacts did), a prompt-compliant report cannot pass. The venus row set failed only on lifecycle — all four issues — while preserving every prose field byte-for-byte: a maximally compliant report, rejected deterministically.

Gate landed in #558 and tightened in #621, both after the last complete benchmark run — never exercised end-to-end until now.

Fix

Skip exactly lifecycle and triage_classification in the bounded branch's generic preservation loop — the two fields with dedicated enrichment-aware validation directly above. Strict byte-preservation is unchanged for everything else (description, proof_of_concept, summary, evidence fields — venus proves models comply). Tampering a non-owned lifecycle field (e.g. source_artifacts) still fails via the ledger comparison; the existing test pinning that stays green.

Validation

  • @ultrafuzz/artifacts: 322/322 pass, including the extended gate test: dedupe row embedding lifecycle + preliminary classification passes with a compliant enriched report row; tampering summary still fails with did not preserve dedupe field "summary".
  • Severity-handoff (strict) branch untouched.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes bounded final-report validation by exempting report-owned lifecycle and triage_classification fields from generic byte-equality checks while retaining dedicated enrichment-aware validation.

  • Preserves strict equality checks for all other deduped-finding fields.
  • Adds regression coverage for embedded lifecycle and preliminary classification enrichment.
  • Confirms unrelated prose tampering and lifecycle-ledger tampering remain rejected.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness or security issues identified.

The exemption is limited to fields owned by bounded final-report enrichment, while authenticated lifecycle provenance, row-to-lifecycle classification consistency, derived disposition, and all other dedupe fields remain validated.

Important Files Changed

Filename Overview
packages/artifacts/src/semantic-gates.ts Narrows bounded dedupe preservation checks for two report-owned enriched fields while retaining dedicated lifecycle, classification, disposition, and ledger validation.
packages/artifacts/test/semantic-gates.test.ts Adds focused regression coverage proving compliant enrichment passes and non-owned dedupe prose remains immutable.

Reviews (1): Last reviewed commit: "fix(artifacts): stop demanding byte-equa..." | Re-trigger Greptile

…ort fields

Bounded classification mode instructs the final-report agent to enrich
the dedupe lifecycle record in the report object and to choose the
row's triage classification, and the gate validates both with
dedicated enrichment-aware checks — the lifecycle against the
authenticated ledger with the exact owned-field set, and the row
classification against the enriched lifecycle record. Its generic
preservation loop then contradicted that contract by deep-equaling
every dedupe-finding field against the report row, so any dedupe row
that itself carried lifecycle or a preliminary triage_classification
made every compliant report fail: all three targets of benchmark run
32898410734 died at final-report this way, one of them failing only on
lifecycle while preserving every prose field byte-for-byte.

Skip exactly those two fields in the generic loop. Everything else,
including description, proof_of_concept, summary, and evidence fields,
keeps strict byte-preservation, and tampering any non-owned lifecycle
field still fails through the dedicated ledger comparison.

Closes #918

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aviggiano
aviggiano merged commit 7a6f619 into main Aug 25, 2026
7 checks passed
@aviggiano
aviggiano deleted the fix/issue-918-bounded-report-enrichment branch August 25, 2026 23:50
aviggiano added a commit that referenced this pull request Aug 26, 2026
…port rows (#932)

Bounded classification mode instructs the final report to author every
promoted row's severity assessment — impact, likelihood, their
rationales, the matrix severity, and its rationale — so a dedupe row
that already carries any of those fields can never byte-equal its
promoted report row: run 32914329945 failed final-report on exactly
impact_rationale and likelihood_rationale after #919 removed the
lifecycle contradiction, and the retry churn ran the other two rows
out of the watch window.

Extend the promoted-row skip set from id/title to the full
prompt-authored field set. The authored severity itself stays
independently validated (canonical_severity equality, High→Medium→Low
stable ordering, canonical M-01-style ids and titles), non-promoted
rows keep full byte-preservation for these fields, and every evidence
field stays strictly preserved for all rows.

Closes #931

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bounded report preservation gate contradicts its own enrichment contract, failing every compliant final report

1 participant