Repository navigation
fix(artifacts): stop demanding byte-equality for enriched bounded report fields - #919
Merged
Merged
Conversation
…ort fields Bounded classification mode instructs the final-report agent to enrich the dedupe lifecycle record in the report object and to choose the row's triage classification, and the gate validates both with dedicated enrichment-aware checks — the lifecycle against the authenticated ledger with the exact owned-field set, and the row classification against the enriched lifecycle record. Its generic preservation loop then contradicted that contract by deep-equaling every dedupe-finding field against the report row, so any dedupe row that itself carried lifecycle or a preliminary triage_classification made every compliant report fail: all three targets of benchmark run 32898410734 died at final-report this way, one of them failing only on lifecycle while preserving every prose field byte-for-byte. Skip exactly those two fields in the generic loop. Everything else, including description, proof_of_concept, summary, and evidence fields, keeps strict byte-preservation, and tampering any non-owned lifecycle field still fails through the dedicated ledger comparison. Closes #918 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 26, 2026
aviggiano
added a commit
that referenced
this pull request
Aug 26, 2026
…port rows (#932) Bounded classification mode instructs the final report to author every promoted row's severity assessment — impact, likelihood, their rationales, the matrix severity, and its rationale — so a dedupe row that already carries any of those fields can never byte-equal its promoted report row: run 32914329945 failed final-report on exactly impact_rationale and likelihood_rationale after #919 removed the lifecycle contradiction, and the retry churn ran the other two rows out of the watch window. Extend the promoted-row skip set from id/title to the full prompt-authored field set. The authored severity itself stays independently validated (canonical_severity equality, High→Medium→Low stable ordering, canonical M-01-style ids and titles), non-promoted rows keep full byte-preservation for these fields, and every evidence field stays strictly preserved for all rows. Closes #931 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #918. Fifth and — per the run evidence — likely final blocker in the chain (#886 → #892 → #901 → #909 → this) between the re-enabled eval lane and a publishable smoke observation.
Problem
Smoke run 32898410734 (candidate a695b0a) ran the full pipeline for the first time — 2.5 h of real agent work, all workflows launched, all review stages green — and all three targets failed at the last node,
final-report, onreport-severity-classification-preservation.The smoke lane omits severity classification, so the gate takes the bounded-classification branch. The final-report prompt for that mode says: enrich the dedupe lifecycle record in the report object and choose the row's
triage_classification. The gate's dedicated checks honor exactly that (lifecycleRecordPreservationIssueswithBOUNDED_REPORT_LIFECYCLE_OWNED_FIELDS; row classification must equal the enriched record's) — but the generic loop then deep-equals every dedupe-finding field against the report row. Whenever a dedupe row itself carrieslifecycleor a preliminarytriage_classification(schema-legal; the venus and stableswap artifacts did), a prompt-compliant report cannot pass. The venus row set failed only onlifecycle— all four issues — while preserving every prose field byte-for-byte: a maximally compliant report, rejected deterministically.Gate landed in #558 and tightened in #621, both after the last complete benchmark run — never exercised end-to-end until now.
Fix
Skip exactly
lifecycleandtriage_classificationin the bounded branch's generic preservation loop — the two fields with dedicated enrichment-aware validation directly above. Strict byte-preservation is unchanged for everything else (description,proof_of_concept,summary, evidence fields — venus proves models comply). Tampering a non-owned lifecycle field (e.g.source_artifacts) still fails via the ledger comparison; the existing test pinning that stays green.Validation
@ultrafuzz/artifacts: 322/322 pass, including the extended gate test: dedupe row embeddinglifecycle+ preliminary classification passes with a compliant enriched report row; tamperingsummarystill fails withdid not preserve dedupe field "summary".🤖 Generated with Claude Code
Greptile Summary
This PR fixes bounded final-report validation by exempting report-owned
lifecycleandtriage_classificationfields from generic byte-equality checks while retaining dedicated enrichment-aware validation.Confidence Score: 5/5
The PR appears safe to merge with no actionable correctness or security issues identified.
The exemption is limited to fields owned by bounded final-report enrichment, while authenticated lifecycle provenance, row-to-lifecycle classification consistency, derived disposition, and all other dedupe fields remain validated.
Important Files Changed
Reviews (1): Last reviewed commit: "fix(artifacts): stop demanding byte-equa..." | Re-trigger Greptile