Follow-up work for the provenance plugin, graduated from the copied-external-content contract
slice when it was pruned in #3456 (topic-docs "Contract-slice lifecycle"). Phases 1 to 5 shipped
in that PR; this issue carries the three phases that remain.
The durable design artifacts graduated to docs/specs/ in the same PR, history preserved:
docs/specs/provenance-type-inventory.md — script contracts, finding record, tier enum, config
schema, golden-set case shape, and the three draft crosswalk rows Phase 7 landsdocs/specs/provenance-capability-matrix.md — the 17 capabilities and their script/model splitdocs/specs/provenance-design-threads.md — T1 to T15, with T15 naming the open probes belowdocs/specs/provenance-plugin-topology.md — layout, load order, and the boundary statementdocs/specs/provenance-convention-engagement.md — the Phase 8 upstream-drift engagement, drafted
in full
Goal, carried verbatim from the pruned Brief
Reduce drift risk and maintenance burden from locally copied external content by giving the fleet
a portable audit-and-fix capability that finds tracked prose restating externally-owned facts,
confirms provenance against the authoritative source, and converts copies into pointers (or, for
load-bearing restatements, conforming four-part stamped records), with measured precision the
consuming repo can trust.
Measured baseline as of 2026-08-28
Over this repository, 1,347 tracked markdown files after carve-outs (20 vendored paths declined):
525 stamp candidates, 482 parsed, 43 declined (19 month-name forms, 24 bare years), 0 expired at
the 180-day default. The oldest parsed stamp is 2026-04-08; at a 60-day window the same corpus
yields 9 findings. The declined count is the honest report the design asks for, not a defect to
tune away.
Phase 6: golden set, measurement, and the improvement loop
Author 5 to 10 synthetic cases per the case shape in docs/specs/provenance-type-inventory.md
(positives as shape-preserving rewrites of real history cases; hard negatives including
paraphrase-styled-never-copied distractors and quoted-and-cited excerpts; plus 2 to 3 adversarial
synonym-rotation cases probing the separation rule, per design thread T15). Run the audit over the
fixture corpus, hand-score case-level precision and recall, tune the separation constants and
budgets from measured telemetry, and record the measured values.
Build the improvement loop: an adjudications section instructing that human-rejected findings and
later-discovered misses are converted (synthetic rewrite, never verbatim externally-owned
text) into new golden cases, so the set grows toward 20 to 50. This phase also measures the
accuracy dials: diverse judge lenses versus identical prompts, and multi-pass nomination union,
scored against the same golden set so each dial's recall gain is a recorded number rather than a
belief.
Two consequences stated plainly, so neither reads as a failure later:
- Every class ships report-only at v1, by the gate's own arithmetic: with 5 to 10 cases across
four classes, no class reaches min_n_per_class 10. Fix mode goes live per class only when the
growth loop carries that class past minimum n at or above the precision bar, verbatim first.
Growing the two fix-eligible classes to n >= 10 is a named exit condition of the first growth
round. At n near 10 the 0.95 bar means one error demotes, so the bar behaves as a ratchet rather
than a statistic, and that is accepted.
- Phase 6 runs are sidecar-only. The emitter is not invoked until Phase 7's registry rows
exist, so no relay file ever carries a rule id with no crosswalk row.
Harness seam: the fixture-tree exclusion is a CONFIG-LAYER entry (this repo's
.claude/provenance.json excluded_paths), never unconditional in list-corpus.sh, exactly the
ai-slop resolution of #3041 — the eval harness's isolation lifts the exclusion so fixtures are
measurable, while every normal run declines them and says so. This repo does not yet carry a
.claude/provenance.json; adding one with that glob is Phase 6's first step. A case directory
with source.md present short-circuits the fetch stage, so golden runs exercise nomination,
resolution, fingerprint and judgment without live fetches; the bypass is scoped to corpus
enumeration and fetch, never to judgment.
Sanity Check: score-golden.sh --golden <dir> --actual <run> exits 0 and emits per-class P/R;
with this repo's config in place, list-corpus.sh plugins/provenance output does NOT contain
fixtures/golden; bash scripts/check-orphaned-fixtures.sh --check exits 0; git diff --name-only for the phase contains no relay findings file.
Phase 7: crosswalk registration and quality gate
Land the three draft crosswalk rows from docs/specs/provenance-type-inventory.md in the
detector-findings registry, then run the full-repo quality gates over every new surface.
Registering the rule-trigger-less-stamp row here does NOT fire the upstream-drift convention
engagement and does not touch that convention's "named but not built" table row: the engagement
fires at sweep completion only, and the row update is part of that engagement's changelog entry.
The interim state (check built and registered, convention table not yet updated) is accepted and
recorded here so it reads as scheduled rather than drifted.
Sanity Check: bash scripts/check-detector-findings-crosswalk.sh --check exits 0;
/ai-slop:audit over plugins/provenance reports clean; the affected-test lane passes (note that
scripts/affected-tests.sh --run exits 3 for this plugin, not 0 — exit 1 is a failing suite,
exit 3 means every shell suite passed and the Node suite needs its own lane; run
node plugins/provenance/skills/audit/scripts/fingerprint.test.mjs separately).
Phase 8: sweep and convention engagement
The repo-wide sweep under the execution contract, the combined upstream-drift convention
engagement (executing docs/specs/provenance-convention-engagement.md), and the narrow CI
regression gate for the cleaned state. Promoted to its own sub-topic: own commit boundaries, more
than five work items, its own measurement needs.
The sub-topic's Brief must carry, beyond the execution contract:
- The T15 live no-breadcrumb (
not-found) probe as a first-run validation item. That disposition
has never been exercised live — every spike source was supplied — so the built plugin's first
no-breadcrumb case validates the searched-surfaces listing.
- Sweep resume semantics: the fetch ceiling and cache are scoped per SWEEP and recorded in the
closure ledger, so a resumed sweep neither resets its spend nor silently reuses a stale cache.
The ledger is checkout-local, stated.
Execution contract: one tracked file at a time; apply the verdict, verify (semantic diff plus
pointer liveness), close. A file is closed when every finding in it carries a disposition or an
explicit neutral outcome. Sweep completion, not spike results, fires the upstream-drift
reopening, recorded as one convention engagement with one changelog entry, major bump only if the
enforceability verdict changes.
Acceptance criteria still open
- The golden set is hand-scored for case-level precision and recall; the fix-mode precision gate
binds only at a stated minimum n per finding class, below which the class ships report-only.
fix is reachable only by explicit argument, applies the three dispositions, passes a
fresh-context semantic-diff guard, and verifies every pointer target live before writing.
(Implemented and documented in phases 4 and 5; unmeasured until Phase 6.)- Sweep completion fires the upstream-drift reopening as one convention engagement.
Related
Follow-up work for the
provenanceplugin, graduated from thecopied-external-contentcontractslice when it was pruned in #3456 (topic-docs "Contract-slice lifecycle"). Phases 1 to 5 shipped
in that PR; this issue carries the three phases that remain.
The durable design artifacts graduated to
docs/specs/in the same PR, history preserved:docs/specs/provenance-type-inventory.md— script contracts, finding record, tier enum, configschema, golden-set case shape, and the three draft crosswalk rows Phase 7 lands
docs/specs/provenance-capability-matrix.md— the 17 capabilities and their script/model splitdocs/specs/provenance-design-threads.md— T1 to T15, with T15 naming the open probes belowdocs/specs/provenance-plugin-topology.md— layout, load order, and the boundary statementdocs/specs/provenance-convention-engagement.md— the Phase 8 upstream-drift engagement, draftedin full
Goal, carried verbatim from the pruned Brief
Reduce drift risk and maintenance burden from locally copied external content by giving the fleet
a portable audit-and-fix capability that finds tracked prose restating externally-owned facts,
confirms provenance against the authoritative source, and converts copies into pointers (or, for
load-bearing restatements, conforming four-part stamped records), with measured precision the
consuming repo can trust.
Measured baseline as of 2026-08-28
Over this repository, 1,347 tracked markdown files after carve-outs (20 vendored paths declined):
525 stamp candidates, 482 parsed, 43 declined (19 month-name forms, 24 bare years), 0 expired at
the 180-day default. The oldest parsed stamp is 2026-04-08; at a 60-day window the same corpus
yields 9 findings. The declined count is the honest report the design asks for, not a defect to
tune away.
Phase 6: golden set, measurement, and the improvement loop
Author 5 to 10 synthetic cases per the case shape in
docs/specs/provenance-type-inventory.md(positives as shape-preserving rewrites of real history cases; hard negatives including
paraphrase-styled-never-copied distractors and quoted-and-cited excerpts; plus 2 to 3 adversarial
synonym-rotation cases probing the separation rule, per design thread T15). Run the audit over the
fixture corpus, hand-score case-level precision and recall, tune the separation constants and
budgets from measured telemetry, and record the measured values.
Build the improvement loop: an adjudications section instructing that human-rejected findings and
later-discovered misses are converted (synthetic rewrite, never verbatim externally-owned
text) into new golden cases, so the set grows toward 20 to 50. This phase also measures the
accuracy dials: diverse judge lenses versus identical prompts, and multi-pass nomination union,
scored against the same golden set so each dial's recall gain is a recorded number rather than a
belief.
Two consequences stated plainly, so neither reads as a failure later:
four classes, no class reaches
min_n_per_class10. Fix mode goes live per class only when thegrowth loop carries that class past minimum n at or above the precision bar,
verbatimfirst.Growing the two fix-eligible classes to n >= 10 is a named exit condition of the first growth
round. At n near 10 the 0.95 bar means one error demotes, so the bar behaves as a ratchet rather
than a statistic, and that is accepted.
exist, so no relay file ever carries a rule id with no crosswalk row.
Harness seam: the fixture-tree exclusion is a CONFIG-LAYER entry (this repo's
.claude/provenance.jsonexcluded_paths), never unconditional inlist-corpus.sh, exactly theai-slop resolution of #3041 — the eval harness's isolation lifts the exclusion so fixtures are
measurable, while every normal run declines them and says so. This repo does not yet carry a
.claude/provenance.json; adding one with that glob is Phase 6's first step. A case directorywith
source.mdpresent short-circuits the fetch stage, so golden runs exercise nomination,resolution, fingerprint and judgment without live fetches; the bypass is scoped to corpus
enumeration and fetch, never to judgment.
.claude/provenance.jsonCREATE (fixture-tree exclusion)plugins/provenance/skills/audit/evals/fixtures/golden/c01-.../CREATE (5 to 10 case dirs:case.md,expected.json,source.mdwhere needed)plugins/provenance/skills/audit/evals/evals.jsonMODIFY (register the golden case dirs asfiles[];check-orphaned-fixtures.sh --checkfails unregistered fixture files, and a newcase is not landed until
evals.jsonnames it)plugins/provenance/skills/audit/reference/dispositions.mdMODIFY (adjudication-to-fixtureloop)
Sanity Check:
score-golden.sh --golden <dir> --actual <run>exits 0 and emits per-class P/R;with this repo's config in place,
list-corpus.sh plugins/provenanceoutput does NOT containfixtures/golden;bash scripts/check-orphaned-fixtures.sh --checkexits 0;git diff --name-onlyfor the phase contains no relay findings file.Phase 7: crosswalk registration and quality gate
Land the three draft crosswalk rows from
docs/specs/provenance-type-inventory.mdin thedetector-findings registry, then run the full-repo quality gates over every new surface.
docs/conventions/detector-findings/README.mdMODIFY — three rows (rule-verbatim-copy,rule-stamp-expired,rule-trigger-less-stamp), RESHAPED into the registry's exact columnset with pipes escaped and the
<name>placeholders substituted toprovenance/audit/rule-*,appended to the existing table, never pasted as a second table: the crosswalk check
locates the table by its exact header and fails duplicates
docs/conventions/detector-findings/CHANGELOG.mdMODIFY (entry for the rows)Registering the
rule-trigger-less-stamprow here does NOT fire the upstream-drift conventionengagement and does not touch that convention's "named but not built" table row: the engagement
fires at sweep completion only, and the row update is part of that engagement's changelog entry.
The interim state (check built and registered, convention table not yet updated) is accepted and
recorded here so it reads as scheduled rather than drifted.
Sanity Check:
bash scripts/check-detector-findings-crosswalk.sh --checkexits 0;/ai-slop:auditoverplugins/provenancereports clean; the affected-test lane passes (note thatscripts/affected-tests.sh --runexits 3 for this plugin, not 0 — exit 1 is a failing suite,exit 3 means every shell suite passed and the Node suite needs its own lane; run
node plugins/provenance/skills/audit/scripts/fingerprint.test.mjsseparately).Phase 8: sweep and convention engagement
The repo-wide sweep under the execution contract, the combined upstream-drift convention
engagement (executing
docs/specs/provenance-convention-engagement.md), and the narrow CIregression gate for the cleaned state. Promoted to its own sub-topic: own commit boundaries, more
than five work items, its own measurement needs.
The sub-topic's Brief must carry, beyond the execution contract:
not-found) probe as a first-run validation item. That dispositionhas never been exercised live — every spike source was supplied — so the built plugin's first
no-breadcrumb case validates the searched-surfaces listing.
closure ledger, so a resumed sweep neither resets its spend nor silently reuses a stale cache.
The ledger is checkout-local, stated.
Execution contract: one tracked file at a time; apply the verdict, verify (semantic diff plus
pointer liveness), close. A file is closed when every finding in it carries a disposition or an
explicit neutral outcome. Sweep completion, not spike results, fires the upstream-drift
reopening, recorded as one convention engagement with one changelog entry, major bump only if the
enforceability verdict changes.
Acceptance criteria still open
binds only at a stated minimum n per finding class, below which the class ships report-only.
fixis reachable only by explicit argument, applies the three dispositions, passes afresh-context semantic-diff guard, and verifies every pointer target live before writing.
(Implemented and documented in phases 4 and 5; unmeasured until Phase 6.)
Related