test: every fixture generator must reproduce its committed fixtures

[lywinged]

Both generators in this repository currently reproduce their committed fixtures. This adds the check that establishes it, and keeps establishing it.

A generated fixture and the script that generates it can disagree for as long as nobody runs the script, and every test stays green while they do: the tests read the committed file, so they measure the fixture and never the generator. Whichever of the two is stale, running the generator once rewrites the other, and the rewrite is silent.

I found this on a set in my own fork rather than here. 03-superseded-version-refused.json recorded superseded_profile_refused, matching the dedicated branch verify_record takes for the v0.1 identifier, while its generator still emitted the generic profile_not_accepted from before that branch existed. Both codes are registered, so nothing failed. Regenerating would have replaced a specific rule with a general one and left the suite green.

Generators are discovered rather than listed. A catalogue of things to keep in sync is correct until the first person forgets it, and that failure is silent too.

Reproduction

git fetch origin pull/171/head:pr171 && git checkout pr171
pytest -q tests/test_generators_reproduce_fixtures.py     # 3 passed

Load-bearing, in two independent directions:

# a changed emitted value
python - <<'PY'
import pathlib
p = pathlib.Path("examples/action-receipts/conformance/gen_rule_coverage_vectors.py")
p.write_text(p.read_text().replace("indent=2", "indent=3", 1))
PY
pytest -q tests/test_generators_reproduce_fixtures.py
# FAILED ...[conformance]
git checkout -- examples

# a generator that produces nothing at all
python - <<'PY'
import pathlib
p = pathlib.Path("examples/canonicalization-boundary/gen_boundary_vectors.py")
p.write_text(p.read_text().replace("def main()", "def _disabled_main()", 1) + "\n\ndef main():\n    pass\n")
PY
pytest -q tests/test_generators_reproduce_fixtures.py
# FAILED ...[canonicalization-boundary]  missing: [...]
git checkout -- examples

The discovery walk carries its own guard: test_generators_were_found fails if the walk returns nothing, since a walk that finds no generators would pass every parametrised case beneath it.

Full suite on 3.11: 379 passed, 1 skipped. Scope is tests/ only. No src, schema, example, workflow or dependency changes.

[github-actions]

❔ Contributor Check: UNKNOWN

Check	Result
Profile	UNKNOWN
Credential	LOW
Overall	UNKNOWN

Automated check by AgenTrust Contributor Check.

[imran-siddique]

The byte comparison is useful, but it only compares names from the committed directory. If a generator begins emitting an additional JSON fixture that is not committed, cmpfiles never sees it and this guard passes—the same silent generator/artifact disagreement this PR is meant to prevent. Please compare the complete generated and committed *.json filename sets first, then compare bytes for the equal set. Add a test or demonstrated mutation showing an extra generated fixture turns the guard red.

[lywinged]

Both changes are in 54d7e09.
The name sets are compared before any bytes are, in both directions, and test_an_extra_generated_fixture_is_caught builds a repository whose generator writes a fixture that is not committed, then asserts the failure.
Fixing that surfaced the mirror case, which was mine: the work directory was seeded with copytree, so a fixture the generator stopped writing was left behind by the copy and compared against itself. It agreed, every time. The generator’s directory is now emptied of fixtures before it runs, so every byte compared is a byte that run wrote.
On this repository, the old guard against the new one:

mutation before after
generator writes an uncommitted fixture passes fails
generator drops a committed fixture passes fails
committed fixture edited by hand fails fails

The clean room also makes a standing fact visible: gen_rule_coverage_vectors.py produces 10-30 of the thirty fixtures beside it. Its own first paragraph gives the reason, that 01-09 pin a key whose private half is not published, so nothing can reissue them.
Those nine are named in NOT_GENERATED, and I would ask you to read that list as a ledger rather than a carve-out. It cannot grow quietly, because a committed fixture no generator produces fails the check and the only way in is an edit there, with a reason, in front of a reviewer. An exemption also withdraws reproduction and not verification: test_action_receipt_fixtures globs the same directory and puts all thirty through the verifier, signature included.
What it is really recording is a cost that has not arrived yet. While this corpus only grows, nine unreproducible vectors cost nothing. The first release that revises it makes every one of them a hand edit, and a hand edit is the failure this module exists to catch. That release is the one #66 and #173 are building toward. I have not touched them here, since reissuing changes pinned key material in vectors other implementations may pin and the timing is yours; I have filed it as #178 so it can be triaged on its own rather than living in a review thread, and I am happy to do the mechanical work whenever you want it.
390 passed, 1 skipped. ruff clean.

[imran-siddique]

The prior filename-set bypass is resolved. The guard now compares produced and committed names in both directions, clears copied fixtures before generation, detects stale/unknown/unnecessary exemptions, and refuses ambiguous multi-generator directories. Its self-tests cover extra output, dropped output, byte drift, generator failure, empty corpora, and exemption lifecycle. All 14 focused assertions passed locally; pytest then encountered a Windows temp-directory cleanup PermissionError after completion, so that local exit is not claimed clean. Hosted Python 3.11/3.12 and CodeQL runs are green. Approved.