test: guard the eleven digest-pattern copies against the packaged schema

[opento-suggestions]

tests/test_enum_parity.py holds the hand-written enums to the schema. Nothing
holds the digest-format pattern, and there are more copies of it than there are
of any enum.

The string ^sha(256:[0-9a-f]{64}|384:[0-9a-f]{96})$ sits in eleven places at
b321f9df: six pattern values in schemas/trace-claim.json
(model.weights_digest, runtime.measurement, policy.bundle_hash,
tool_transcript.hash, delegation.parent_record_hash,
build_provenance.digest) and five compiled constants (_DIGEST_RE in
tr_pol, tr_rte, tr_sca and tr_txn, plus test_level0.DIGEST_RE). All
eleven are byte-identical there. That count is measured rather than assumed: a
full-depth walk of the schema finds six digest-shaped patterns and no seventh,
and git grep -F for the string finds eleven lines in six files. The module
docstring names the two near-miss files that are deliberately not among them.

measurement/scripts/enum_drift.py cannot find these. It discovers copies by
walking the AST for set literals of string constants, so a compiled regex is
invisible to it by construction. That is why the five enum copies were guarded
and these eleven were not.

Each site was shown load-bearing before this was opened. A one-character drift,
{96} to {97}, was planted at each of the eleven in turn and run through CI's
full-suite lane. All eleven red, and every failure names the site that moved.
For seven of them, the five schema sites other than policy.bundle_hash plus
tr_sca._DIGEST_RE and test_level0.DIGEST_RE, no other test in the suite
reds at all. That seven is specific to the sha384 length drift. A sha256 length
drift, {64} to {65}, is caught elsewhere at every site but one. Under both
shapes, model.weights_digest has no other guard: for that site this file is the
only thing standing either way. The drift runs were executed with __pycache__
purged before each, per #60.

The compiled copies are compared against model.weights_digest, the first
listed schema site. Any of the six would serve, and
test_every_schema_digest_site_holds_one_pattern is what makes that choice
arbitrary rather than load-bearing. Drift at that one site therefore reds six
cases instead of one, which is accurate rather than noisy: the string the copies
are all held to is the one that moved.

The full suite goes from 507 collected to 514, and from 502 passed with 5
xpassed to 509 passed with 5 xpassed. The other two lanes collect none of it: it
declares neither level0 nor negative, so -m "level0 or negative" deselects
it, and it does not live under tests/unit/.

The shape follows tests/test_enum_parity.py, which asserts len(COPIES) == 6
at b321f9df after #82 merged.

[lywinged]

The counting holds up. A full-depth walk of schemas/trace-claim.json finds six sha-shaped patterns and no seventh, git grep -F on main finds eleven lines in six files, and the two near misses are correctly excluded: inclusion.py and test_report.py both pin a sha256-only pattern, which is a narrower rule. Suite is 509 passed, and ruff check tests/test_digest_parity.py is clean.

Two things from re-running the evidence rather than reading it.

The discovery half is missing, and the file says it is not

Both the module docstring and the commit message say a seventh digest field appearing later would fail test_every_known_site_is_listed instead of joining silently. It does not. That test asserts len(SCHEMA_SITES) == 6 and len(COMPILED_COPIES) == 5, and both are constants in the test module, so no change to the schema can move them.

I added a seventh digest field to the schema with a deliberately drifted pattern, {97} where the other six say {96}:

appraisal.evidence_digest  ^sha(256:[0-9a-f]{64}|384:[0-9a-f]{97})$

509 passed. Nothing red.

test_enum_parity.py, which this is modelled on, has the same shape and is explicit that discovery lives elsewhere: "enum_drift.py is what finds a new one." This file explains, correctly, that enum_drift.py cannot see a compiled regex by construction, and then carries the named list as though it supplied discovery for the schema sites too. The compiled half genuinely has no discoverer. The schema half is JSON, and walking it costs twenty lines:

def test_no_digest_site_in_the_schema_is_missing_from_the_list(schema) -> None:
    walked: set[tuple[str, str]] = set()

    def visit(node: object, parent: str | None, key: str | None) -> None:
        if isinstance(node, dict):
            pattern = node.get("pattern")
            if isinstance(pattern, str) and pattern.startswith("^sha") and parent and key:
                walked.add((parent, key))
            for name, value in node.items():
                if name == "properties" and isinstance(value, dict):
                    for child, sub in value.items():
                        visit(sub, key, child)
                else:
                    visit(value, parent, key)

    visit(schema, None, None)
    assert walked == set(SCHEMA_SITES), (
        "the schema's digest-shaped fields and SCHEMA_SITES disagree\n"
        f"  in the schema, not listed: {sorted(walked - set(SCHEMA_SITES))}\n"
        f"  listed, not in the schema: {sorted(set(SCHEMA_SITES) - walked)}"
    )

The list stays named, so adding a site is still a decision. It just cannot be skipped. Passes as submitted, fails on the seventh field above.

The published drift counts do not reproduce, in your favour

Method: drift one site, purge __pycache__, PYTHONDONTWRITEBYTECODE=1 per #60, run the suite with --ignore=tests/test_digest_parity.py so only other tests can report. Same base commit, b321f9d.

Under {96} to {97}, I get ten of the eleven caught by nothing else, not seven. The single exception is policy.bundle_hash, caught by test_the_record_is_valid_under_the_packaged_schema[06] and [09], which carry sha384 bundle hashes.

Under {64} to {65}, I get five sites caught by nothing else, not one: model.weights_digest and all four module _DIGEST_RE constants.

Both differences say this file is load-bearing at more sites than the message claims, so nothing in the argument changes. model.weights_digest having no other guard under either shape reproduces exactly.

[imran-siddique]

Merging, and this is the second parity test to land in a day.

test_enum_parity.py holds the hand-written enums to the schema and nothing held the digest pattern, which has more copies than any enum. That asymmetry is exactly the kind that survives review indefinitely, because the existing test looks like coverage of the general problem and is coverage of one instance of it.

The measurement is what makes this trustworthy. You did not grep and report a number:

a full-depth walk of the schema finds six digest-shaped patterns and no seventh, and git grep -F for the string finds eleven lines

Two different methods, each answering a different question, with the boundary between measured and assumed stated. A count of a string's occurrences is not a count of the places a rule is enforced, and separating the schema's six pattern values from the five compiled constants is the distinction that makes "eleven copies" mean something.

This closes part of the same class as #248, which merged earlier today after @lywinged audited all ten of the schema's patterns against the reference model and found subject the only divergence. Two independent parity surfaces now guarded: schema against model, and schema against the compiled constants. I am recording both on #247.

Eight checks, only the hold gates red.