test: guard the hand-written enum copies against the packaged schema - #62
Merged
imran-siddique merged 1 commit intoAug 14, 2026
Conversation
Closes agentrust-io#53. agentrust-io#57 closed the first half of that issue: the ten unguarded conformance failure paths are now guarded, 33 of 33 sites verified. The second half was still open - tests/test_level0.py:7 restates the runtime.platform enum and holds nine of the schema's ten values, missing software-only. That is the value docs/trust-levels.md requires for Level 0. A Level 0 record as documented would fail test_runtime_platform_registered; nothing noticed because valid_level0.json uses intel-tdx. The value is added, and so is the general guard. Five set literals in this repository restate a schema enum: tr_rte._VALID_PLATFORMS runtime.platform tr_pol._VALID_ENFORCEMENT policy.enforcement_mode test_level0.VALID_PLATFORMS runtime.platform test_level0.VALID_ENFORCEMENT policy.enforcement_mode test_level0.VALID_APPRAISAL appraisal.status tests/test_enum_parity.py asserts each against schemas/trace-claim.json on every run, by equality rather than containment: a superset means the copy accepts a value the schema does not define, which is the same drift pointing the other way. Load-bearing, checked rather than assumed - removing software-only again fails exactly that parametrized case. measurement/scripts/enum_drift.py already reported this and exits 1. It walks the AST, which is what finds a copy nobody has listed yet; this file is the other half, so a known copy fails in the suite rather than waiting for someone to run the script. 167 pass. ruff reports the same 68 pre-existing errors either side of this change and none in the new file. Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
github-actions
Bot
added
the
needs-review:UNKNOWN
imran-siddique
approved these changes
Aug 14, 2026
imran-siddique
left a comment
imran-siddique
left a comment
Member
There was a problem hiding this comment.
The parity test derives enum values from the packaged schema and checks each known hand-written copy, while the production enforcement set is now explicitly named and the level-0 platform copy gains the schema's software-only value.
imran-siddique
pushed a commit
that referenced
this pull request
Aug 20, 2026
…ld whose absence hid it (#68) * fix(schema): resync the packaged copy from the normative schema The packaged schemas/trace-claim.json was a v0.1 body under a v0.2 label: 13 top-level properties, no delegation block, no provenance_depth. Records signed against the normative schema were rejected by the packaged one. Since this branch was first written the gap widened - the normative source gained the delegation link block and the provenance depth field (trace-spec #173) - so the resync now carries those too. Packaged copy is byte-identical to schema/trace-claim.json at trace-spec 1c5a9b9. The hand-written enforcement enums gain 'declared' to match, which keeps the #62 parity guard green on the moved schema rather than green on the stale one. 183 passed, 5 xpassed (the TEE-hardware smoke set, strict=False by design, unrelated). Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com> * test(schema): a vector that uses the field whose absence hid the drift The resync in the previous commit fixes the packaged schema. This adds what would have caught it, because the reason nothing caught it is still here otherwise. `schemas/trace-claim.json` sits at `$id` trace-v0.1.json and pins `eat_profile` to the v0.2 sentinel, so it requires a record to declare v0.2 and then rejects two properties v0.2 defines: `signature` and `delegation`. additionalProperties is false, deliberately -- test_unknown_fields_rejected asserts it. Measured against trace-spec's corpus before writing anything: a signed record fails with "'signature' was unexpected" and a signed delegated hop with "'delegation', 'signature' were unexpected". Where that does and does not reach, stated precisely because the first draft of this message overstated it. The file is not shipped -- the wheel packages src/trace_tests only -- and no code under src/ reads it; grep for jsonschema there returns nothing, and TR-ENV checks fields by hand. Its one reader is tests/conftest.py, which feeds tests/test_schema.py, the suite's own Level 0 "structural and schema validity" tests. README.md also links the directory as "Test schemas", so a reader browsing the suite for a reference copy gets the stale one. So the cost is to what this suite can assert, not to any record anyone has scored. A conformance suite whose schema tests run a version behind cannot accept a signed record in them -- which is self-reinforcing, and is why no vector here has a signature. Nothing failed because not one of the six vectors carried a `signature` at all. The schema forbade a field the corpus never used. That is the same shape as the canonicalization defect in tr_sig: the corpus avoided the case the defect lived in. Two in one repository is a pattern, and the answer to both is a vector that uses the field. The two vectors are copied from trace-spec's examples/delegation-link/, generated from a published seed and reproducible by anyone. Nothing compares them to the originals -- a guard that needs another repository checked out is a guard that gets skipped. They are held instead to the property that makes them worth having: they carry the fields whose absence hid the defect, so a vector edited into something unsigned fails loudly rather than quietly. Verified by reverting the schema: the two validation tests and the `$id` check fail, and the self-validation test passes, which is the shape a regression test should have. 180 passed. Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com> --------- Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #53.
#57 closed the first half of that issue — the ten unguarded conformance failure paths are guarded now, 33 of 33 sites verified. This is the second half, which was still open.
The drift
tests/test_level0.py:7restates theruntime.platformenum and holds nine of the schema's ten values.software-onlyis missing.That is the value
docs/trust-levels.mdrequires for Level 0:So a Level 0 record as documented fails
test_runtime_platform_registered. Nothing noticed becausevalid_level0.jsonusesintel-tdx.The guard
Adding the value fixes today. Five set literals in this repository restate a schema enum, and any of them can drift the same way:
tests/test_enum_parity.pyasserts each againstschemas/trace-claim.jsonon every run, by equality rather than containment — a superset means the copy accepts a value the schema does not define, which is the same drift pointing the other way.Load-bearing rather than decorative: removing
software-onlyagain fails exactly that parametrized case, which I checked rather than assumed.measurement/scripts/enum_drift.pyalready reports this and exits 1. It walks the AST, which is the right tool for finding a copy nobody has listed yet; this file is the other half, so a known copy fails in the suite instead of waiting for someone to run the script.Verification
Fresh clone, directory that has never held the repository: 167 pass, 5 xpass.
ruffreports the same 68 pre-existing errors either side of this change, and none in the new file.Deliberately not in this PR
enum_drift.pyreports two further drifts, both against the normative schema rather than the packaged copy — it fetchestrace-claim.jsonfromtrace-specmain, and the copy inschemas/is behind it.policy.enforcement_modegaineddeclaredupstream and neithertr_pol.pynor the packaged copy has it.That is a different problem with a wider blast radius than an enum, and I would rather raise it separately than widen this one. Filing it next.