test(verifier-compatibility): close two margins, and record why the other two cannot - #7
Merged
Merged
Conversation
…ther two cannot Continues agentrust-io#116, which asked for obligations 2 and 3 back "with the separation figure measured and pinned". Three things here, in order of how much they matter. ## The module's own headline number had drifted `test_verifier_compatibility_separation.py` exists so the set's separating power cannot change unnoticed. Its opening paragraph said the set separates "3 of its 8 vectors" while the figure pinned in code twenty lines lower said 4 of 9. A reader arrives at the docstring first and left with the superseded number. The figure now lives in one place, next to `SEPARATING`, and the paragraph says so rather than restating it. ## Two of the four recorded margin shortfalls are closed `KNOWN_THIN` recorded four refusal rules carried by a single vector each, which agentrust-io#124 established as insufficient. Two now have a second vector: `10-superseded-first-in-set-innocent-record-refused` pins `superseded_profile_in_accepted_set` from the other two directions at once. The v0.1 identifier is first in the accepted set rather than last, so an implementation scanning only the tail passes 08 and fails here, and the record presented is an ordinary v0.2 record with nothing wrong with it. The second half is what makes it separate where 08 cannot: 08 carries a v0.1 record, which the schema refuses on its own whether or not a profile rule ran. `11-empty-profile-string-refused` pins `profile_absent` with a claim that is present and empty. 07 removes the member outright, so an implementation testing `"eat_profile" not in record` passes 07 and reads this one as an unrecognised profile, or as absent and therefore current. It does not separate, and the reason is structural rather than a flaw: the schema pins `eat_profile` with a `const`, so any record reaching this rule is already schema-invalid. Separation measured after: 5 of 11, up from 4 of 9. Vector 10 separates, 11 does not, both as designed. ## The other two were measured and are not closable Recorded in `KNOWN_THIN` with the reason, because "measured and shown to have no second axis" is a different state from "not yet done": `no_accepted_profiles` fires on the verifier's configuration before any record is read, and that configuration has one shape. The only other axis, pairing the empty set with a defect the verifier would catch later, needs `check_freshness`, which every vector in this set asserts is False for a good reason: a fixed `iat` would make the set expire. `superseded_profile_refused` needs a record carrying v0.1 and an accepted set excluding it, and the set can hold only v0.2, since any other member trips `unschemaed_profile_in_accepted_set` first. What is left for both is varying record content, which pins nothing: no plausible implementation branches on record content when deciding that an empty set accepts nothing. A second vector written to close a count rather than to catch a defect reports a margin that does not exist, which makes the record worse. 966 passed, 1 skipped. Ruff clean. The generator reproduces every committed fixture byte for byte, per agentrust-io#171. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues agentrust-io#116. The ask there was obligations 2 and 3 back "with the separation figure measured and pinned the way you did here", and "a set that separates 3 of 8 and says so is worth more than one that separates 8 of 8 and has never been asked."
Obligations 2 and 3 are already implemented:
verify_recordtakesaccepted_profilesand refuses outside it, andVerificationStatementcarriesprofileandaccepted_profiles. What was outstanding is the vector set and the number.The module's own headline number had drifted
test_verifier_compatibility_separation.pyexists so the set's separating power cannot change unnoticed. Its opening paragraph said the set separates 3 of its 8 vectors while the figure pinned in code twenty lines below said 4 of 9. A reader arrives at the docstring first and left with the superseded number.This is the defect this module was written to prevent, happening to the module's own headline. The figure now lives in one place, next to
SEPARATING, and the paragraph says so instead of restating it.Two of four recorded margin shortfalls closed
KNOWN_THINrecorded four refusal rules carried by one vector each, the shape agentrust-io#124 established as insufficient.10-superseded-first-in-set-innocent-record-refusedclosessuperseded_profile_in_accepted_set. It moves two things at once: the v0.1 identifier sits first in the accepted set rather than last, so an implementation scanning only the tail passes 08 and fails here; and the record presented is an ordinary v0.2 record with nothing wrong with it. The second half is what makes it separate where 08 cannot, since 08 carries a v0.1 record that the schema refuses on its own whether or not any profile rule ran.11-empty-profile-string-refusedclosesprofile_absentwith a claim that is present and empty. 07 removes the member outright, so an implementation testing"eat_profile" not in recordpasses 07 and reads this one as an unrecognised profile, or as absent and therefore current. It does not separate, and the reason is structural rather than a flaw: the schema pinseat_profilewith aconst, so any record that reaches this rule is already schema-invalid and a verifier implementing nothing refuses it too.The other two were measured and are not closable
Recorded in
KNOWN_THINwith the reason, because "measured and shown to have no second axis" is a different state from "not yet done":no_accepted_profilesfires on the verifier's own configuration before any record is read, and that configuration has one shape: the set is empty. The one other axis, pairing the empty set with a second defect the verifier would catch later, needscheck_freshness, and every vector in this set asserts that isFalsefor a good reason already written into the fixtures test: a fixediatwould make the set expire.superseded_profile_refusedneeds a record carrying v0.1 and an accepted set excluding it. The set can hold only v0.2, because any other member tripsunschemaed_profile_in_accepted_setfirst, so there is no second configuration to present.What remains for both is varying record content, which pins nothing. No plausible implementation branches on record content when deciding that an empty set accepts nothing. A second vector written to close a count rather than to catch a defect an implementation could plausibly have reports a margin that does not exist, which makes the record worse than leaving the shortfall named.
Checks
966 passed, 1 skipped. Ruff clean acrosssrc,testsandexamples. The generator reproduces every committed fixture byte for byte, per agentrust-io#171.Both new vectors were validated against the real verifier before the records were updated: the fifteen other fixture tests passed on the first run, so each produces the failure code it claims. The separation figure and the margin shortfall were each measured and then recorded, not recorded and then assumed.
Generated by Claude Code