Skip to content

Go verifier and E21: schema copy check, parity on reasons, v2 message, blind pools scored #501

Closed

Description

@LoopGlitch26

From a review of the independent Go verifier (contrib/independent-go-verifier) and the E21 blind pools:

  1. The Go verifier's schema copy has no sync check. schema.go embeds a copy of tracekit/schema/tracekit.event.v2.json. The TypeScript copy is checked (sdk/typescript/test/verify.test.mjs); the Go one isn't, so a schema change leaves Go on the old one.
    • Fix: a Go test that compares the copy with the repo file.
  2. Parity checks verdicts only. KNOWN_GAPS in scripts/differential_golden.py exempts 15 of the 16 check names the Python verifier reports, so "same reasons" is untested.
    • Fix: make Go report the same check names and trim KNOWN_GAPS. Where that isn't possible, say "verdicts only" in the script and the README.
  3. Wrong error message. On a v2 bundle without -trust, the Go CLI says "invalid v1 manifest". It picks the format from the flag, not the bundle.
    • Fix: read the format from the manifest first, and say "a v2 bundle needs -trust" as Python does.
  4. The E21 blind pools aren't part of E21. eval/e21_blind* (feat: add E21 blind evaluation corpus (fixes #344) #357, feat: add E21 blind evaluation pool for amandeavor (fixes #368) #401) is read by nothing in eval/ or the docs, so the blind scores live only in PR comments.
    • Fix: score them in eval/e21_policy_robustness.py as a blind set, record the results in eval/results/, and report them in docs/evaluation.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions