Skip to content

benchmark: frozen-repo snapshot builder — leak-proof repo state at commit T #9259

Description

@JSONbored

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Scoring an arbitrary agent against realized history is only meaningful if the agent sees exactly the repository state a maintainer saw at commit T and nothing that happened after it. Any future-information leak silently inflates every score built on top, and it is the single hardest correctness property in this harness — so it gets its own issue, built and tested before anything consumes it.

Requirements

  1. Snapshot reference: a FrozenRepoSnapshot describing a repo at commit T — { repoFullName, commitSha, frozenAt, openPullRequests[], openIssues[], recentDecisions[] } — where every element is filtered to its state as of frozenAt, never its current state.
  2. Leak-proofing is the deliverable, not a side effect. Every field must be derived from data timestamped <= frozenAt. Explicitly: no PR that was opened after T, no issue comment written after T, no label applied after T, no merge/close outcome for anything the agent is about to be asked to predict.
  3. Checksummed and reproducible: a snapshotChecksum over canonicalized content (same discipline as checksumCases in scripts/backtest-corpus-export-core.ts — sorted keys, sha256), so a third party can confirm two runs scored the same task.
  4. Pure core + thin IO wrapper, matching the established repo pattern: the filtering/canonicalization/checksum logic is pure and unit-tested at 100%; a separate CLI does the GitHub/DB reads.
  5. Invariant tests are the point of this issue. At minimum: a fixture containing post-T records must produce a snapshot that provably excludes them; two builds of the same snapshot must produce an identical checksum; a snapshot built at T must never differ based on when the build ran.

Deliverables

  • scripts/frozen-repo-snapshot-core.ts (pure) + scripts/frozen-repo-snapshot.ts (CLI)
  • Unit tests at 100% branch coverage, including the leak-proofing invariants above
  • Documented snapshot schema with an explicit statement of what is and is not included

Expected outcome

A task snapshot that can be handed to an untrusted agent with confidence that any score derived from it reflects prediction, not hindsight.

References

Sub-issue of #9216. Primitives to mirror: scripts/backtest-corpus-export-core.ts. Epic #8534.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions