Context
Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Scoring an arbitrary agent against realized history is only meaningful if the agent sees exactly the repository state a maintainer saw at commit T and nothing that happened after it. Any future-information leak silently inflates every score built on top, and it is the single hardest correctness property in this harness — so it gets its own issue, built and tested before anything consumes it.
Requirements
- Snapshot reference: a
FrozenRepoSnapshot describing a repo at commit T — { repoFullName, commitSha, frozenAt, openPullRequests[], openIssues[], recentDecisions[] } — where every element is filtered to its state as of frozenAt, never its current state.
- Leak-proofing is the deliverable, not a side effect. Every field must be derived from data timestamped
<= frozenAt. Explicitly: no PR that was opened after T, no issue comment written after T, no label applied after T, no merge/close outcome for anything the agent is about to be asked to predict.
- Checksummed and reproducible: a
snapshotChecksum over canonicalized content (same discipline as checksumCases in scripts/backtest-corpus-export-core.ts — sorted keys, sha256), so a third party can confirm two runs scored the same task.
- Pure core + thin IO wrapper, matching the established repo pattern: the filtering/canonicalization/checksum logic is pure and unit-tested at 100%; a separate CLI does the GitHub/DB reads.
- Invariant tests are the point of this issue. At minimum: a fixture containing post-T records must produce a snapshot that provably excludes them; two builds of the same snapshot must produce an identical checksum; a snapshot built at T must never differ based on when the build ran.
Deliverables
scripts/frozen-repo-snapshot-core.ts (pure) + scripts/frozen-repo-snapshot.ts (CLI)
- Unit tests at 100% branch coverage, including the leak-proofing invariants above
- Documented snapshot schema with an explicit statement of what is and is not included
Expected outcome
A task snapshot that can be handed to an untrusted agent with confidence that any score derived from it reflects prediction, not hindsight.
References
Sub-issue of #9216. Primitives to mirror: scripts/backtest-corpus-export-core.ts. Epic #8534.
Context
Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Scoring an arbitrary agent against realized history is only meaningful if the agent sees exactly the repository state a maintainer saw at commit T and nothing that happened after it. Any future-information leak silently inflates every score built on top, and it is the single hardest correctness property in this harness — so it gets its own issue, built and tested before anything consumes it.
Requirements
FrozenRepoSnapshotdescribing a repo at commit T —{ repoFullName, commitSha, frozenAt, openPullRequests[], openIssues[], recentDecisions[] }— where every element is filtered to its state as offrozenAt, never its current state.<= frozenAt. Explicitly: no PR that was opened after T, no issue comment written after T, no label applied after T, no merge/close outcome for anything the agent is about to be asked to predict.snapshotChecksumover canonicalized content (same discipline aschecksumCasesinscripts/backtest-corpus-export-core.ts— sorted keys, sha256), so a third party can confirm two runs scored the same task.Deliverables
scripts/frozen-repo-snapshot-core.ts(pure) +scripts/frozen-repo-snapshot.ts(CLI)Expected outcome
A task snapshot that can be handed to an untrusted agent with confidence that any score derived from it reflects prediction, not hindsight.
References
Sub-issue of #9216. Primitives to mirror:
scripts/backtest-corpus-export-core.ts. Epic #8534.