You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Harden benchmark source pinning across Smithers worktrees #139
Benchmark workers can validate a parent checkout at an immutable revision while per-task Smithers worktrees still derive from mutable branch defaults or rebase targets. That can make the effective agent-visible source differ from the configured benchmark source.
This issue intentionally omits private benchmark repositories, revisions, ground truth, and security findings.
Proposed hardening
Build one reusable sanitized source archive containing only the configured pinned commit and tree.
Remove the original repository metadata, remotes, mutable branches, tags, reflogs, alternate object stores, and unrelated objects before packaging.
Retain only the minimal shallow Git metadata required for Smithers worktrees, under an explicit internal pinned ref.
Never clone, fetch, or resolve a mutable source ref in benchmark workers.
Set every Smithers worktree base explicitly to the internal pinned ref; never fall back to main or origin/main.
Before the first model call in every final task worktree, verify the exact commit and tree, verify the allowed ref inventory, and persist source-proof evidence.
Require paired benchmark variants and all repetitions to use the same source archive digest.
Keep evaluation ground truth outside the bug-finding workers and materialize it only in the evaluator phase.
Fail closed and invalidate a row when any source proof is absent or mismatched.
Acceptance criteria
A regression fixture places a different revision on main while the benchmark is pinned to an older commit.
Every task worktree starts at the configured pinned commit and tree.
main, origin/main, remotes, tags, and unrelated commit objects are unavailable inside the worker repository.
Resume remains based on the explicit pinned ref.
Worker startup performs no target clone or fetch.
Source proof is saved before model execution and included in benchmark artifacts.
Evaluation data is unavailable to target agents.
Documentation describes the fail-closed validity contract without disclosing private benchmark data.
The target materializer now admits only one explicitly pinned commit, removes remote/history reachability, rejects unsupported repository shapes, and writes fail-closed source proofs before agent work.
Added frozen-runtime compatibility coverage and validated the relevant build, typecheck, and generated-workflow suites.
Started a new 10-row fresh eval generation on new images and volumes. Launch gate: 10/10 parent proofs valid, 10/10 initial worktree proofs valid, 10/10 model workers live, 0 failed workflow nodes.
The earlier failed canary generation remains quarantined and preserved; it was not resumed or overwritten.
No benchmark target, ground truth, pinned source identifiers, or findings are included here.
Activity
aviggiano commentedon Jul 27, 2026
Implementation status:
No benchmark target, ground truth, pinned source identifiers, or findings are included here.