Skip to content

Harden benchmark source pinning across Smithers worktrees #139

Closed

Description

@aviggiano

Problem

Benchmark workers can validate a parent checkout at an immutable revision while per-task Smithers worktrees still derive from mutable branch defaults or rebase targets. That can make the effective agent-visible source differ from the configured benchmark source.

This issue intentionally omits private benchmark repositories, revisions, ground truth, and security findings.

Proposed hardening

  • Build one reusable sanitized source archive containing only the configured pinned commit and tree.
  • Remove the original repository metadata, remotes, mutable branches, tags, reflogs, alternate object stores, and unrelated objects before packaging.
  • Retain only the minimal shallow Git metadata required for Smithers worktrees, under an explicit internal pinned ref.
  • Never clone, fetch, or resolve a mutable source ref in benchmark workers.
  • Set every Smithers worktree base explicitly to the internal pinned ref; never fall back to main or origin/main.
  • Before the first model call in every final task worktree, verify the exact commit and tree, verify the allowed ref inventory, and persist source-proof evidence.
  • Require paired benchmark variants and all repetitions to use the same source archive digest.
  • Keep evaluation ground truth outside the bug-finding workers and materialize it only in the evaluator phase.
  • Fail closed and invalidate a row when any source proof is absent or mismatched.

Acceptance criteria

  • A regression fixture places a different revision on main while the benchmark is pinned to an older commit.
  • Every task worktree starts at the configured pinned commit and tree.
  • main, origin/main, remotes, tags, and unrelated commit objects are unavailable inside the worker repository.
  • Resume remains based on the explicit pinned ref.
  • Worker startup performs no target clone or fetch.
  • Source proof is saved before model execution and included in benchmark artifacts.
  • Evaluation data is unavailable to target agents.
  • Documentation describes the fail-closed validity contract without disclosing private benchmark data.

Activity

  1. aviggiano commented on Jul 27, 2026

    @aviggiano
    CollaboratorAuthor

    Implementation status:

    • Prepared isolated pinned-source hardening and incorporated the artifact-preservation work from Preserve verified artifacts before Smithers worktree cleanup #125.
    • The target materializer now admits only one explicitly pinned commit, removes remote/history reachability, rejects unsupported repository shapes, and writes fail-closed source proofs before agent work.
    • Added frozen-runtime compatibility coverage and validated the relevant build, typecheck, and generated-workflow suites.
    • Started a new 10-row fresh eval generation on new images and volumes. Launch gate: 10/10 parent proofs valid, 10/10 initial worktree proofs valid, 10/10 model workers live, 0 failed workflow nodes.
    • The earlier failed canary generation remains quarantined and preserved; it was not resumed or overwritten.

    No benchmark target, ground truth, pinned source identifiers, or findings are included here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions