Skip to content

Extend the cross-repo evaluation harness from readiness-only to full execution on benchmark repos #7634

Description

@JSONbored

Context

packages/loopover-miner/lib/cross-repo-evaluation.js (#4788) benchmarks whether AMS can attempt a repo — it exercises detectRepoStack, resolveMinerGoalSpec, and buildCodingTaskSpec against six diverse public repos (Node/Python/Rust/Go, see packages/loopover-miner/benchmarks/cross-repo/manifest.json) without ever running the coding agent or calling forge APIs (confirmed in PR #5790's own description: "readiness-only"). That answers "can the miner form a plan for this repo," not "does the miner actually produce working, correct code for this repo" — which is the real open question behind #4810's launch-readiness pilot bar (a real external repo, end-to-end, no manual intervention).

Requirements

Extend the harness with a full-execution mode that actually runs the discover → plan → code → test loop against a subset of the existing benchmark repos, through generating a real diff and running the target repo's own test suite locally — dry-run only: no live GitHub PR submission, no write access to the third-party repos, no credentials beyond a local clone. This is read/execute-locally-and-discard, the same safety posture the existing readiness harness already has, just one step further. Report pass/fail per repo with concrete failure categories (e.g. plan formed but code didn't compile, code compiled but tests failed, tests passed but diff was a no-op) — extend the existing report taxonomy, don't replace it.

⚠️ This issue's deliverable is a --full-execution (or equivalently named) CLI mode added to the existing packages/loopover-miner/scripts/cross-repo-evaluation.mjs script and its underlying lib module — not a new script, and not anything that opens a real PR against any of the benchmark repos. A PR that adds live PR-submission against a third-party benchmark repo does NOT satisfy this issue and will be closed.

Deliverables

  • --full-execution mode on the existing CLI, running against at least 2 of the current 6 benchmark repos in dry-run.
  • Extended pass/fail report with execution-specific failure categories, in the same report format the existing readiness mode already produces.
  • packages/loopover-miner/docs/cross-repo-evaluation.md updated to document the new mode.

Test Coverage Requirements

packages/loopover-miner/** is outside coverage.include (src/** only per codecov.yml) — no Codecov patch-coverage gate applies here, but new logic still needs real unit tests following this package's existing test conventions (see test/unit/miner-cross-repo-evaluation.test.ts for the pattern this extends).

Expected Outcome

Real signal on whether AMS's generalized agent actually produces working code on genuinely external repos, not just whether it can form a plan for one — the missing piece before committing further platform/hosting investment on top of it.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    In Progress

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions