You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
packages/loopover-miner/lib/cross-repo-evaluation.js (#4788) benchmarks whether AMS can attempt a repo — it exercises detectRepoStack, resolveMinerGoalSpec, and buildCodingTaskSpec against six diverse public repos (Node/Python/Rust/Go, see packages/loopover-miner/benchmarks/cross-repo/manifest.json) without ever running the coding agent or calling forge APIs (confirmed in PR #5790's own description: "readiness-only"). That answers "can the miner form a plan for this repo," not "does the miner actually produce working, correct code for this repo" — which is the real open question behind #4810's launch-readiness pilot bar (a real external repo, end-to-end, no manual intervention).
Requirements
Extend the harness with a full-execution mode that actually runs the discover → plan → code → test loop against a subset of the existing benchmark repos, through generating a real diff and running the target repo's own test suite locally — dry-run only: no live GitHub PR submission, no write access to the third-party repos, no credentials beyond a local clone. This is read/execute-locally-and-discard, the same safety posture the existing readiness harness already has, just one step further. Report pass/fail per repo with concrete failure categories (e.g. plan formed but code didn't compile, code compiled but tests failed, tests passed but diff was a no-op) — extend the existing report taxonomy, don't replace it.
⚠️ This issue's deliverable is a --full-execution (or equivalently named) CLI mode added to the existingpackages/loopover-miner/scripts/cross-repo-evaluation.mjs script and its underlying lib module — not a new script, and not anything that opens a real PR against any of the benchmark repos. A PR that adds live PR-submission against a third-party benchmark repo does NOT satisfy this issue and will be closed.
Deliverables
--full-execution mode on the existing CLI, running against at least 2 of the current 6 benchmark repos in dry-run.
Extended pass/fail report with execution-specific failure categories, in the same report format the existing readiness mode already produces.
packages/loopover-miner/docs/cross-repo-evaluation.md updated to document the new mode.
Test Coverage Requirements
packages/loopover-miner/** is outside coverage.include (src/** only per codecov.yml) — no Codecov patch-coverage gate applies here, but new logic still needs real unit tests following this package's existing test conventions (see test/unit/miner-cross-repo-evaluation.test.ts for the pattern this extends).
Expected Outcome
Real signal on whether AMS's generalized agent actually produces working code on genuinely external repos, not just whether it can form a plan for one — the missing piece before committing further platform/hosting investment on top of it.
Links & Resources
packages/loopover-miner/lib/cross-repo-evaluation.js, packages/loopover-miner/scripts/cross-repo-evaluation.mjs (the harness this extends)
Context
packages/loopover-miner/lib/cross-repo-evaluation.js(#4788) benchmarks whether AMS can attempt a repo — it exercisesdetectRepoStack,resolveMinerGoalSpec, andbuildCodingTaskSpecagainst six diverse public repos (Node/Python/Rust/Go, seepackages/loopover-miner/benchmarks/cross-repo/manifest.json) without ever running the coding agent or calling forge APIs (confirmed in PR #5790's own description: "readiness-only"). That answers "can the miner form a plan for this repo," not "does the miner actually produce working, correct code for this repo" — which is the real open question behind #4810's launch-readiness pilot bar (a real external repo, end-to-end, no manual intervention).Requirements
Extend the harness with a full-execution mode that actually runs the discover → plan → code → test loop against a subset of the existing benchmark repos, through generating a real diff and running the target repo's own test suite locally — dry-run only: no live GitHub PR submission, no write access to the third-party repos, no credentials beyond a local clone. This is read/execute-locally-and-discard, the same safety posture the existing readiness harness already has, just one step further. Report pass/fail per repo with concrete failure categories (e.g. plan formed but code didn't compile, code compiled but tests failed, tests passed but diff was a no-op) — extend the existing report taxonomy, don't replace it.
Deliverables
--full-executionmode on the existing CLI, running against at least 2 of the current 6 benchmark repos in dry-run.packages/loopover-miner/docs/cross-repo-evaluation.mdupdated to document the new mode.Test Coverage Requirements
packages/loopover-miner/**is outsidecoverage.include(src/**only percodecov.yml) — no Codecov patch-coverage gate applies here, but new logic still needs real unit tests following this package's existing test conventions (seetest/unit/miner-cross-repo-evaluation.test.tsfor the pattern this extends).Expected Outcome
Real signal on whether AMS's generalized agent actually produces working code on genuinely external repos, not just whether it can form a plan for one — the missing piece before committing further platform/hosting investment on top of it.
Links & Resources
packages/loopover-miner/lib/cross-repo-evaluation.js,packages/loopover-miner/scripts/cross-repo-evaluation.mjs(the harness this extends)