feat(miner): add dry-run full-execution mode to the cross-repo evaluation harness - #7672
Conversation
…tion harness Extends the cross-repo evaluation harness (JSONbored#4788) from readiness-only to a full-execution dry-run mode (JSONbored#7634): a new --full-execution CLI flag runs the discover -> plan -> code -> test loop against the benchmark repos, driving the configured coding agent to generate a real diff in the local clone, building it, running the target repo's own test suite locally, and flagging a no-op diff. Dry-run only -- it edits the clone and hard-resets it; it never opens a PR, pushes, or writes to any third-party repo. Adds the CROSS_REPO_EXECUTION_CATEGORY taxonomy (plan_not_formed, code_build_failed, tests_failed, no_op_diff, plus shared clone_setup/other), evaluateRepoFullExecution / runFullCrossRepoExecution (readiness reused for the plan stage; code/build/test delegated to injectable seams so the orchestration is unit-testable without a live agent), and summarize/format helpers in the same shape the readiness report produces. The CLI wires real dry-run seams (child_process build/test in the clone; the driver via runCodingAgentAttempt + git diff capture + reset). Docs updated.
…ensive arms Covers the non-Error agent throw, the build/test no-detail reason fallbacks, and the null-manifest guard directly, and narrows the post-readiness stack to its detected shape so the (unreachable-after-readiness) optional-chain arms are gone -- bringing the new code to full patch coverage.
…xecution Split the stack-detected command and spawn it directly (shell:false) instead of sh -c, so there is no shell-metacharacter interpretation of the command string.
…execution evaluateRepoFullExecution hardcoded built:true in the tests-failed, no-op, and success results even when no build ran (no inferred build command or no build runner wired), misreporting a skipped build as built. Track built explicitly: null when skipped, true only after a build actually ran and passed, false on a build failure. Adds regression tests for the skipped-build paths.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7672 +/- ##
=======================================
Coverage 88.51% 88.51%
=======================================
Files 724 724
Lines 75992 76068 +76
Branches 22618 22646 +28
=======================================
+ Hits 67263 67332 +69
- Misses 7681 7688 +7
Partials 1048 1048
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:feature
|
Caution 🛑 LoopOver review result - reject/close recommendedReview updated: 2026-07-21 07:30:57 UTC
Review summary Blockers
Nits — 6 non-blocking
Why this is blocked
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
LoopOver is closing this pull request on the maintainer's behalf (Linked issue overlaps another open PR; AI reviewers agree on a likely critical defect: The new CLI seam wiring in packages/loopover-miner/scripts/cross-repo-evaluation.mjs (buildAgentAttemptSeam, runFullCrossRepoExecutionCli, spawnRepoCommand, resetRepo) has zero unit test coverage — the test suite explicitly skips it per its own comment, and the coding-agent lifecycle (writing acceptanceCriteriaPath, git diff capture, hard-reset in `finally`, tempdir cleanup) is the highest-risk new code in this PR (it edits a real clone) yet is unverified.; duplicate of open PR #7668). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed. |
1 participant
Closes #7634
Summary
Extends the cross-repo evaluation harness (#4788) from readiness-only to a full-execution dry-run mode, so it answers not just "can the miner form a plan for this repo?" but "does the miner actually produce working, correct code?" — the real signal behind the #4810 launch-readiness bar.
A new
--full-executionflag on the existingpackages/loopover-miner/scripts/cross-repo-evaluation.mjsruns the discover → plan → code → test loop against the benchmark repos: it drives the configured coding-agent driver to generate a real diff in the local clone, builds the result, runs the target repo's own test suite locally, and checks the diff isn't a no-op. Dry-run only — it edits the clone and hard-resets it afterward; it never opens a PR, never pushes, and needs nothing beyond a local clone + a configured driver (the same safety posture the readiness harness already has).What's added
CROSS_REPO_EXECUTION_CATEGORY), extending the readiness taxonomy per the issue:plan_not_formed,code_build_failed,tests_failed,no_op_diff(tests passed but the diff was empty), plus the sharedclone_setup/other.evaluateRepoFullExecution/runFullCrossRepoExecution— reuseevaluateRepoReadinessfor the plan stage, then delegate the code/build/test steps to injectable seams (runAgentAttempt,buildRepo,runRepoTests) so the orchestration + taxonomy are unit-testable without a live agent.summarizeCrossRepoExecution/formatCrossRepoExecutionReport— an execution report in the same shape the readiness report already produces.--full-executionmode wiring real dry-run seams: build/test spawn the repo's own commands in the clone; the agent seam runs the configured driver viarunCodingAgentAttemptand capturesgit diff, then resets.docs/cross-repo-evaluation.mddocuments the new mode, taxonomy, and operator prerequisites.Scope
--full-executionmode on the existing CLI + lib module (no new script; no live PR submission against benchmark repos)packages/loopover-miner/**,test/**), noblockedPaths; no secretssite//CNAME/lovableeditsTest Coverage
packages/loopover-miner/**is outside Codecov'scoverage.include(src/**only), so no patch-coverage gate applies — but the new logic has real unit tests (test/unit/miner-cross-repo-full-execution.test.ts) following this package's conventions: every execution category and the pass path are exercised with injected fake seams, plusrunFullCrossRepoExecution, the summary, the report, and the CLI arg/dispatch wiring.Validation
tsc -p packages/loopover-miner/tsconfig.json— new module compiles clean; emitted.js/.d.tscommittednode --checkon the emitted.jsand the CLI script--full-executionis additive); CLI--helpdocuments the new flagvitest runon the new test suite — greenmainSafety