Implement Closed Issues Case Runner

[jharris1679]

Overview

Implement the "Closed Issues" case type - the gold standard for evaluation. Uses real closed issues from the user's repository with merged PRs as ground truth.

How It Works

1. Scan: Find closed issues with associated merged PRs
2. Extract: Get issue description + pre-PR codebase state
3. Challenge: Agent solves issue from pre-PR state
4. Compare: Compare agent's solution to actual merged PR

Tasks

Issue Detection

• Scan git history for merged PRs
• Extract linked issue (from PR body, branch name, commit message)
• Filter for suitable issues:
  • Has clear description
  • Reasonable size (not massive refactors)
  • Has tests that validate the fix
  • Self-contained (no external dependencies)

Case Extraction

• Parse issue description as prompt
• Identify commit before PR merge
• Extract PR diff as reference solution
• Store case metadata (issue ID, PR ID, commits)

Sandbox Setup

• Checkout repo at pre-PR commit
• Ensure dependencies can be installed
• Verify tests exist and fail before fix

Solution Comparison

• Functional match: Do tests pass?
• Diff similarity: How close to actual PR?
• Scope match: Same files modified?
• Style check: Does it pass linting?

CLI Integration

• sniff closed-issues scan - find suitable issues
• sniff closed-issues run - run agent on closed issues
• sniff closed-issues add <issue> - manually add specific issue

Acceptance Criteria

• Can scan repo and find suitable closed issues
• Extracts issue + PR into runnable case format
• Correctly restores pre-PR state in sandbox
• Agent attempts to solve issue
• Compares solution to actual merged PR
• Shows meaningful similarity metrics

[jharris1679]

Implemented in PR #43