Lab: compare review roles, false findings and repeated review rounds #99

Capture for later. Factory v0.1.26 selects two fresh reviewers as its method default; this Lab test evaluates that baseline and alternatives. Automation defaults remain open. This issue does not start runs or change Factory policy.

The Omarchy review project combines a technical reviewer, a second model looking for edge cases, triage of its leads, and a check of the final claims. Its evaluations are useful examples of prompt tuning and held-out cases, but their small, self-reported samples do not establish results for our work.

Addy Osmani's post concerns a different project: the Claude Code team's review process. The useful experiment ideas are testing reviewers on past incidents and clean changes, counting false positives and harmful suggested fixes, and measuring whether repair/re-review reaches a useful conclusion.

Proposed comparisons, one change at a time:

1. Current independent Factory review versus adding a complementary reviewer with a different search task.
2. With that flow fixed, test a separate evidence/claim check before presenting findings.
3. Separately test reviewing the changed portion after repairs, while retaining checks for newly introduced regressions.

Use matched, pinned cases and settings, including incident-causing and clean cases. For model comparisons, use two independent copies of the same frozen session/thread checkpoint, with identical pre-checkpoint history and candidate state, under the replay protocol in #50. For reviewer comparisons, this means the same review handoff and candidate; it does not give reviewers the implementer's private reasoning. Keep reviewers' conclusions separate until both finish. Hide later commits, completed PRs and expected answers; disposable replay must not recreate external writes. These replay pitfalls are also discussed at 29:16 in the Factory conversation.

Report confirmed defects, false positives, harmful fixes, review/fix rounds, elapsed time, human effort and all-role cost. Keep failures and unknown measurements. Choose the sample, budget and decision criterion before any run; no universal reviewer count, model or round cap is selected here.

A short result and adopt/reject/inconclusive/defer recommendation is sufficient. Any method change gets its own reviewed proposal; no custom reviewer service, public-comment bot or scheduler.

Related: #50 owns replay comparisons; #97 tests ZeroShot-style fresh loops/model cost; #78, #82 and #88 retain review eligibility, usage and human-decision questions. This Lab test targets review quality and noise, not merge eligibility.