Research: universal Codex REQUEST_CHANGES patterns + targeted Codex false-alarm prompt

[waleedkadous]

Research goal

Codex's REQUEST_CHANGES feedback during CMAP cycles costs us iter time. Two questions:

1. What are the UNIVERSAL patterns Codex flags — recurring objections that generalize across protocols and subsystems, not project-specific catches?
2. What does Codex consistently FALSE-ALARM on — so we can give it an explicit "you tend to over-flag X" instruction in its consult prompt?

Why now

Initial pass via background sub-agent (codev/research/codex-request-changes-patterns.md) processed 71 rebuttals files in 234s and surfaced 12 patterns + a ~24% false-positive estimate. The architect wants:

• Deeper, context-aware analysis — investigators must read the underlying spec, plan, and actual code for each rebuttal, not just the rebuttal text. Determines whether Codex was responding to genuine ambiguity or hallucinating.
• Across multiple projects — full scan of all codev/projects/*/ rebuttals files (currently 71, plus any we can recover from PR comments / git history).
• Universal-only synthesis — strip project-specific catches; keep what generalizes.

Two deliverables

1. codev/research/codex-request-changes-patterns.md (replaces current draft)

Data-derived universal tips for architects + builders to follow when drafting specs/plans, ordered by frequency × generalizability. Each tip must be:

• Phrased actionably ("When X, do Y")
• Backed by ≥3 distinct projects (not just multiple iters of the same project)
• Universal — applies regardless of subsystem
• Specific enough to be testable in a draft spec/plan ("did I do this?")

Suggested structure: TL;DR top 5-8 tips, then detailed patterns by frequency, then "Codex false alarms" section feeding deliverable #2.

2. codev/research/codex-false-alarm-prompt.md (NEW)

A drop-in prompt fragment to append to the Codex consult system prompt (packages/codev/skeleton/consult-types/integration-review.md and per-protocol equivalents). Tells Codex what it consistently over-flags so it self-corrects before issuing REQUEST_CHANGES. Examples of known false-alarm modes from the initial pass:

• Flagging missing Playwright tests in repos with no Playwright infrastructure
• Flagging "tests don't exercise the actual handler" when handlers are thin orchestrators over tested primitives
• Misreading porch's pending-gate semantics as "incomplete"
• Treating intentional dual-mode phases as incomplete migrations
• Codex literally cannot run tests in its sandbox (EPERM) and flags any test it can't validate

Format: concise instruction block (≤500 words), written in second-person ("Before issuing REQUEST_CHANGES, verify…"), with each known false-alarm pattern as a check.

Method (RESEARCH protocol, context-aware)

Each of the 3 investigators (Gemini, Codex, Claude):

• Reads all 71 rebuttals files
• For each rebuttal, also reads the corresponding codev/specs/<id>-*.md, codev/plans/<id>-*.md, and the merged PR's diff (via gh pr diff <pr> where the PR can be found in pr_history of status.yaml or by grepping commit messages)
• Determines for each Codex objection: was it (a) genuinely actionable, (b) addressed correctly in spec/plan but builder failed to defend, or (c) hallucinated / out-of-context
• Identifies the universal pattern behind it

Synthesis: cross-reference all three investigators; consensus patterns make the tips list; disagreements get flagged in the report; false-alarm patterns feed deliverable #2.

Scope boundaries

• In scope: all 71 rebuttals files under codev/projects/*/, plus the underlying spec/plan/PR diff for context
• Out of scope: PR comments on Amr's direct-merge PRs (those didn't use CMAP); any rebuttals filed since bugfix-742 that weren't committed to main (those project dirs were cleaned up)

Acceptance criteria

• Both files written, committed, and reviewed
• Top tips list cites ≥3 distinct projects per tip
• False-alarm prompt is ≤500 words, drop-in ready
• Critique-phase (Phase 4) verdicts from all 3 models documented
• At least one tip overturned, refined, or merged based on critique feedback (i.e., critique phase is real, not rubber-stamp)

Suggested protocol

RESEARCH (afx spawn <id> --protocol research) — multi-phase, 3-way parallel investigation, then synthesis + critique. Architect approves at scope-approval before the investigation phase fires.