Skip to content

[Phase 2] Rewire the deep-review + rubber-duck tier prompts to consume pre-fed context #1104

Closed as not planned

Description

@github-actions

Story

As a pipeline maintainer,
I want update the deep-review and rubber-duck prompts to read the pre-fed metadata/diff files when present instead of running gh pr view / gh pr diff, while keeping dynamic exploration tools available,
so that the two tier-2 reviewers stop spending tool round-trips and reasoning tokens on deterministic re-fetching, capturing most of the ET saving.

Acceptance Criteria

  1. When the prefetch flag is on and PR_CONTEXT_* files are present, prompts/deep-review.md and prompts/rubber-duck.md instruct the model to read the pre-fed metadata + diff files instead of running gh pr view / gh pr diff for the primary PR context.
  2. When the flag is off (or the files are absent), both prompts fall back to the existing in-loop gh pr view / gh pr diff steps unchanged — behavior is byte-identical to today.
  3. Both prompts explicitly retain permission to use Bash/Read/Grep/Glob and gh api for DYNAMIC needs (cross-file reads, org-standard lookups, secret-scan MCP) — only the deterministic metadata+diff fetch is replaced.
  4. The pre-fed diff the deep tier consumes is the FULL diff (per Story 2), so the tier's reviewable surface is not reduced vs today.
  5. The deep-review held-out eval is run via scripts/evals/run-eval.sh deep-review (5 cases, llm-judge scorer, pass_threshold 0.7) with the prompt change applied and the flag on, and its aggregate score is NOT lower than the flag-off baseline (no regression) — it is not required to improve.

Tasks / Subtasks

Dev Notes

  • The deep tier reads $SAFETY_CHECKS_FILE, $ADVISORY_BOT_FEEDBACK_FILE, $DOWNSTREAM_IMPACT_FILE today (prompts/deep-review.md Inputs). Add $PR_CONTEXT_METADATA_FILE / $PR_CONTEXT_DIFF_FILE the same way, with an explicit 'if set, read these INSTEAD of gh pr view/diff; if unset, run the gh commands as before' instruction — this keeps the flag-off path byte-identical (required for rollback + holdout-eval stability, per the SAFETY_CHECKS precedent).
  • Do NOT weaken the tier's scope guardrails: keep the FORBIDDEN: gh search prs / gh pr list / enumeration block and the secret-scan MCP step. The model still needs Bash/Read/Grep/Glob (granted by engine.sh run_agentic/run_duck) for dynamic cross-file analysis — only the two deterministic fetches move to files.
  • rubber-duck runs cross-engine (often a different model family, e.g. Gemini via run_duck); its prompt's gh pr view --json ...,repository,... field list differs slightly from deep's — ensure the pre-fed metadata superset (Story 2) covers repository too, or the duck prompt reads defensively.
  • Regression gate: scripts/evals/run-eval.sh deep-review scores against evals/deep-review/holdout/cases.jsonl using the llm-judge scorer (pass_threshold in scorer.json). The held-out cases are immutable (scripts/lib/holdout-guard.sh, CODEOWNERS over evals/) — never edit them to make the number move.
  • This story is independently reviewable: it touches only the two prompt files and their eval run; it depends on Story 2's plumbing but not on Story 4.
  • Eval mechanics: scripts/evals/run-eval.sh executes the skill prompt once per held-out case via engine.sh run_triage with the case context INLINED (not fetched via tools). So the eval already exercises pre-fed-style context — a valid guard that the rewritten prompt still reasons correctly on inlined metadata/diff. It cannot catch a full-diff-vs-truncated mismatch; that is covered by Story 2's full-diff constraint and Story 6's A/B smoke.

Project Structure Notes

Touches prompts/deep-review.md and prompts/rubber-duck.md only. Eval invocation via existing scripts/evals/run-eval.sh; no eval-case edits (guarded).

References

  • prompts/deep-review.md#inputs
  • prompts/rubber-duck.md#steps
  • scripts/evals/run-eval.sh
  • evals/deep-review/holdout/cases.jsonl
  • scripts/lib/holdout-guard.sh

Likely target surface

  • prompts/deep-review.md
  • prompts/rubber-duck.md

Story prepared by the BMAD Scrum Master (Bob) for epic #1101. Status: ready-for-dev.

Activity

  1. don-petry commented on Jul 10, 2026

    @don-petry
    Collaborator

    Dev-Lead Implementation Plan

    Issue: #1104 — [Phase 2] Rewire the deep-review + rubber-duck tier prompts to consume pre-fed context

    Scope

    Update prompts/deep-review.md and prompts/rubber-duck.md so that when the Story 2 prefetch flag is on and the PR_CONTEXT_METADATA_FILE / PR_CONTEXT_DIFF_FILE files are present, the model reads them instead of running gh pr view / gh pr diff; when unset/absent, fall back to the existing gh commands unchanged.

    Implementation Checklist

    Tests to Write

    • N/A — pure prompt/documentation change (Phase 2 TDD exception). Behavior-preserving when flag off (byte-identical fallback path). Existing tests/test_prefetch_context.bats covers Story 2 plumbing; no prompt-content assertion tests exist to extend.

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh + scripts/evals/run-eval.sh deep-review (held-out regression gate, AC #5)

  2. don-petry commented on Jul 10, 2026

    @don-petry
    Collaborator

    Dev-Lead: cannot implement issue #1104 — needs human attention

    A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

    This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

    Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-1104-20260710-1251 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.

  3. removed
    dev-leadFor dev-lead agent pickup
    dev-lead:needs-humandev-lead could not complete this issue; needs human attention
    on Jul 14, 2026
  4. don-petry commented on Jul 14, 2026

    @don-petry
    Collaborator

    Split & superseded. dev-lead timed out (exit 124, 1800s action budget) implementing both prompt files in one pass — a non-retryable timeout. Per the escalation guidance, this is split into two independently-implementable, per-file stories that each fit the action budget:

    Both are labeled dev-lead and linked as sub-issues of epic #1101. Closing this as superseded; no code from the timed-out run was recoverable (ephemeral runner, unpushed branch). — Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions