Skip to content

Review agent should flag PR description claims that contradict the actual diff #3935

New issue
New issue
Open
Open
Review agent should flag PR description claims that contradict the actual diff#3935
Labels
agent/reviewReview agentfeatureFeature-category issue awaiting human prioritizationpriority/lowNice to have, address when convenienttriagedTriaged but awaiting human prioritizationtype/featureNew capability request

Description

@fullsend-ai-retro
fullsend-ai-retro
bot
opened on Jul 9, 2026

What happened

In PR quay/quay#6029, the description stated FEATURE_PUBLIC_OAUTH_APPS defaulted to True while config.py set it to False. The contributor updated the code default to False in response to review feedback on May 27 (commit 33e0393) but never updated the PR description. This inconsistency was only caught in the 4th review round on July 9 (day 51) by bcaton85, who noted the JIRA ticket and PR description both said default True but config.py set it to False. The PR was merged without the description being updated.

What could go better

After the code was changed to default=False on May 27, subsequent review rounds on July 8-9 should have caught that the PR description was now stale. An automated check comparing explicit value claims in the description against the diff would have flagged this immediately, avoiding confusion for future readers and the JIRA record. This is especially important for security-sensitive configuration changes where the default value determines whether existing deployments are affected.

Proposed change

Add a lightweight review agent heuristic that parses the PR description for explicit default-value or configuration claims (patterns like 'default: True', 'defaults to X', 'set to Y') and cross-references them against assignment statements in configuration files visible in the diff. When a mismatch is found, emit a low-severity finding suggesting the description be updated. This catches the common case where code evolves across review rounds but the description is not refreshed.

Validation criteria

On a PR where the description claims a specific default value (e.g., 'default: True') that differs from the value assigned in the diff (e.g., FEATURE_X = False in config.py), the review agent should produce a finding noting the inconsistency.

Generated by retro agent from quay/quay#6029

Activity

  1. github-project-automation
    added this to fullsend-ai Workstreamon Jul 9, 2026
  2. github-project-automation
    moved this to Todo in fullsend-ai Workstreamon Jul 9, 2026

  3. fullsend-ai-prioritize commented on Aug 1, 2026

    @fullsend-ai-prioritize
    fullsend-ai-prioritizeBot
    on Aug 1, 2026 – with fullsend-ai-prioritize

    RICE Priority Score: 1.07

    Score breakdown
    Dimension Score Reasoning
    Reach 2 This enhancement applies to the review agent, which runs on all PRs across orgs. Any PR that includes configuration or default-value claims in its description would benefit from this check, making it relevant to most active users of the platform.
    Impact 1 This is a medium-impact improvement. It catches stale PR descriptions that contradict the diff, which prevents confusion for reviewers and future readers. However, the finding is low-severity, and the existing workaround is manual review (which already caught the issue in the referenced PR, albeit late). The security-sensitive configuration angle adds some weight.
    Confidence 0.8 The issue is well-described with a concrete real-world example (quay/quay#6029), clear validation criteria, and a specific proposed implementation approach. The scope is well-bounded: parse description for value claims, cross-reference against diff assignments, emit a finding on mismatch.
    Effort 1.5 Moderate complexity. Requires implementing description parsing for default-value patterns (e.g., 'default: True', 'defaults to X', 'set to Y'), cross-referencing matched claims against assignment statements in configuration files visible in the diff, and emitting findings. This touches the review agent's heuristic system, needs regex/pattern matching logic, and requires testing against various description formats and diff patterns.

    Formula: (2 x 1 x 0.8) / 1.5 = 1.07

  4. ralphbean commented on Sep 23, 2026

    @ralphbean
    ralphbean
    on Sep 23, 2026
    Member

    /fs-triage lots of things have changed between when this was originally filed and now across both the https://github.com/fullsend-ai/fullsend and https://github.com/fullsend-ai/agents repos. Is this still relevant? This is a stale-issue review, not an implementation request: even if the issue is still a valid bug and has sufficient information, do not recommend or trigger ready-to-code yet. Leave it for human prioritization after confirming whether it remains relevant. Do not close a still-relevant bug merely to avoid coding it.

  5. fullsend-ai-triage commented on Sep 23, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Sep 23, 2026 – with fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 7:03 AM UTC · Completed 7:07 AM UTC

    Commit: 5c8aefa · View workflow run →

    Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.33

  6. fullsend-ai-triage
    added
    featureFeature-category issue awaiting human prioritization
    triagedTriaged but awaiting human prioritization
    type/featureNew capability request
    agent/reviewReview agent
    priority/lowNice to have, address when convenient
    on Sep 23, 2026

  7. fullsend-ai-triage commented on Sep 23, 2026

    @fullsend-ai-triage
    fullsend-ai-triageBot
    on Sep 23, 2026 – with fullsend-ai-triage

    This issue was re-triaged at a maintainer's request to confirm it is still relevant given how much the review agent's architecture has changed since it was filed (multi-forge support, the pr-review orchestrator fan-out, and verdict/severity refactors).

    The underlying gap still exists: the review agent's guidance to treat PR descriptions as unverified claims (present in agents/review.md and the pr-review skill) predates this issue and is general — it has not been extended into the specific, repeatable check this issue proposes for config default-value claims. No PR or commit implements this check, and no open/merged PR references this issue.

    Two related open issues cover adjacent but distinct territory and are worth coordinating with rather than treating as duplicates: one proposes escalating the severity of description/diff mismatches in general, and another targets description "Changes table"/file-list mismatches. Neither addresses the default-value-vs-config-assignment pattern described here.

    This remains a valid, well-scoped, low-priority feature request. Per the maintainer's instructions for this stale-issue review, it is left open for human prioritization rather than auto-promoted to ready-to-code.

    Labels: Matches this repo's convention for review-agent feature proposals awaiting prioritization (compare #2280, #1273), and its low RICE score supports a low priority label.

    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/reviewReview agentfeatureFeature-category issue awaiting human prioritizationpriority/lowNice to have, address when convenienttriagedTriaged but awaiting human prioritizationtype/featureNew capability request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions