Review agent: escalate PR-description-vs-diff discrepancies above Info severity

What happened

On PR #2179, the review agent detected that the PR description claimed to fix a pre-existing broken link (-inference.md → getting-inference.md) in docs/guides/README.md, but this change was not present in the diff. The agent correctly identified this discrepancy but rated it as Info severity under the broken-link-fix tag. The human reviewer approved the PR without addressing this or any other finding.

The review agent ran as workflow 27355898627 (2026-06-11 14:55–15:09 UTC). The human approved on 2026-06-12 with an empty review body.

What could go better

When a PR description explicitly claims a change that isn't reflected in the diff, this indicates one of two problems: (1) the change was accidentally omitted and should be included, or (2) the PR description is inaccurate and misleading for future readers. Either case is more significant than Info severity.

I'm fairly confident this is a repeatable pattern. Agent-generated PRs include structured summaries, and human-authored PRs often describe intended changes that may not all land in the final diff. Catching these discrepancies at a higher severity would increase the chance of human engagement.

Uncertainty: I don't have data on how often this pattern occurs across other PRs. It's possible this is rare enough that escalating severity would add noise rather than signal. A sample of 10-20 PRs with Info-level findings would help validate frequency.

Proposed change

In the review agent's finding classification logic (likely in the code-review skill or review sub-agent definitions), add a heuristic that detects when the PR description/body explicitly claims a change (e.g., 'Fixes a broken link in X', 'Adds Y to Z') that cannot be corroborated by the diff. When detected, classify this as Medium severity with a tag like description-diff-mismatch rather than Info.

The detection should look for action verbs in the PR body (fixes, adds, removes, updates, changes) paired with specific file or path references, then verify those files appear in the diff with corresponding changes. This is a targeted check, not a general NLP problem — PR descriptions tend to use predictable phrasing.

This change belongs in the review agent's finding evaluation layer, not in per-repo configuration.

Validation criteria

Over the next 20 review agent runs on PRs with non-trivial descriptions:

At least 1-2 PRs should trigger the description-diff-mismatch finding when applicable.
False positive rate should be below 20% (i.e., at most 1 in 5 triggered findings should be incorrect).
When triggered, the finding should be rated Medium, not Info.
Spot-check 5 triggered findings manually to confirm they identify genuine discrepancies between description and diff.

Generated by retro agent from #2179

🤖 Finished Triage · ✅ Success · Started 6:28 AM UTC · Completed 6:34 AM UTC
Commit: 4e21a60 · View workflow run →

Per the request to re-check relevance: this remains a valid, unimplemented gap. As of 2026-09-22, the intent-coherence finding taxonomy in fullsend-ai/agents's skills/pr-review/SKILL.md still has no description-diff-mismatch category, and skills/pr-review/sub-agents/intent-coherence.md still contains no logic for cross-checking PR-description claims against the diff. No PR (open or merged) in either fullsend-ai/fullsend or fullsend-ai/agents addresses this, and there is no closing PR reference on this issue.

Two related retro-agent issues have since been filed and remain open: one proposing a Low-severity check for PR descriptions with structured Changes tables that list files absent from the diff, and another proposing a Low-severity check for explicit default-value claims that contradict the diff. Both are narrower variants of the same underlying description-vs-diff verification idea proposed here at Medium severity. Since this issue predates both, it is not being treated as a duplicate, but a maintainer may want to consolidate the three into a single implementation effort rather than building three overlapping heuristics.

This is left open for human prioritization rather than auto-dispatched, consistent with the request to treat this as a stale-issue relevance check rather than an implementation trigger.

Labels: The issue proposes new review-agent capability (not a defect fix), matching the type/feature label; existing agent/review, triaged, and component/harness labels remain accurate.

Next steps:

/fs-code — agent creates a PR to implement this issue
/fs-code <your instruction> — agent implements with your specific guidance

Previous run

Triage Summary

Severity: Medium · Category: Feature

Problem

The review agent rates PR-description-vs-diff discrepancies as Info severity, which means they are routinely overlooked during human review. When a PR description claims a change that isn't in the diff, this indicates either an accidentally omitted change or a misleading description — both warrant reviewer attention.

Root Cause

The intent-coherence finding dimension in the pr-review skill lacks a dedicated category for description-diff mismatches. The closest existing category (misleading-label) covers issue label mismatches, not PR body accuracy. Without a specific category and severity mapping, these findings default to Info.

Recommended Fix

Add a description-diff-mismatch category to the intent-coherence dimension in skills/pr-review/SKILL.md (finding category taxonomy, around lines 195–202)
Update skills/pr-review/sub-agents/intent-coherence.md with a heuristic that detects action verbs in the PR body (fixes, adds, removes, updates, changes) paired with file/path references, then verifies those files appear in the diff
Default severity for this category should be Medium

Proposed Test Case

Scenario: PR description claims a file change not present in the diff
  Given a PR whose body says "Fixes broken link in docs/guides/README.md"
  And the diff does not touch docs/guides/README.md
  When the review agent runs
  Then a finding with tag "description-diff-mismatch" is emitted
  And the finding severity is Medium
  And the finding identifies the specific unsubstantiated claim

Scenario: PR description claims match the actual diff (negative case)
  Given a PR whose body says "Updates docs/guides/README.md"
  And the diff includes changes to docs/guides/README.md
  When the review agent runs
  Then no "description-diff-mismatch" finding is emitted

Validation Criteria

The issue includes well-defined validation criteria: over 20 review runs, expect 1–2 true positives with <20% false positive rate. Spot-check 5 triggered findings manually to confirm genuine discrepancies.

Labels: This is a review agent feature request for improving finding classification in the pr-review skill/harness.

RICE Priority Score: 1.2

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	The review agent runs on all PRs across the platform, so an improvement to its finding classification affects most active users. However, the specific description-diff mismatch pattern likely triggers on a subset of PRs, not all. Score 1.5 reflects broad but not universal reach.
Impact	1	This is a noticeable workflow improvement — catching misleading PR descriptions or accidentally omitted changes helps maintain code quality. However, it's not blocking any workflow; the current Info-level findings are just being overlooked. Score 1 for medium impact.
Confidence	0.8	The issue is exceptionally well-described with specific files to modify, concrete validation criteria, and a detailed triage comment identifying exact locations in the codebase. The main uncertainty is the frequency of the pattern in practice, which the issue itself acknowledges.
Effort	1	The triage identifies specific files (skills/pr-review/SKILL.md and sub-agents/intent-coherence.md) and a clear heuristic to implement. It's a targeted change touching a few files with well-scoped logic, plus test scenarios are already defined. Score 1 for medium effort.

Formula: (1.5 x 1 x 0.8) / 1 = 1.2

/fs-triage lots of things have changed between when this was originally filed and now across both the https://github.com/fullsend-ai/fullsend and https://github.com/fullsend-ai/agents repos. Is this still relevant? This is a stale-issue review, not an implementation request: even if the issue is still a valid bug and has sufficient information, do not recommend or trigger ready-to-code yet. Leave it for human prioritization after confirming whether it remains relevant. Do not close a still-relevant bug merely to avoid coding it.

🤖 Finished Triage · ✅ Success · Started 1:57 PM UTC · Completed 2:01 PM UTC

Commit: b9e5099 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.29

Navigation Menu

Repository navigation

Review agent: escalate PR-description-vs-diff discrepancies above Info severity #2280

What happened

What could go better

Proposed change

Validation criteria

Triage Summary

Problem

Root Cause

Recommended Fix

Proposed Test Case

Validation Criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Review agent: escalate PR-description-vs-diff discrepancies above Info severity #2280

Description

What happened

What could go better

Proposed change

Validation criteria

Activity

fullsend-ai-triage commented on Jun 15, 2026

fullsend-ai-triage commented on Jun 15, 2026

Triage Summary

Problem

Root Cause

Recommended Fix

Proposed Test Case

Validation Criteria

fullsend-ai-prioritize commented on Jun 15, 2026

ralphbean commented on Sep 22, 2026

fullsend-ai-triage commented on Sep 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions