Skip to content

fix(review): guard hallucinated missing-symbol blockers + collapse nits - #1094

Merged
JSONbored merged 1 commit into
mainfrom
feat/nits-collapsible
Jun 23, 2026
Merged

fix(review): guard hallucinated missing-symbol blockers + collapse nits#1094
JSONbored merged 1 commit into
mainfrom
feat/nits-collapsible

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

The false-close (soak finding)

An adversarial soak of 15 recent decisions found 14/15 correct but one confirmed false close (high severity): the bot auto-closed contributor PR metagraphed #1528 (green CI, mergeable, 5 new MCP tools + tests) on a hallucinated blocker — it claimed loadArtifactData was "not imported → ReferenceError," but that function is defined locally (line 174) and used by ~24 existing call sites; CI passed, empirically disproving it.

Root cause: the model sees only the diff, not the whole file, so new code referencing an existing out-of-diff symbol looks undefined. Both free models made the same mistake → consensus → close. (Full-file grounding is enabled but didn't prevent it here, so a prompt-level guard is the reliable fix; grounding effectiveness is a separate follow-up.)

Fix

  • DIFF SCOPE guard in the review prompt: never report a missing import / undefined symbol / 'X not defined → ReferenceError' as a blocker unless the diff itself removes the definition or introduces the symbol without defining it — otherwise it's at most a nit. Kills the false-close-on-hallucination class.
  • Nits collapsed into a <details> toggle (assessment + blockers stay visible), per request.

Full suite green (3486). I'll reopen #1528 once this deploys.

…lapse nits

Soak finding: the bot FALSE-CLOSED a good contributor PR (metagraphed #1528,
green CI, mergeable) on a hallucinated blocker — it claimed loadArtifactData was
'not imported -> ReferenceError', but that function is defined locally and used by
~24 call sites. Root cause: the model sees only the DIFF, not the whole file, so
new code referencing an existing (out-of-diff) symbol looks undefined; both free
models made the same mistake -> consensus -> close. (Full-file grounding is enabled
but did not prevent it, so a prompt-level guard is the reliable fix.)

- DIFF SCOPE guard in the review prompt: never report a missing import / undefined
  symbol / 'X not defined -> ReferenceError' as a blocker unless the diff ITSELF
  removes the definition or introduces the symbol without defining it; otherwise it
  is at most a nit. Directly kills the false-close-on-hallucination class.
- Nits now render inside a collapsed <details> toggle (assessment + blockers stay
  visible), per request — keeps the comment focused.
@dosubot dosubot Bot added the size:XS label Jun 23, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.77%. Comparing base (f759e88) to head (605e10a).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1094   +/-   ##
=======================================
  Coverage   94.77%   94.77%           
=======================================
  Files         153      153           
  Lines       18552    18553    +1     
  Branches     6707     6707           
=======================================
+ Hits        17582    17583    +1     
  Misses        418      418           
  Partials      552      552           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 26f2c80 into main Jun 23, 2026
17 checks passed
@JSONbored
JSONbored deleted the feat/nits-collapsible branch June 23, 2026 18:45
@ghost

ghost commented Jun 23, 2026

Copy link
Copy Markdown

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review — safe to merge

2 files · 2 AI reviewers · no blockers · readiness 93/100 · CI green · unstable

✅ Approved — safe to merge

Review summary
The update adds a DIFF SCOPE rule to the system prompt and modifies composeAdvisoryNotes to wrap the Nits section in a collapsible <details> block, improving review focus. Corresponding unit tests are adjusted to check for the new markdown structure. The changes compile and the test expectations align with the new output format.

Nits

  • The added comment before the <details> push is long; consider shortening for readability.
  • Using lines.push with multiple arguments is fine but could be clearer as separate pushes or an array spread.
  • Update any documentation that references the previous '**Nits**' heading to reflect the new collapsible format.
  • Consider adding a test case for the scenario where both blockers and nits are present to ensure the ordering of sections remains correct.
Signal Result Evidence
Code review ✅ No blockers 2 reviewers, synthesized
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Review load ✅ 20/20 Readiness component derived from cached public PR metadata and labels; size label size:XS.
Validation evidence ✅ 25/25 PR body includes validation/test evidence.
Open PR queue ✅ 10/10 0 open PR(s), 0 likely reviewable.
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 84 PR(s), 254 issue(s).
Gate result ✅ Passing No configured blocker found.
Nits — 1 non-blocking
  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 84 PR(s), 254 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Review load = cached public PR metadata such as size labels, changed paths, and preflight status.
  • Open PR queue = repo-wide review pressure; it is not a PR quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
Review details

Generated from public PR metadata and the diff. Advisory only; deterministic signals remain authoritative.

The update adds a DIFF SCOPE rule to the system prompt and modifies composeAdvisoryNotes to wrap the Nits section in a collapsible <details> block, improving review focus. Corresponding unit tests are adjusted to check for the new markdown structure. The changes compile and the test expectations align with the new output format.

Nits

  • The added comment before the <details> push is long; consider shortening for readability.
  • Using lines.push with multiple arguments is fine but could be clearer as separate pushes or an array spread.
  • Update any documentation that references the previous '**Nits**' heading to reflect the new collapsible format.
  • Consider adding a test case for the scenario where both blockers and nits are present to ensure the ordering of sections remains correct.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@ghost ghost added gittensory:reviewed gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant