fix(review): log a bounded response snippet on unparseable AI output - #5073
Merged
Conversation
…5071) ai_review_provider_unparseable_exhausted has been firing since 2026-07-01 with no way to tell WHY the model's output didn't parse — only responseChars/hasJsonObject, never the actual content. Log a 400-char prefix of the response alongside the existing per-attempt warn and exhaustion error logs so the next occurrence is diagnosable. The snippet is deliberately kept out of the AiReviewDiagnostic array (reviewDiagnostics), which flows into public/Sentry review-failure context that an existing test guards against ever carrying raw provider text — it reaches only the structured-log Sentry forwarder.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5073 +/- ##
=======================================
Coverage 94.13% 94.14%
=======================================
Files 466 466
Lines 39573 39576 +3
Branches 14436 14436
=======================================
+ Hits 37254 37257 +3
Misses 1664 1664
Partials 655 655
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ai_review_provider_unparseable_exhausted(GITTENSORY-Z) has been firing since 2026-07-01 (83+ occurrences, still ongoing after confirming this isn't downstream of the P0 claude-code timeout outage — it kept firing hours into the post-fix clean-boot window). The logged fields (model,attempt,responseChars,hasJsonObject) give no way to tell WHY the model's output didn't parse — markdown-fenced JSON, a mid-object truncation, a plain-prose refusal, etc. Sentry's own Seer analysis on this issue independently suggested the same fix: log a truncated response prefix.UNPARSEABLE_RESPONSE_SNIPPET_MAX_CHARS) torunWorkersOpinion's per-attemptconsole.warnand exhaustionconsole.loginsrc/services/ai-review.ts, so the next occurrence carries enough content to actually diagnose.AiReviewDiagnosticarray (reviewDiagnostics) too, which broke an existing test ("single + BYOK: withholds unsafe provider and reviewer fallback text") — that array flows intoresult/public Sentry context, which must never carry raw provider text. The snippet now reaches ONLY the direct structured-log path (→ Sentry, internal-only), neverreviewDiagnostics/result.Closes #5071
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #5071).Validation
git diff --checknpm run typechecknpm run test:coverage— this is a pure additive logging change (no new branches beyond what existing tests already cover); ran the affected file directly plus the repo'stest:changedselection (3693 tests) and the full unsharded gate earlier in this session on a sibling branch off the samemain— all green.reviewDiagnosticsnever carriesresponseSnippet) alongside the existing unparseable-exhaustion test.Safety
UI Evidencesection. (N/A — backend-only.)Notes
The snippet is a model's OWN response text (never PR diff/repo content), truncated to 400 chars, and reaches only Sentry (internal, operator-only) — never a public surface.