Skip to content

Analyzer silently swallows JSON parse failures, shows misleading 'Analysis complete' #56

Description

@Treelovah

Bug

When the LLM analyzer returns truncated or malformed JSON (e.g. SyntaxError: Unterminated string in JSON at position 3526), parseAnalysisResponse() catches the error, logs it to stderr, and returns a zeroed-out AnalysisResult with narrative.summary: 'Analysis parsing failed'.

The caller (run-flow.ts:555) then unconditionally prints spinnerAnalysis.succeed('Analysis complete') and saves the empty result — giving the user no indication that the analysis actually failed.

What happens

ℹ Running analysis...
Failed to parse analysis response: SyntaxError: Unterminated string in JSON at position 3526  ← stderr, easily missed
✔ Analysis complete  ← misleading

All scores show 0, all fields empty. The user sees "Analysis complete" and thinks the model just scored poorly.

Expected behavior

The caller should detect that parsing failed and show a warning:

⚠ Analysis failed: could not parse LLM response (truncated JSON)

And ideally offer to retry, since this is often a transient issue (the LLM hit its output token limit).

Root cause

  • parseAnalysisResponse() (analyzer.ts:336) catches the JSON parse error and returns a valid-shaped but empty result — there's no way for the caller to distinguish "analysis succeeded with low scores" from "analysis parsing failed entirely"
  • run-flow.ts:555 calls .succeed() unconditionally after analyzeRun() resolves

Suggested fix

Either:

  1. Add a parseFailed flag to AnalysisResult that callers can check
  2. Have parseAnalysisResponse throw a typed error that analyzeRun can wrap
  3. Return null on parse failure and let callers handle it

Option 1 is the least disruptive.

Observed with Gemini 3 Pro on substring-bypass challenge — the model's analysis response was truncated at 3526 bytes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions