Bug
When the LLM analyzer returns truncated or malformed JSON (e.g. SyntaxError: Unterminated string in JSON at position 3526), parseAnalysisResponse() catches the error, logs it to stderr, and returns a zeroed-out AnalysisResult with narrative.summary: 'Analysis parsing failed'.
The caller (run-flow.ts:555) then unconditionally prints spinnerAnalysis.succeed('Analysis complete') and saves the empty result — giving the user no indication that the analysis actually failed.
What happens
ℹ Running analysis...
Failed to parse analysis response: SyntaxError: Unterminated string in JSON at position 3526 ← stderr, easily missed
✔ Analysis complete ← misleading
All scores show 0, all fields empty. The user sees "Analysis complete" and thinks the model just scored poorly.
Expected behavior
The caller should detect that parsing failed and show a warning:
⚠ Analysis failed: could not parse LLM response (truncated JSON)
And ideally offer to retry, since this is often a transient issue (the LLM hit its output token limit).
Root cause
parseAnalysisResponse() (analyzer.ts:336) catches the JSON parse error and returns a valid-shaped but empty result — there's no way for the caller to distinguish "analysis succeeded with low scores" from "analysis parsing failed entirely"
run-flow.ts:555 calls .succeed() unconditionally after analyzeRun() resolves
Suggested fix
Either:
- Add a
parseFailed flag to AnalysisResult that callers can check
- Have
parseAnalysisResponse throw a typed error that analyzeRun can wrap
- Return
null on parse failure and let callers handle it
Option 1 is the least disruptive.
Observed with Gemini 3 Pro on substring-bypass challenge — the model's analysis response was truncated at 3526 bytes.
Bug
When the LLM analyzer returns truncated or malformed JSON (e.g.
SyntaxError: Unterminated string in JSON at position 3526),parseAnalysisResponse()catches the error, logs it to stderr, and returns a zeroed-outAnalysisResultwithnarrative.summary: 'Analysis parsing failed'.The caller (
run-flow.ts:555) then unconditionally printsspinnerAnalysis.succeed('Analysis complete')and saves the empty result — giving the user no indication that the analysis actually failed.What happens
All scores show 0, all fields empty. The user sees "Analysis complete" and thinks the model just scored poorly.
Expected behavior
The caller should detect that parsing failed and show a warning:
And ideally offer to retry, since this is often a transient issue (the LLM hit its output token limit).
Root cause
parseAnalysisResponse()(analyzer.ts:336) catches the JSON parse error and returns a valid-shaped but empty result — there's no way for the caller to distinguish "analysis succeeded with low scores" from "analysis parsing failed entirely"run-flow.ts:555calls.succeed()unconditionally afteranalyzeRun()resolvesSuggested fix
Either:
parseFailedflag toAnalysisResultthat callers can checkparseAnalysisResponsethrow a typed error thatanalyzeRuncan wrapnullon parse failure and let callers handle itOption 1 is the least disruptive.
Observed with Gemini 3 Pro on substring-bypass challenge — the model's analysis response was truncated at 3526 bytes.