[Phase A2] Stream LLM responses via SSE for follow-up and draft generation

[terisuke]

Tracked under ADR 0002 — Phase A. Detail: implementation plan §A2.

Problem

Every LLM call blocks the UI behind a full-screen spinner (static/js/script.js:407). A 3000-character draft on gemma4:31b can take 60–180 seconds; the user has no signal that progress is happening. Comparable products (Claude, ChatGPT, Gemini) stream tokens, which dramatically improves perceived responsiveness.

Scope

• Add SSE support to internal/infrastructure/llamacpp/client.go (OpenAI-compatible stream: true).
• Wire streaming into:
  • application/brief.InterviewService.GenerateFollowUp — chunks append to the live deep-dive bubble.
  • application/draft.GenerateDraftService.Generate — chunks append to the right-rail draft preview.
• application/authorstyle.AnalyzeAuthorStyleService stays non-streaming (single short call, no UX win).
• Frontend swaps the global spinner for token-by-token append.
• New endpoints (or Accept: text/event-stream overload of existing endpoints): clearly documented in the implementation plan.

Acceptance criteria

• A 3000-character draft visibly streams; first token < 3s on warm gemma4:31b.
• Network tab shows text/event-stream content type with incremental chunks.
• Cancelling mid-stream stops the LLM call (server-side context cancellation propagates to llama.cpp).
• Non-streaming path remains for tests; goldenfile tests use concatenated chunks as the canonical comparison.
• No regression in existing go test ./....

Out of scope

• Persona/format awareness (Phase B).
• Retry-on-disconnect resilience (note as follow-up if needed).

Files likely touched

• internal/infrastructure/llamacpp/client.go
• internal/application/brief/service.go
• internal/application/draft/service.go
• internal/handlers/workflow.go
• static/js/script.js

[terisuke]

2026-05-02 planning update: this is now the immediate next implementation target.

Additional scope from Tailnet Evo X2 validation:

• Default runtime is the Tailnet OpenAI-compatible API, not SSH: http://evo-x2:11434/v1.
• Add SSE event types for status, heartbeat, token, done, and error.
• Emit status immediately with endpoint/model/phase/started_at so the UI is responsive before the first model token.
• Emit heartbeat at least every 10 seconds during long Evo X2 runs.
• Support cancellation from browser disconnect and an explicit Cancel control; cancellation must propagate to the upstream request context.
• Preserve partial draft output and surface evaluation failures instead of dropping work when validation fails late.
• Record endpoint, model, elapsed time, draft length, and score in scenario/validation output.

Reason for priority: Tailnet Evo X2 full workflow reached the correct endpoint but took 1396.80s and missed quality gates. Spinner-only UX is no longer acceptable.

[terisuke]

Implementation update from codex/issue-18-sse-streaming:

Implemented code paths:

• OpenAI-compatible streaming in llamacpp.Client.GenerateStream with non-streaming Generate retained.
• Streaming application paths for brief follow-up generation and draft generation.
• Existing POST endpoints now support Accept: text/event-stream for /api/brief-sessions/{id}/answers and /api/drafts.
• SSE events: status, chunk, heartbeat, result, done, error.
• Browser Cancel buttons abort the fetch request; request context propagates upstream.
• Draft UI preserves partial Markdown while streaming and on cancellation.
• Scenario output now records elapsed_seconds, streaming, first_chunk_ms, chunks, base URL, model, score, and runes.

Tailnet Evo X2 OpenAI-compatible API validation on 2026-05-02:

Sample 1, full workflow, original scenario:

• base_url: http://evo-x2:11434/v1
• model: gemma4:31b
• passed: true
• score: 85.4
• runes: 2900
• elapsed_seconds: 352.96
• streaming: true
• first_chunk_ms: 153819
• chunks: 1701

Sample 2, draft-only alternate Tailnet-primary scenario:

• base_url: http://evo-x2:11434/v1
• model: gemma4:31b
• passed: true
• score: 82.4
• runes: 2699
• elapsed_seconds: 339.55
• streaming: true
• first_chunk_ms: 170402
• chunks: 1426

Comparison against previous Tailnet run:

• Previous: 1396.80s and missed quality gates.
• Current two-sample average: elapsed 346.26s, score 83.9, first chunk 162.11s, runes about 2800.

Conclusion: #18 code path is working and quality/time are materially better than the prior run. First chunk is still far above the warm 3s target, so Evo X2 model warmup/queue/runtime tuning should stay tracked in #40 rather than blocking this SSE UX PR.

[terisuke]

PR #43 has been merged into develop at 0566e28. Closing this issue as completed. Remaining Evo X2 first-chunk latency and runtime tuning are tracked separately in #40.