Stabilize Tailnet Evo X2 draft quality and runtime metrics

[terisuke]

Problem

Tailnet Evo X2 is the correct primary runtime path, but generation quality and latency vary enough that single-case validation is no longer sufficient. The product now supports multiple publishing targets, so runtime quality must be measured across the media matrix rather than only a Terisuke/note-style scenario.

Evidence from 2026-05-02:

• Endpoint: http://evo-x2:11434/v1
• Transport: Tailscale VPN/MagicDNS, no SSH tunnel
• Model set: gemma4:31b draft, gemma4:latest style, gemma4:e2b brief
• Elapsed: 1396.80s
• Result: failed quality gates
• Score: 82.0
• Draft length: 2653 runes, below the 2800 scenario gate
• First-person score: 49, below the 60 strict threshold

Evidence from 2026-05-03:

• cmd/scenario/media_matrix now defines varied cases for note, Cor blog, Zenn, Qiita, and homepage output.
• Live source acquisition succeeded for note:cor_instrument, zenn:cloudia, qiita:Cloudia_Cor_Inc, rss:https://cor-jp.com/rss.xml, and github:Cor-Incorporated/corsweb2024/src/content/blog/ja.
• The full live LLM run should use this matrix but should not be forced into every PR because Evo X2 runs are long and can occupy the primary runtime.

This is separate from Issue #36, which is about workstation-local llama.cpp fallback quality. This issue is for the primary Tailnet Evo X2 path.

Scope

• Make Tailnet Evo X2 scenarios repeatable enough to distinguish model variance from prompt/regression bugs.
• Add scenario output that records endpoint, model, elapsed time, attempts, rune count, score, failed metrics, verification result, and whether the run was primary or fallback.
• Tune draft generation for primary Evo X2 so it reliably hits length, first-person, and format-specific requirements.
• Use cmd/scenario/media_matrix as the canonical cross-media input set.
• Track the live runner implementation under the child issue "Add live LLM media-matrix runner and aggregate evaluator".

Media matrix cases

• terisuke_note_essay — note, reflective essay, note:cor_instrument
• cor_blog_technical_report — company blog, technical report, Cor GitHub Markdown
• cor_blog_vision_sharing — company blog, vision sharing, Cor GitHub Markdown
• cloudia_zenn_tutorial — Zenn tutorial, zenn:cloudia
• cloudia_qiita_how_to — Qiita how-to, qiita:Cloudia_Cor_Inc
• cor_homepage_section — homepage section, format sanity check

The final user-facing publishing comparison is note / Qiita / Zenn / company blog. Homepage remains useful as a shorter format-control case.

Acceptance criteria

• Three consecutive Tailnet Evo X2 scenario runs produce at least two passes under the strict gates for the selected phase slice.
• Each run records endpoint, model, elapsed time, score, rune count, failed metrics, verification result, and attempt count.
• Media-matrix live runs produce comparable rows for note, Qiita, Zenn, and company blog.
• The draft service does not silently return a failed draft as successful.
• The UI can show failed evaluation details without discarding the generated draft.
• Failures are grouped by source selector, persona, output format, target length, verifier result, and runtime path.

Related

• [Phase A2] Stream LLM responses via SSE for follow-up and draft generation #18: streaming/cancellation makes long primary-runtime runs usable.
• [Phase C1] Replace JSON store with SQLite-backed schema (extends #14) #26: persistence should land before the full expensive media run if possible.
• ローカル llama.cpp fallback の下書き品質を合格基準まで上げる #36: local llama.cpp fallback quality; separate from this issue.

[terisuke]

Follow-up from #18 / PR #43 Evo X2 validation:

The SSE code path is working and draft quality/time improved versus the previous 1396.80s failed run, but first chunk latency remains high and should stay in this runtime-quality issue.

Measurements on 2026-05-02 against http://evo-x2:11434/v1, gemma4:31b:

• Sample 1 full workflow: score 85.4, runes 2900, elapsed 352.96s, first_chunk_ms 153819, chunks 1701.
• Sample 2 alternate draft scenario: score 82.4, runes 2699, elapsed 339.55s, first_chunk_ms 170402, chunks 1426.
• Average: elapsed 346.26s, score 83.9, first chunk 162.11s.

Recommended next #40 focus: reduce first-token latency through model warmup, queue visibility, model choice, or server runtime tuning. #18 is closed because UX streaming/cancellation/partial-retention are now implemented and merged.

[terisuke]

PR #56 added cmd/scenario/media_matrix and docs/validation/media-matrix-integrated-evaluation-2026-05-03.md as the concrete cross-media input matrix for the next Tailnet Evo X2 quality/runtime pass. Use it to vary medium/style/length across six cases: Terisuke note essay, Cor technical report, Cor vision sharing, Cloudia Zenn tutorial, Cloudia Qiita how-to, and homepage section. The source phase on 2026-05-03 confirmed usable inputs: note 20 articles avg 5,177 chars; Zenn 7 avg 4,139; Qiita 12 avg 3,274; Cor RSS 10 avg 69; Cor GitHub Markdown 10 avg 4,218. Full live LLM scoring remains in this issue because it occupies the primary Evo X2 runtime and should record elapsed seconds, score, verification result, and rune count per case.

[terisuke]

PR #60 is merged into develop and closes #57. The live runner is now available via cmd/scenario/live_media_matrix and make scenario-media-matrix-live. Recommended next step for this issue: run one bounded case first, for example LIVE_MEDIA_MATRIX_CASES=cloudia_zenn_tutorial RUN_LIVE_MEDIA_MATRIX=1 SCENARIO_OUTPUT_DIR=tmp/media_matrix LIVE_MEDIA_MATRIX_OUTPUT_DIR=tmp/media_matrix/live go run ./cmd/scenario/live_media_matrix, then attach aggregate.json / aggregate.md and compare elapsed seconds, score, runes, and verification before running the full Note/Qiita/Zenn/Cor blog matrix.

[terisuke]

2026-05-03 live media-matrix result

Command:

make scenario-media-matrix-live

Runtime:

• Primary endpoint: http://evo-x2.tailb30e58.ts.net/v1
• Draft model: gemma4:31b
• Verification model: gemma4:latest
• Transport: Tailscale VPN / OpenAI-compatible API
• Result: failed overall, because only 1 of 6 cases passed strict gates
• Aggregate report: tmp/media_matrix/live/aggregate.md
• Aggregate JSON: tmp/media_matrix/live/aggregate.json

Case	Medium	Status	Seconds	Score	Runes	Verification	Failure
terisuke_note_essay	note	passed	260.96	90.7	2383	passed	-
cor_blog_technical_report	Cor blog	failed	0.00	0.0	0	not reached	Draft contained preamble before the article
cor_blog_vision_sharing	Cor blog	failed	0.00	0.0	0	not reached	Draft contained preamble before the article
cloudia_zenn_tutorial	Zenn	failed	0.00	0.0	0	not reached	Used Qiita :::note where Zenn requires :::message
cloudia_qiita_how_to	Qiita	failed	0.00	0.0	0	not reached	Used Zenn-specific notation in Qiita output
cor_homepage_section	homepage	failed	360.30	73.7	796	passed	Style score below 80.0 minimum

Notes:

• Tailnet preflight succeeded only after running the scenario command outside the sandboxed network path. Direct curl -s and tailscale checks worked, but the script's sandboxed curl -fsS could not resolve evo-x2.tailb30e58.ts.net.
• The current strict validators are useful: they caught cross-format Zenn/Qiita notation leakage and article preamble leakage.
• The runner currently records 0.00 seconds and no draft path for early unusable-draft failures, so the next fix should preserve failed drafts/metrics for diagnosis.
• This does not satisfy the Stabilize Tailnet Evo X2 draft quality and runtime metrics #40 acceptance criteria yet. Next work should focus on prompt/validator recovery for:
  • Cor blog: suppress preamble and preserve Markdown body/frontmatter expectations.
  • Zenn/Qiita: enforce mutually exclusive container notation.
  • Homepage: tune short-form structure so heading/style score can pass without forcing long-article behavior.