tts: the unhealthy-verdict warn re-fires every probe cycle — ~8,600 stack traces/day on a piper-only box

[genwave-radio]

🔊 Field observation (Pi 5, v2.9.0, --pinned --piper-only)

warn: GenWave.Tts.DependencyHealthProber[0]
      kokoro health probe failed (Name or service not known (kokoro:8880)) — 29 consecutive
      failures, cached verdict is now unhealthy
      System.Net.Http.HttpRequestException: Name or service not known (kokoro:8880)
       ---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known
         [full stack trace]

Plus two info entries per probe from the KokoroHealthProbe typed client (LogicalHandler + ClientHandler), each carrying its own stack trace.

🐛 Two defects

1. The warn re-announces a transition that already happened.

DependencyHealthProber.RecordFailure (DependencyHealthProber.cs:129) branches on verdict.Healthy. The Debug branch is careful and correct — "N of M consecutive failures needed before the verdict flips; cached verdict still healthy". The Warn branch is not: it fires on every subsequent failure, with the wording "cached verdict is now unhealthy".

The verdict flipped at failure 2 (F70.2 default threshold). Failure 29 changed nothing. The message reads as an incident every 30s when there was exactly one incident.

2. On piper-only the condition is permanent by construction, so it never stops.

compose.piper-only.yaml profile-disables kokoro and maps all six SegmentKinds to piper. Its header explains why Tts__Endpoint must keep pointing at the absent host (KokoroTtsSynthesizer speaks kokoro-fastapi's wire protocol, piper's server does not, and TtsOptions.Endpoint is [Required]), and describes the probe flipping unhealthy as the intended mechanism. That design is sound — but it means the probe fails forever, and nothing anticipated the log volume:

Per 30s probe	Per day
1 × warn + stack trace	2,880
2 × info HttpClient + stack traces	5,760
	~8,640 stack traces/day

For a condition that will never change and carries zero information after the first entry.

📐 It violates the method's own stated principle

From RecordFailure's doc comment:

"This is what keeps a busy-but-alive dependency off the warning stream entirely: gh-#125's Kokoro blocks its own event loop for the length of a render, so it misses isolated probes forever, and paging on that is noise."

The reasoning applies with more force here. gh-#125's kokoro was busy-but-alive and might recover. This kokoro is deliberately absent and definitionally cannot. Same class as gh-#251's icecast warn spam, and on a Loki-shipping box it is real ingest volume.

🔧 Proposed shape

• Warn on the flip only. Steady-state repeats with an unchanged reason drop to Debug (which per current fleet config does not reach Loki), and drop the exception argument — the stack trace is identical every cycle and the reason string already carries the cause. Fleet volume goes from 2,880/day to ~1 per outage.
• Consider skipping the probe entirely when Tts:EngineByKind routes no SegmentKind to the dependency. On piper-only that is all six, so the probe is answering a question nothing asks. This is knowable from config at registration time.
• Quiet the typed client's info logging for probe clients — a health probe's request/response pair is not application-interesting at info, and it is 2 of the 3 stack traces.

Care needed on the first bullet: the flip → Warn → recover → flip cycle must still produce a Warn each time a real dependency genuinely drops, so the suppression has to key on "verdict unchanged since last cycle", not on "verdict is unhealthy".

📋 Notes

• The cadence itself is fine — 29 failures × 30s ≈ 14.5 min matched the box's elapsed uptime exactly. This is purely about volume and wording.
• Unrelated to gh-library: enrichment saturates every core on a 4-core box, and an appliance box has no lever to lower it #334 (enrichment concurrency) beyond both surfacing on the same Pi session.