Skip to content

fix(tts): warn on the probe's failing edge, not on every cycle after it (gh-#338) - #342

Merged
genwave-radio merged 2 commits into
mainfrom
fix/gh-338-probe-warning-edge-triggered
Aug 2, 2026
Merged

fix(tts): warn on the probe's failing edge, not on every cycle after it (gh-#338)#342
genwave-radio merged 2 commits into
mainfrom
fix/gh-338-probe-warning-edge-triggered

Conversation

@genwave-radio

Copy link
Copy Markdown
Collaborator

Closes #338.

🔊 The field symptom

On the gh-#213 Pi 5, kokoro is profile-disabled by the piper-only overlay, so its probe can never succeed. The api logged this every 30 s, forever:

warn: kokoro health probe failed (Name or service not known (kokoro:8880))
      — 29 consecutive failures, cached verdict is now unhealthy
      [full stack trace]

The verdict flipped at failure 2. Failure 29 changed nothing, and the wording claimed a transition each time.

Per 30 s probe Per day
1 × warn + stack trace 2,880
2 × info from the typed client + stack traces 5,760
~8,640 stack traces/day

For a condition reported once and permanent by construction.

📐 It was already the stated contract

RecordFailure's own doc comment:

"Only the failure that actually flips the verdict (and therefore starts diverting renders to the fallback engine) is a warning."

That was always the intent. The code branched on verdict.Healthy — which is false for the flip and every cycle after it — so it never honoured it. It does now.

🔧 The change

  • Warn once, with the exception, on the probe that flips the verdict.
  • Repeats while the verdict is unchanged → Debug, without the exception. The trace is identical every cycle; the reason string carries the cause. Debug doesn't reach the fleet sink, so 2,880/day becomes ~1 per outage.
  • Recovery → one Information line on the way back up. The up-edge was previously logged nowhere at all — an operator saw "kokoro unhealthy" and never learned it came back. A down-edge without an up-edge is only half a transition model, so this is included deliberately rather than left for later.
  • appsettingsKokoroHealthProbe / PiperHealthProbe / OllamaHealthProbe typed clients drop to Warning. A health probe's own request/response pair isn't application-interesting, and it was 2 of the 3 stack traces per cycle.

🪤 The trap, and why the edge is tracked rather than derived

ConsecutiveFailureCount == UnhealthyThreshold looks like a free edge test. It's wrong: the threshold is re-read live every cycle (gh-#125), so an operator lowering it mid-outage flips the verdict at a count already past the new threshold — and the arithmetic version logs that flip at Debug and loses it.

So the edge is tracked explicitly, in a per-dependency ConcurrentDictionary of the reason already warned about. TryAdd succeeds exactly once per outage and is atomic. ScenarioTheThresholdIsRetunedWhileTheDependencyIsDown pins exactly this case.

Equally, suppression keys on "have we warned since the verdict last changed", not "is the verdict unhealthy" — otherwise a dependency that drops, recovers and drops again goes silent on the second drop. That has its own spec too.

🚫 Deliberately not changed

  • A reason that changes mid-outage stays at Debug. Re-warning on it would reopen this hole for any dependency whose failure mode oscillates.
  • Sub-threshold Debug keeps its stack trace. It's the first observation of a real problem and is bounded at threshold-1 per outage (default: one). Only the unbounded stream is stripped.
  • The "not configured" path still logs nothing (empty Llm:Endpoint, F34.2 — a declaration, not a fault), and now also can't produce a phantom "recovered" line for an outage nobody was told about.

✅ Verification

Gh338_ProbeWarningIsEdgeTriggered.cs — 10 specs driving RunCycleAsync one cycle at a time, so every fact is about which cycle logged what. No timers, no clock, no waiting. Uses a small in-file recording logger rather than adding a package.

  • 4 of the 10 fail against the old prober.
  • dotnet build GenWave.sln — 0 warnings, 0 errors.
  • dotnet test GenWave.sln — all five green (140 / 199 / 414 / 1286 / 632, 0 failed).

📋 Note

This needs a release to reach the Pi — both changed files ship inside the api image, unlike the compose/launch.sh fixes earlier today.

…it (gh-#338)

On the gh-#213 Pi 5, kokoro is profile-disabled by the piper-only overlay, so
its probe can never succeed. The api logged this every 30s, forever:

  warn: kokoro health probe failed (Name or service not known (kokoro:8880))
        -- 29 consecutive failures, cached verdict is now unhealthy
        [full stack trace]

The verdict flipped at failure 2. Failure 29 changed nothing, and the wording
claimed a transition each time. At the F70.2 default cadence that is ~2,880
warnings a day carrying an identical stack trace, plus 5,760 info lines from the
typed client -- ~8,600 stack traces a day for a condition reported once and
permanent by construction.

RecordFailure's own doc comment already stated the intended contract: "Only the
failure that actually flips the verdict is a warning." The code did not honour
it. It does now.

  * Warn once, with the exception, on the probe that flips the verdict.
  * Repeats while the verdict is unchanged: Debug, without the exception. The
    trace is identical every cycle and the reason string carries the cause.
  * Recovery: one Information line on the way back up. Previously the up-edge
    was logged nowhere at all, so an operator saw "kokoro unhealthy" and never
    learned it came back -- the down-edge alone is only half a transition model.
  * appsettings: the KokoroHealthProbe/PiperHealthProbe/OllamaHealthProbe typed
    clients drop to Warning. A health probe's own request/response pair is not
    application-interesting, and it was 2 of the 3 stack traces per cycle.

The edge is TRACKED (a per-dependency ConcurrentDictionary of the reason already
warned about), not derived from ConsecutiveFailureCount == UnhealthyThreshold.
The threshold is re-read live every cycle (gh-#125), so an operator lowering it
mid-outage flips the verdict at a count already past the new threshold, and the
arithmetic version would log that flip at Debug and lose it. There is a spec for
exactly that.

Deliberately NOT re-warned: a reason that changes mid-outage. Re-warning on it
would reopen this hole for any dependency whose failure mode oscillates.

Sub-threshold Debug keeps its stack trace -- it is the first observation of a
real problem and is bounded at threshold-1 per outage. Only the unbounded stream
is stripped.

10 specs; 4 fail against the old prober.
@genwave-radio
genwave-radio merged commit 2931b7b into main Aug 2, 2026
11 checks passed
@genwave-radio
genwave-radio deleted the fix/gh-338-probe-warning-edge-triggered branch August 2, 2026 23:40
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 2, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tts: the unhealthy-verdict warn re-fires every probe cycle — ~8,600 stack traces/day on a piper-only box

1 participant