Recovery "Not scored": a truncated-substrate re-derive NULLs the readiness baseline, then the z-cap abstains with no diagnostic #305

Summary

On a device running 0.9.28, Recovery reads "Not scored" while sleep derives normally every night. The cause is a chain that begins with a truncated-substrate re-derive shadowing a complete historical day and replacing its baseline scalars with NULL, and ends with a readiness abstention that is invisible: it emits no diagnostic, and the card shows a reason belonging to a different driver.

This writeup has been through three rounds of independent adversarial review against the code; ten findings were accepted and several of my initial claims were wrong and have been removed. What follows is stated at the confidence the evidence actually supports, with overclaims marked where they remain.

Environment

Evidence is the on-device openstrap.db plus adb logcat. No crash, no exception, no failed derive, and no sync failure — sync_ledger shows status: complete, history_complete_at set, backlog_remains: false, dropped_this_burst: 0. The data arrived; the derive discarded it.

Symptom

Home, 2026-08-28 16:25:

• SLEEP 7h 27m ✓ — from the night ending 2026-08-27 (tst_min 447)
• RECOVERY Not scored, with the composite's prose beneath it: "…Refused: temp: no settled fraction measured for this night — an ungated nightly mean cannot tell skin from a warming strap."
• STRAIN No strain — investigated and CLEARED, see "Not a defect" below.

The Recovery copy is misleading: it names the temp driver refusal, which is a renormalisation detail and not why the score is absent. The actual reason is never surfaced anywhere in the payload or the UI.

Day labels are the local calendar day in which the sleep ends (lib/data/day_label.dart; substrate.dart:1143 attributes on the offset landing inside the day). Label 08-27 = the night 00:06 → 07:51 CEST on the 27th.

Link 1 — a truncated-substrate row shadowed a complete one and NULLed its baseline scalars

day_result for 2026-08-23, both versions still present:

Precisely: a truncated-substrate v76 row shadowed the complete v74 row and replaced its unversioned metric_series baseline scalars with NULL. Three distinctions matter and I had them wrong initially:

• The v74 day_result row was not overwritten — it is still on disk as a sibling row, merely shadowed, because readers take the highest algo_version <= _servedAlgoCeiling (lib/data/db.dart:6110).
• metric_series is unversioned, so its values were genuinely replaced (putDayResult is ConflictAlgorithm.replace on both tables). That is the destructive half.
• "Partial" here means truncated input, not the engine's partial row state — the v76 row is persisted with partial = 0.

The sleep window was 2026-08-22 23:27 → 2026-08-23 08:52 CEST, entirely behind the rolling prune cutoff (dataNowSec - rawRetentionDays * 86400 ≈ 2026-08-23 17:05 CEST at derive time). So every RR beat under it was gone while ~7 h of post-cutoff HR survived.

Sleep nonetheless came through unchanged (spt_sec 33879, identical onset_ms/offset_ms) because a same-version richer-candidate guard retained the earlier staged sleep candidate. Persistence alone does not guarantee this — 08-22 is the counterexample: its v76 candidate is flags:["NO_SLEEP_DETECTED"], confidence 0.0, window: null.

Scope note: the v76 bundle still carries daytime HRV and HR curves, so this is not "derives nothing HR-derived". The loss is specifically the sleep-window headline metrics — rhr, rmssd, ln_rmssd, resp_rate, skin_temp_adc.

Link 2 — the "never write nothing over something" guard is all-or-nothing

lib/compute/derivation_engine.dart:3636-3661:

final producedNothing = daySub.isEmpty &&
    sleepSub.isEmpty &&
    (scMap == null || !scMap.values.any((v) => v != null));
if (producedNothing) { /* keep existing row */ }

The v76 08-23 bundle is not nothing — tst_min 535, deep_min 92, light_min 433, efficiency 94.7, steps 5590, worn_min 419 are all non-null — so producedNothing == false, the guard is skipped, and the null HR scalars are written.

The guard's own comment scopes it to a day with "a good day_result and NO raw". It has no defence against a day at the retention edge with partial raw, which is the commoner case because the cutoff is a rolling timestamp, not a day boundary.

Link 3 — the readiness baseline collapsed from six nights to three

_BaselineHistoryCache rebuilds from metric_series (derivation_engine.dart:1781) using days strictly before the day being scored. Immediately before scoring 08-27 the unclobbered history would have held six usable lnRMSSD/RHR nights — 08-19, 08-22, 08-23, 08-24, 08-25, 08-26. NULLing 08-22 → 08-24 reduced it to three: 08-19, 08-25, 08-26.

date        rhr      rmssd     ln_rmssd     had at v74
2026-08-19  69.02    91.72     4.5188       ✓ (survives)
2026-08-22  —        —         —            69.77 / (lost)
2026-08-23  —        —         —            49.24 / 110.87 (lost)
2026-08-24  —        —         —            51.84 / 103.18 (lost)
2026-08-25  53.97    90.98     4.5106       ✓
2026-08-26  51.90    100.98    4.6150       ✓

Confirmed in the 08-27 payload: baselines.hrv.n_valid = 3, resting_hr.n_valid = 3, both status: "calibrating".

Link 4 — the composite scored on that triplet, and the headline gate discarded it

At minBaseline = 3 (the shipped pin) the composite fires. The surviving lnRMSSD triplet [4.5188, 4.5106, 4.6150] has an unusually small non-zero MAD ≈ 0.008, and robustZ only returns null on an exactly zero MAD — so it returns a very large z. From the stored 08-27 payload:

{"value": {"score": 0.002137, "composite_z": -10.753498},
 "inputs_used": ["HRV", "RHR"],
 "drivers": [
   {"label": "HRV", "contribution": -10.380796,
    "detail": "oriented robust-z (median+MAD)=-18.166394"},
   {"label": "RHR", "contribution": -0.372702,
    "detail": "oriented robust-z (median+MAD)=-0.869638"}]}

headlineReadinessScalar (onehz_pipeline.dart:94-104) then drops it: if (r.compositeZ.abs() > kReadinessZCap) return null; with kReadinessZCap = 5.0.

Stated at the right strength: minBaseline = 3 permits unstable baselines; it does not make every three-night baseline degenerate, and the cap does not reject every z obtainable from three points. What is proven is that this device's triplet produced −18.17 and crossed the cap.

And the abstention is silent. readiness_absent_diag is only populated when !composite.present (onehz_pipeline.dart:606). Here the composite is present and the headline gate drops it — so the diag is null on 08-27 and local_repository_impl.dart:352 falls through to the composite's prose, which describes the temp driver. That is why the card blames skin temperature.

One day earlier the honest path did fire (08-26 readiness_absent_diag): "note": "need_baseline:have=2,need=3 …" → "Need 1 more night". So on the shipped build, crossing have=2 → have=3 makes the UX strictly worse: an informative wait message becomes an unexplained blank with a misattributed cause.

Not established: permanence. metric_series keeps filling nightly, readiness is not fed back into its own baseline, and neither robustZ nor the cap latches — a wider MAD on later nights clears it unaided. Proven: one silent abstention on 08-27, plus a plausible recurring erosion loop while retention-edge clobbering keeps trimming the baseline.

What is still broken on main

Under minBaseline = 14, defects 1–2 produce an honest-but-possibly-unending "calibrating" state rather than a silent blank. Whether 14 nights is reachable while retention-edge clobbering continues is exactly the open risk.

Attribution — which writer did it

The destructive current-version re-derive over truncated retained substrate is proven. Attributing it specifically to a baseline-dirty rescanRecent is not: after a version bump, ordinary run() already treats every raw-backed day without a finalized row at the exact new version as pending and derives it directly (derivation_engine.dart:3054-3080), and rescanRecent only launches after the heavy run. Because same-version writes use REPLACE, the final DB cannot distinguish them. rescanRecent is another reachable writer, not the proven trigger.

Timeline — retention-boundary timing, not a race

The determining condition is whether the sleep raw had already been deleted before the new version became active — not which pass happens to win. Once a current-version day is pending, the raw-first barrier holds its remaining substrate through derivation.

Surviving substrate: decoded_onehz and decoded_rr both start 2026-08-25 — nothing older.

Not a defect (investigated and cleared)

Strain "No strain" on 08-28 is correct behaviour. day_result 2026-08-28 wear.segments:

{"on": false, "start": 1787868000, "end": 1787910962, "len_min": 716}
{"on": true,  "start": 1787910962, "end": 1787927018, "len_min": 268}

The band was off the wrist for 716 minutes overnight, sleep_charging.present = false. No night to score → no nocturnal RHR → need_input:name=resting_hr. TRIMP deliberately accepts only that day's nocturnal RHR or a manually entered resting HR (onehz_pipeline.dart:658-670, derivation_engine.dart:5754-5758) — an intended contract. Recovery (08-27) and Strain (08-28) are independent absences on different dates; getToday() composes the latest overnight bundle with today's activity bundle.

Possible separate bug, noted not diagnosed: compute_freshness reports overnight_state: "ready", overnight_day: "2026-08-28", showing_prior_overnight: false because a NO_SLEEP_DETECTED row satisfies its hasSleep || noSleep test (db.dart:8147-8156), while _latestBundle() independently skips that row and returns 08-27 (local_repository_impl.dart:62-82). Home therefore does display the prior complete night for Recovery while the freshness row says it isn't. The two disagree.

Proposed fixes

1 — A "protected — unrecomputable at this version" row state. The naive versions of this fix are both wrong, and I published both before review caught them:

• Per-scalar preservation ("never replace non-null with null from a thinner substrate") would resurrect scalars a newer algo deliberately retired — the engine legitimately writes NULL to remove previously fabricated strain/steps — and would stamp old-version numbers as current.
• Simply declining the re-derive wedges the pruner. LocalDb.dayResultIds(kAlgoVersion) is version-exact (db.dart:6277-6285) and feeds rawPruneCutoffSec via _pruneOldDecoded (:4896), so a day with no row at the current version counts as pending, drags the prune barrier back to its own start for up to _maxRawHoldDays (14), and is retried every pass forever.

The state must therefore: not shadow the served older result; satisfy scheduling and pruning bookkeeping so the barrier advances; and reopen if missing coverage later arrives or a forced replay restores it. Tests: repeated passes, pruning, version bump, late backfill.

2 — Decide recomputability from retained coverage, not from a row-level heuristic. A sleep-window boundary check is insufficient: 08-22's v76 candidate has no window at all, candidate lookup is current-version-only, and boundary checks cannot see holes inside a nominally covered interval. coverage also cannot rank rows — it holds day-wide aggregate counts and a derived sleep duration, with no timestamps, gap topology, or per-signal interval coverage, so hr_samples / rr_beats / sleep duration / non-null-scalar count each select a different row. Use the highest prior complete, non-skipped row as provenance only, and evaluate temporal, per-signal retained coverage across the whole requested derivation range.

3 — Give the z-cap abstention its own reason. Do not map it to need_baseline: — on main an input needs ≥14 baseline points before the composite is present at all, so "Need N more nights" would state a false cause, which is exactly what this codebase's absent-note discipline exists to prevent. Emit a dedicated unstable-normalisation / value-outside-supported-range reason carrying the composite z and per-input baseline counts, and populate readiness_absent_diag on that path. Regression should use a constructed ≥14-night case.

Raw no longer recomputable

The 1 Hz substrate behind 08-20 → 08-24 is gone, so no re-derive can restore those scalars. But the v74 derived values are still on disk — nothing was deleted, only shadowed.

Any repair must preserve v74 provenance and keep day_result and metric_series consistent. Do not promote a whole row by an undefined richness score — that is the same ill-defined criterion rejected above. If a repair is attempted it should be a version-preserving, cohort-validated migration (recovery / sleep / activity ranked separately), designed alongside the protected-state mechanism in fix 1, not bolted on.

Review provenance: three adversarial rounds via Codex CLI (xhigh, read-only sandbox) against the actual code and the pulled device DB. Ten findings accepted, none rejected. Claims removed as unsupported: strain being downstream of readiness; all three defects being present on main; the abstention being permanent; a 21-day rescan being the trigger; the per-scalar and decline-only fixes; global richest-row promotion; and a hole-filled-slice inference drawn from two substrate snapshots taken 19 hours apart.