Repository navigation
Description
Summary
Split out from #1008. A WHOOP 4.0 strap log (9.3.1, user pilleuspulcher-blip) shows every night over-counted: coverage=2.1–2.4 / rrIntegrity=crossSecondOverCount, i.e. ~2× more R-R intervals than wall-clock seconds. Crucially this holds on the provenance=measured nights (NOOP's own live capture, days 2026-08-03/04), not just the imported my-whoop spine — so it is a live WHOOP 4.0 capture/persist problem, not an import artifact.
This is the root cause behind the #1008 "HRV doesn't match WHOOP" report: RMSSD/SDNN are built from successive differences, so a duplicated beat train corrupts the variability figure while meanNN / resting HR stay correct.
Why the existing machinery doesn't catch it
srcChannel([oura][hrv]rrIntervalmixes two optical channels, so every beat is stored twice #1071) is Oura-only. Its codes are Oura IBI channels and the read filter just drops the Oura SpO2 duplicate (srcChannel <> 2). WHOOP beats are writtensrcChannel = NULL, so they pass straight through.- The over-count is cross-second and non-exact.
dupBeats(exact(ts,rrMs)repeats) is small;collapsedCov(same-second near-equal collapse) stays ~1.8 — so a same-second de-dup can't fix it, and it isn't a plain double-insert on the(deviceId,ts,rrMs,seq)key. #1072/#823fixed emission ORDER, not the over-count. So RMSSD is currently left un-gated on the bet that a post-fix capture would be clean — this log is that capture, and it isn't.
Diagnosis groundwork (done)
#1117 added densestSecondWindowSample — an always-on, over-count-only raw-row dump around the densest second — so the mechanism is now readable from a strap log:
- near-equal values (
[1200,1199]) → same beat stored twice (de-dupable / taggable), - full + partial (
[1200,600]) → two distinct interval trains (a genuine second stream), @N/src=→ channel tag;src=noneconfirms the WHOOP beats are untagged.
Next: get a fresh log on a build with #1117 to read the shape, then pick the fix.
Candidate fixes (decided by the shape)
- Double-write (live + offload persisting the same window with divergent
seq) → makeseqpath-independent so the duplicate collides on the PK and never lands; idempotent cleanup / read-collapse for legacy rows. - A second decode channel (WHOOP analog of Oura's dual IBI) → extend the [oura][hrv]
rrIntervalmixes two optical channels, so every beat is stored twice #1071srcChanneltag+filter pattern to the WHOOP decoder — additive nullable column, filter at read, byte-parity twin, no re-score. - Cross-second bucketing of one beat → read-time cross-second de-dup with a conservative tolerance (last resort; the only option that can drop a real beat, so it needs validation).
Backstop (independent of the above)
RMSSD is un-gated under crossSecondOverCount while SDNN is already withheld. Now that a measured post-fix capture confirms the over-count persists, add the RMSSD/HRV trust-gate too (withhold, don't fabricate) — but only after measuring how widespread the over-count is, since a blind gate trades a wrong number for a blank across every affected WHOOP 4.0 user.
Not in scope
The HrvWindow / LAST_SWS work in #1008 stays parked until the R-R stream is clean — tuning the window over a contaminated RMSSD is polishing a wrong number.
Activity
ryanbr commentedon Aug 8, 2026
Mechanism characterized from a fresh #1117 rrsample dump (WHOOP 4.0, capture 2026-08-08). The over-count is not a single artifact:
Intra-second R-R distributions (t0 buckets):
So
coverage=1.75decomposes into: a small exact-duplicate component + a dominant two-optical-channel component (every beat reported twice,30–35ms apart) + coarse 1-second timestamps.1.4) but the ~34ms channel pairs survive →collapsedCoverage(30ms)removes the near-dupes (1.75→crossSecondOverCount,beatAccurate≈0.42–0.45.This is structurally the #1071 two-optical-channel artifact — but WHOOP 4.0 R-R carries no
srcChanneltag (that decode is Oura-only), so the two channels can't be separated by a stored field; it'd need heuristic pairing.Why this can't be a code-first fix (blockers, not a to-do):
crossSecondOverCountwould zero HRV/recovery for every 4.0 user (it's the dominant recovery input and every measured night trips it).Proposed path forward (validation-first, not code-first): (1) confirm the ~34ms two-cluster pairing holds across subjects/nights (more #1117 dumps); (2) prototype heuristic channel separation (keep one beat per ~34ms pair) as instrumentation — log the corrected RMSSD beside the incumbent, don't feed recovery; (3) validate the corrected RMSSD tracks a varying ground truth before it can back a shipped metric. Until then RMSSD stays as-is (SDNN already gated via
beatSpreadIsTrustworthy).pilleuspulcher-blip commentedon Aug 8, 2026
@ryanbr
Android 17, Whoop 4, latest test build NOOP Staging — base 9.3.2 · 2026-08-07
After a night with the last test build there is no change in hrv readings (deep sleep window) for me.
Whoop app 29, Noop 49. RHR also differs (Whoop app 50, Noop 43).
noop-hrv-Android-v9.3.2-staging-260808-0929.zip
noop-strap-log-260808-0930.txt
artemc commentedon Aug 8, 2026
I have a small independent, single-participant dataset that may be useful for the validation-first path proposed here: five WHOOP 4.0 nights with raw RR stored by NOOP, official WHOOP daily HRV, and simultaneous Polar H10 ECG/RR recordings.
I ran a read-only integrity check over the primary-sleep sessions and reproduced the diagnostics used by
densestSecondWindowSamplefrom #1117. The archived captures separate into two groups:All rows had
srcChannel = NULL/absent. The densest-second contexts contained both same-second near-equal values and groups of nearly identical values appearing again in an adjacent second. I am intentionally not posting the raw RR values or timestamps.The full classifier is more informative than the raw sample alone. With the current 1.10 ceiling, two development nights classify as
sameSecondOverCount: their collapsed coverage falls to 1.06 and 1.01. The remaining night lands only fractionally on thecrossSecondOverCountside: collapsed coverage is approximately 1.100004 against a 1.10 threshold. I would treat that as a boundary case, not evidence of the same dominant cross-second mechanism as the fresh 1.4 collapsed-coverage sample reported above.These archived captures predate the current instrumentation and fixes, so they do not independently confirm that either historical shape persists post-#1071/#1072. Their main value here is the simultaneous external reference and the contrast between clearly over-counted and near-unit-coverage nights.
The external references also show why RR integrity matters. Across all five nights, rounded group averages were:
The Polar window is not identical to WHOOP's proprietary window or NOOP's detected sleep session, so the H10 number is an independent ECG/RR directional reference rather than exact metric equivalence.
Using a predeclared three-night development/two-night holdout split, rounded MAE was:
n=3)n=2)The lower disagreement in the near-unit-coverage group is consistent with RR over-count being a material contributor. It does not identify a safe correction, and I am not proposing a de-duplication tolerance from one participant. It also illustrates why the raw densest-second shape should be interpreted alongside
coverage,collapsedCov, and the final integrity verdict.One result also helps separate this issue from the staging/window question in #1008. On the two near-unit-coverage held-out nights, NOOP-detected deep-only RMSSD still had MAE of about 12.0 ms against official WHOOP and 10.1 ms against Polar H10, versus about 2 ms for whole-night aggregation. Thus the deep-window failure remained when obvious nightly over-count was absent; cleaning the RR stream is necessary, but it does not by itself validate
DEEP_SLEEPas a WHOOP-comparable default.Limitations: one participant, only two held-out nights, separate metric windows, and archived NOOP captures rather than a current-build reproduction. No participant-specific offset was fitted. No dates, daily pairs, raw RR/ECG rows, exports, logs, or device identifiers are included.
If useful, I can run the same read-only diagnostics on additional synchronized NOOP + Polar H10 nights collected with a current build, using a protocol fixed before examining the results.
ryanbr commentedon Aug 9, 2026
Fresh log (the one this was waiting on) — WHOOP 4.0, fw 41.17.6.0, 9.3.2-staging
A 2026-08-09 capture shows
rrIntegrity=crossSecondOverCounton every one of 30+ measured nights — it's not sporadic, it's the steady state on this strap.The signature is unambiguous. A representative
hrv rrsample:That
0sbucket has 8 R-R intervals in a single 1-secondts(477,500,623,653,654,686,810,849 ms). Eight beats in one second is ~480 bpm — physically impossible; the strap is over-emitting R-R into a second.beatsPerSecsits ~2.0–2.2 across the night (a real ~65 bpm night should be ~1.1).It's inflating raw coverage and the diagnostic already sees it —
coverageruns ~1.68–1.85 whilecollapsedCov(same-second collapse) drops to ~1.28–1.50, i.e. ~20–30% of the "coverage" is duplicate same-second beats:dupBeatsranges 30–828/night;beatAccurate0.39–0.49 — consistently below the 0.50 gate, so respiration is correctly reading no-data (#1127 holding).Useful implications for the fix:
maxInSec/dupBeats/collapsedCovfields are enough to quantify the over-count per night; the RMSSD built on the raw (un-collapsed) beats is contaminated by those zero-/small-difference same-second pairs.-3s[682,715] … 0s[8 beats]shape suggests the over-count clusters in specific seconds rather than spreading evenly — worth confirming whether it's a decode-time duplication (same beat emitted N×) vs genuinely distinct sub-second beats, since the collapse strategy differs.beatAccuratenever clears 0.50 on this strap, so whatever the fix does, it should keep respiration gated here.Full capture + strap-log available on request (personal data scrubbed). This is the multi-night measured evidence #1118 was blocked on.
feat(hrv): persist a per-night R-R over-count flag for the HRV caveat (…
feat(hrv): caption an over-counted night's HRV reading "unverified" (#…
ryanbr commentedon Aug 15, 2026
Interim: honest "unverified" caveat shipped — the real de-dup fix is next
Thanks @pilleuspulcher-blip for the fresh logs and @artemc for the Polar H10 dataset — both are exactly what this needed.
Mechanism is now nailed down (from the #1117
hrv rrsampledumps): the over-count is dominated by the two-optical-channel artifact — every beat reported twice ~30–35 ms apart — plus coarse 1-second timestamps and a small exact-duplicate component.collapsedCoverage(30 ms)removes the internal near-dupes (~1.75 → ~1.4) but the ~34 ms channel pairs survive →crossSecondOverCount,beatAccurate ≈ 0.42. This is structurally the #1071 two-channel artifact, but WHOOP 4.0 R-R carries nosrcChanneltag, so the existing Oura machinery doesn't catch it.What shipped now (honesty, not the fix)
Since the fix needs validation and every measured 4.0 night is affected, the interim step is to stop presenting a contaminated HRV as a clean measurement:
hrv_rr_overcountflag (written only for NOOP's own measured R-R nights).So a 4.0 user who sees NOOP's HRV not matching WHOOP now gets told the reading is a known-unreliable strap value. RMSSD is still shown raw — this is honesty about the number, not a correction of it.
The real fix (next, and validation-gated)
The actual correction is the two-channel de-dup — either extending the #1071
srcChanneltag+read-filter to the WHOOP decoder, or a read-time collapse of the ~34 ms same-second channel pairs. This will not ship on a guess: @artemc's 5-night WHOOP 4.0 + simultaneous Polar H10 ECG set (3 over-count nights at coverage 1.39–1.52, 2 near-unit) is the ground truth, and any de-dup has to track the varying Polar RMSSD across all of them — not match a single night — before it goes in (the #194 lesson).Leaving this open to track that fix.
diag(hrv): shadow-log a deduped R-R stream to validate the over-count…
gdorgian commentedon Aug 16, 2026
WHOOP 5.0/MG: same symptom, different mechanism — clock correlation, not double-write
Following up on my 5.0 numbers in #1331. I got live console output off a cabled device (mirroring the strap log to stderr under an env flag) and it rules out both candidate mechanisms in the issue body for this generation.
The live path contributes essentially no R-R
Across captures,
HR notifylines carryrr=0, occasionallyrr=1orrr=2. Against 37,512–56,053 R-R rows per night, the live 2A37 stream is a rounding error. Every interval that reaches the store comes from the historical offload.That rules out candidate #1 (double-write, live + offload persisting the same window) on a 5.0: there is no meaningful second writer.
It also isn't a second interval train
I initially read the
hrv rrsampledumps the other way — some densest seconds show a quantised250/500/1250cluster that looks exactly like the "full interval beside a partial one" shape. It isn't representative.rrMinMs = 300already excludes those, and the diag line reportsrr=37512/36743, 2% rejected. 769 intervals of ~250 ms is ~192 s across a ~28,000 s night — it cannot account forcoverage=1.81.The excess coverage is made of plausible-valued intervals. So candidate #2 (second decode channel) doesn't fit either.
What the log does show
Clock correlation never lands, and the backfill completes anyway — decoding under the
IDENTITY fallback (no clock correlation at decode)that appears on every offload here.A mis-scaled time axis at decode produces all four observed symptoms from one fault, which no other hypothesis I have does:
coveragerrIntegritycrossSecondOverCountbeatAccurateFractionmaxInSecNote the last one especially:
beatsPerSecaverages 1.63–2.36 againstmeanNN831–916 ms (~1.1 beats/s expected), while individual seconds hold 61–109 beats. That is a time-axis fault, not duplicated values.Why this matters for the fix
If this is right, the two-channel de-dup would not fix a 5.0 — there is no second channel to de-dup, and the beats are not duplicates. They are real beats on a wrong time axis. A de-dup tuned on 4.0 data would either no-op here or discard genuine beats.
It would also mean the 4.0 and 5.0 need separate fixes despite presenting identically through #1127's gate.
I am treating the clock-correlation link as the strongest hypothesis, not established fact — I have not yet traced the offload decode path to confirm the identity mapping is what stamps these rows. That is what I am looking at next, and I will report either way.
Environment
WHOOP 5.0/MG (
hwRev=WG50_r52, FW 50.35.1.0), iOS 27, 10.0.0 sideload, ~14 nights continuous wear,src=noneon every night. Happy to capture anything specific from a live session — I can now stream the strap log off the device while it syncs, so targeted instrumentation is cheap to run if you want a particular value printed at decode.ryanbr commentedon Aug 16, 2026
Traced this against the decoder to confirm the mechanism, and @gdorgian's 5.0/MG read holds up in the code.
The over-count is time-collapse, not literal duplicates. In
HistoricalStreams.swift, every R-R value inside one record is stamped with that record's single coarse timestamp — there's no cumulative spread across the beats:So
maxInSec(R-R-per-second) is really measuring R-R-per-record, which lines up with what's reported: ~60–109 on 5.0/MG vs ~8 on 4.0. On 5.0/MG the strap packs many beats per record onto one coarse ts; on 4.0 the record is small, but there's a second effect — the two optical channels double-write near-identical R-R (~34 ms apart), which is the classic #1008 ~2× cross-second over-count.That separates the problem into two independent fixes, which is why single de-dup attempts (e.g. #1352
collapseOverCount) can't be right for both gens:beatTs = anchor ± Σrr, so the beats land at their real inter-beat spacing instead of all onts. This alone fixes the 5.0/MG count and gives every gen a physiologically-spaced tachogram.Caveats before anyone builds it:
tsthe window start or end? Cumsum direction depends on it. Best resolved against @artemc's Polar H10 dataset (thanks for that — it's the ground truth that makes this validatable rather than guessed).tsthe record carries; the FIX Settings: Sex picker → .menu, fixes oversized Settings screen (#43) #72 RTC correction happens elsewhere), so a wrong strap clock shifts where the beats sit but doesn't change how many collapse per record.Next concrete step is a targeted log confirming
maxInSec == rrPerRecordon a live 5.0/MG offload (via the live-console path), then the cumsum spread + 4.0-gated de-dup as one parity PR validated against the Polar set.feat(rhythm): a truthful empty state when a device can't support the …
pilleuspulcher-blip commentedon Aug 16, 2026
@ryanbr
I hope it's the correct thread here to share a new straplog with the last test build (v10. 1.0-staging).
noop-strap-log-260816-1205.txt
noop-hrv-Android-v10.1.0-staging-260816-1205.zip
feat(hrv): persist a per-night R-R over-count flag for the HRV caveat (…
feat(hrv): caption an over-counted night's HRV reading "unverified" (r…
diag(hrv): shadow-log a deduped R-R stream to validate the over-count…
42 remaining items
rodrigosa7 commentedon Sep 20, 2026
Hey @ryanbr
Tonight the same issue occurred, I'll add the logs here!
report_connection.txt
report_hrv.txt
afansv commentedon Sep 21, 2026
WHOOP 4.0, NOOP 11.8.0 (iOS 18.6, sideload). The signature is back: every night 09-14 → 09-20 is crossSecondOverCount, coverage 1.55–1.78, beatAccurate 0.44–0.50, avgHrv=nil. Per-path census looks clean (live-standard ratio 0.90–0.92, historical ratioRep 0.97–1.21) while stored nights read ~2 beats/s — consistent with the two-transport union. hrv rrsample pairs differ by a constant ≈1.049 (e.g. 765/802, 780/819, 741/778), matching medPPT=1049. Strap log attached.
noop-strap-log-260921-1427.txt
fix(oura): score one R-R beat channel per window, not 0x80 on top of …
fix(hrv): do not replace a refused main night's HRV with the day's naps
log(hrv): describe the main night the gate judged in the hrv diag line
hrv: do not replace a refused main night's HRV with the day's naps (#…
log(hrv): describe the main night the gate judged in the hrv diag line
log(hrv): describe the main night the gate judged in the hrv diag line (
fix(oura): score one R-R beat channel per window, not 0x80 on top of …
fix(oura): score one R-R beat channel per window, not 0x80 on top of …
ryanbr commentedon Sep 26, 2026
Cause identified and addressed on main. @rodrigosa7's commits are
c1beffa51and5e063cd75, with a follow-up of mine at1106f6ddd, via #2478.The title said the cause was unidentified. It was this: a WHOOP 4.0 banks type-47 historical R-R AND an unlabelled live standard-BLE feed, the two overlap over the same beats, and both carried
srcChannelNULL, so nothing at read time could tell them apart. Scored together they double-count across the seam, which is whatcrossSecondOverCountwas reporting, and the #1118 coverage gate then withheld a whole-night RMSSD that was otherwise plausible.Historical rows now carry code 8, and
rrIntervalsselects ONE stream for the requested window: the labelled history when the window holds any, the unlabelled rows otherwise. Analysis fingerprints bump so cached nights rescore. The same shape #2423 settled on for two Oura channels.Marked done on the strength of the identified mechanism rather than a field confirmation, because the over-count had a specific and now-removed cause. Hardware proof is owed, and worth reopening for if it disagrees: a whole-night RMSSD on a 4.0 whose window carries both a type-47 offload and live BLE, with
hrv diagshowing coverage at or below 1.0 instead ofcrossSecondOverCount. The 2026-08-25 non-reproducibility noted in the title fits the mechanism, since whether the two feeds overlap in a given window depends on when the offload ran.One residual, recorded in #2478 rather than left to be found: promotion relabels only exact key matches, so a PARTIAL offload can leave un-promoted unlabelled rows that the all-or-nothing selection drops, lowering coverage. That fails toward a refusal to score rather than a wrong number, and it shows in the coverage line.
#2335 stays open. This removes one cause of blank HRV; the calibration half of that report is a separate question.
WHOOP 4 R-R sources: repin the ingress test, pair extractStreams
rr: choose a WHOOP's R-R delivery path per five-minute segment (recip…
fix(oura): score one R-R beat channel per window, not 0x80 on top of …