tools(spo2): let the @82 validator read a whoop_sync.py capture DB directly (#103) by ryanbr · Pull Request #819 · ryanbr/noop

ryanbr · 2026-07-26T04:21:43Z

Removes an export round-trip between two tools in the same directory.

Correction first, since it was my argument for opening this. I originally justified this as letting the validator read "the database the app already wrote." That's wrong. The SQLite frames table is created and written by whoop_sync.py (captures/whoop.db) — the Linux capture script. Neither shipped app has a frames table: Android uses Room, iOS/macOS uses GRDB. This cannot be pointed at a phone's store or a .noopbak. I inferred it from whoop_activity.py reading a .db and never checked what writes it.

The honest benefit, which is smaller. whoop_sync.py writes .db; this validator read only .json. So a Linux capture had to be round-tripped through whoop_sync.py export --only-type 47 before a tool sitting in the same directory could analyse it. This removes that step.

python3 validate_spo2_candidate.py captures/whoop.db my_whoop_data/ --device strap-a --postable

It does not make an existing corpus reachable unless that corpus came from whoop_sync.py.

The change itself. The SQLite path reads the same frames WHERE device_id=? AND inner_type=47 that whoop_activity.records() uses and yields the same {"hex": ...} shape the existing decode seam takes. Nightly means, pairing, r/MAE/bias and the offset-specificity scan are untouched.

Detected by the 16-byte SQLite header, not the extension — capture DBs get renamed freely.
Both failure modes report rather than returning nothing. A wrong --device-id, and a SQLite file that isn't a frame store, each exit naming the cause. Silently finding zero nights would read as "the candidate doesn't track", which is the wrong conclusion to draw from a filter mismatch.

Verification: 5 new tests (SQLite read, header-not-extension, JSON path unchanged, both error paths). Suite 170 → 175, OK (1 skipped), on Linux. Measured at scale too — 50k frames peak ~93 MB RSS, extrapolating to ~290 MB for a 258k-frame DB, which is fine for a desktop diagnostic but worth knowing before pointing it at something much larger.

Python tooling and docs only. #103 stays open — this is plumbing for the instrument, not the answer.

…e.json (#103) The #103 bar is nightly spo2_candidate_82 values tracking the app's own SpO2 across several straps, including the device whose two checked nights moved OPPOSITE to the app. validate_spo2_candidate.py could only read a capture.json from hci_extract / whoop_capture — so collecting that evidence meant asking every owner to re-capture over HCI, when their offloaded history is already sitting in NOOP's SQLite store. That is a much harder ask than opening a database they already have, and it was the practical blocker on getting multi-device evidence at all. The frame source now accepts either. The SQLite path reads the same `frames WHERE device_id=? AND inner_type=47` that whoop_activity.records() reads, yields the same {"hex": ...} shape the existing decode seam takes, and leaves the rest of the pipeline untouched — nightly means, pairing, r/MAE/bias and the offset-specificity scan are all unchanged. Identified by the 16-byte SQLite header rather than the extension: a store copied off a phone is routinely renamed, and trusting the suffix would hand it to json.load. --device-id (default 2, matching whoop_activity.py) covers a non-default row filter, and batch entries may carry their own. Both failure modes report instead of returning nothing. A wrong --device-id and a SQLite file that is not a frame store each exit with a message naming the cause; silently finding zero nights would read as "the candidate does not track", which is the wrong conclusion to hand someone from a filter mismatch. Five tests: SQLite read, header-not-extension detection, JSON path unchanged, and both error paths. Suite 170 -> 175, OK (1 skipped). Verified end to end against a throwaway NOOP-shaped store.

The justification for this change was wrong. I described the SQLite `frames` table as "where a NOOP install's own offloaded history already lives" and the feature as letting the validator read "the database the app already wrote". Neither is true: `whoop_sync.py` creates and writes that table (captures/whoop.db). Neither shipped app has a `frames` table — Android is Room, iOS/macOS is GRDB — so this path cannot reach a phone store or a .noopbak. Inferred from whoop_activity.py reading a .db without checking what writes it. The code is unchanged and still worth having, but the honest benefit is smaller: whoop_sync.py writes .db while this tool read only .json, so a Linux capture had to be round-tripped through `whoop_sync.py export --only-type 47` to be analysed by a tool in the same directory. This removes that step. It does NOT make an existing corpus reachable if that corpus did not come from whoop_sync.py. Corrected in the module docstring, the loader docstring, both SystemExit messages, the --device-id help, the test docstring and the README, which now carries an explicit note that this is the capture tooling's database. Suite still 175, OK.

@82

…d bytes Re-review measured the SQLite path on a realistic corpus and found it loads far more than it uses. `inner_type=47` matches every historical layout, but only v18 carries @82 — and newer 5/MG firmware serves a v20 (2140 B) + v21 (1244 B) pair every second beside the 124-byte v18. So the rows this tool discards are ~17x the size of the rows it wants. Measured, 20k usable v18 records in a v18+v20 store: before 40,000 rows loaded, 115 MB peak RSS after 20,000 rows loaded, 28 MB peak RSS The filter is `substr(hex, 19, 2) = '12'` — hist_version at frame offset 9 is hex chars 19-20 (substr is 1-indexed) and 0x12 is 18. It costs nothing: the store already indexes (device_id, inner_type), and the version test is a string compare on rows that index already selected. An empty result now distinguishes its two causes. "N type-47 frames but none are v18" is a different problem from "no frames for this device_id", and they need different fixes; the device-id diagnostic added earlier is unaffected, which has its own test. Suite 175 -> 178, OK (1 skipped).

@82

…lassify a strap that never emits it (#860) tools/linux-capture/validate_spo2_candidate.py (#814, extended by #819) decides whether spo2_candidate_82 graduates to a real spo2Pct. Three properties of @82 made its numbers unreliable. All three established on an 18,650-record v18 corpus: 18,602 from @digitalerdude's public PacketLogger capture of an official-app overnight sync (posted on #103), plus 48 from a NOOP sync on a subscription-free strap. 1. @82 IS DUTY-CYCLED, AND THE HARNESS DID NOT KNOW. Nonzero in 450/18,650 records (2.41 %), 24 distinct values in 1..160, falling in 15 contiguous runs of exactly 30 records each, every run starting at unix % 1200 == 337 with zero phase variance. A 30-second measurement once every 20 minutes. A capture not aligned to that phase reads all zeros and is indistinguishable from a strap with the feature off — sub-sampling the corpus at one record per minute off-phase reproduces it exactly: 310 records, zero nonzero @82, 0/16 windows. A plausible contributor to the contradictory cross-device evidence on #103, where one device correlated +0.99 and another moved opposite; the second may not have disagreed so much as missed the window. detect_duty_cycle() now measures period, phase, window length, count and jitter per capture and classifies it absent / continuous / duty_cycled / irregular. Only duty_cycled changes aggregation, so a capture it cannot characterise is never silently reweighted. The phase is deliberately NOT a constant: the robust fact is that a stable periodic window exists, and 337 is one strap's firmware. Verified independently on four schedules the tests do not use, including phase > period and a 2-minute period — all recovered exactly — and it falls through to irregular on two windows, scattered firings and phase jitter, with the jitter tolerance scaling to capture cadence rather than fixed. Robust to 3 of 10 windows missing without the period collapsing to a multiple. Nightly aggregate is now the mean of per-window means, not of every second: a 30-second burst is one measurement, and per-second averaging hands the night to whichever window was sampled most densely (95.92 vs 93.70 on the real corpus). Coverage is reported per night and warned on below --min-window-coverage. 2. IN-BAND-ONLY SCREENING IS NOT SPECIFIC. Screening on "value in 70-100" is a range, not a signature: six offsets are in-band on a majority of records on a real strap, including @17 — byte 2 of the u32 unix timestamp at @15 — and the already-decoded cardiac_flags, step_cadence and two aux thermal channels. A nightly SpO2 cannot have 2 distinct values. The scan now requires >=5 distinct in-band values and stdev >=0.5 before ranking a correlation, reports rejected offsets rather than dropping them silently, and holds @82 to the same floor. The regression test plants a 2-valued byte that correlates BETTER than @82; without the floor it wins the scan and the tool concludes "not specific to @82" from a byte that cannot be a measurement. 3. A STRAP THAT NEVER EMITS @82 IS CLASSIFIED, NOT FAILED. It is now feature_absent, neither PASS nor FAIL, gated behind >=100 records and >=3 nominal periods of scored sleep so a short capture that merely missed the window stays a plain FAIL. KNOWN GAP, follow-up immediately: that evidence bar measures duration, not phase, so it does not catch a capture whose cadence aliases with the duty period. The same working strap at 1200/337/30 over six nights of eight hours scored sleep classifies duty_cycled/fail at 1 Hz and absent/feature_absent at one sample per five minutes — a 300 s cadence occupies only 4 residues mod 1200, so it either always hits the window or never does. This is finding 1 applied to finding 3. It matters because it is asymmetric: feature_absent removes the device from `eligible`, so all_pass is computed without it and promotion gets easier. record_interval_s is already computed when the mode is absent, so the gate is a few lines. PRIVACY. --postable still carries no raw SpO2 values — verified with distinctive per-night values, none appear. Added columns are device timing and a classification; the phase is the useful thing to share, being what another owner needs to align a capture. TESTS. tools/linux-capture 178 -> 194, validator file 16 -> 32, all 16 pre-existing unchanged and still passing; confirmed on both branches with unittest discover. SCOPE. Python capture tooling plus two doc sections. No app code, no decoder, no schema, no Swift/Kotlin — spo2_candidate_82 remains instrumentation-only and this does not move it. Serves the #103 promote gate, where until now a device could fail for having captured the wrong 30 seconds out of every 20 minutes. Related: #845. Corpus credit to @digitalerdude (#103).

@82

…18 in SQL (ryanbr#103) (ryanbr#819) validate_spo2_candidate.py read only capture.json, while whoop_sync.py writes its captures to SQLite — so a Linux capture had to be round-tripped through `whoop_sync.py export --only-type 47` before a tool in the same directory could analyse it. This reads the .db directly. python3 validate_spo2_candidate.py captures/whoop.db my_whoop_data/ --postable The SQLite path reads the same `frames` table whoop_activity.py reads and yields the same {"hex": ...} shape the existing decode seam takes; nightly means, pairing, r/MAE/bias and the offset-specificity scan are untouched. SCOPE, STATED PLAINLY This is the CAPTURE TOOLING's database, not the app's. Neither shipped app has a `frames` table — Android is Room, iOS/macOS is GRDB — so it cannot be pointed at a phone store or a .noopbak. The PR originally claimed otherwise; that was inferred from whoop_activity.py reading a .db without checking what writes it, and the second commit corrects it everywhere the claim appeared. It also does not, on its own, advance ryanbr#103. The open question there is the cross-device contradiction — two nights on the original device moving OPPOSITE to the app value — which needs captures and CSV exports from further straps, not a better reader. V18 FILTERED IN SQL, WHICH MAKES THIS THE CHEAPER PATH `inner_type=47` matches every historical layout, but only v18 carries @82, and newer 5/MG firmware serves a v20 (2140 B) + v21 (1244 B) pair every second beside the 124-byte v18. Filtering with `substr(hex, 19, 2) = '12'` (hist_version at frame offset 9; 0x12 == 18) costs nothing on top of the existing (device_id, inner_type) index. Measured on 40,000 frames (20k v18 + 20k v20), same 20k records decoded: capture.json path 40,000 rows loaded 199 MB peak .db path 20,000 rows loaded 112 MB peak Provably equivalent output: decode_v18 gates on `frame[9] != 18 -> None` (whoop_activity.py:61), so every row the filter excludes was already discarded. Confirmed empirically on v20 and v26 frames. The JSON path keeps its 199 MB — its peak occurs inside json.load, before any filter can run, so improving it needs streaming and is left alone here. FAILURES REPORT THEIR CAUSE A wrong --device-id, a SQLite file that is not a frame store, and a store with type-47 frames but no v18 each exit naming the reason. Silently returning nothing would read as "the candidate does not track", which is the wrong conclusion to hand someone from a filter mismatch — and quiet empty results are exactly what this tooling has been bitten by. Detection is by the 16-byte SQLite header rather than the extension, since capture DBs get renamed freely; verified case-insensitive against uppercase-hex stores. 8 new tests including one that checks the version filter does not mask the device-id diagnostic. Suite 170 -> 178, OK (1 skipped), on Linux. Python tooling and docs only.

@82

…lassify a strap that never emits it (ryanbr#860) tools/linux-capture/validate_spo2_candidate.py (ryanbr#814, extended by ryanbr#819) decides whether spo2_candidate_82 graduates to a real spo2Pct. Three properties of @82 made its numbers unreliable. All three established on an 18,650-record v18 corpus: 18,602 from @digitalerdude's public PacketLogger capture of an official-app overnight sync (posted on ryanbr#103), plus 48 from a NOOP sync on a subscription-free strap. 1. @82 IS DUTY-CYCLED, AND THE HARNESS DID NOT KNOW. Nonzero in 450/18,650 records (2.41 %), 24 distinct values in 1..160, falling in 15 contiguous runs of exactly 30 records each, every run starting at unix % 1200 == 337 with zero phase variance. A 30-second measurement once every 20 minutes. A capture not aligned to that phase reads all zeros and is indistinguishable from a strap with the feature off — sub-sampling the corpus at one record per minute off-phase reproduces it exactly: 310 records, zero nonzero @82, 0/16 windows. A plausible contributor to the contradictory cross-device evidence on ryanbr#103, where one device correlated +0.99 and another moved opposite; the second may not have disagreed so much as missed the window. detect_duty_cycle() now measures period, phase, window length, count and jitter per capture and classifies it absent / continuous / duty_cycled / irregular. Only duty_cycled changes aggregation, so a capture it cannot characterise is never silently reweighted. The phase is deliberately NOT a constant: the robust fact is that a stable periodic window exists, and 337 is one strap's firmware. Verified independently on four schedules the tests do not use, including phase > period and a 2-minute period — all recovered exactly — and it falls through to irregular on two windows, scattered firings and phase jitter, with the jitter tolerance scaling to capture cadence rather than fixed. Robust to 3 of 10 windows missing without the period collapsing to a multiple. Nightly aggregate is now the mean of per-window means, not of every second: a 30-second burst is one measurement, and per-second averaging hands the night to whichever window was sampled most densely (95.92 vs 93.70 on the real corpus). Coverage is reported per night and warned on below --min-window-coverage. 2. IN-BAND-ONLY SCREENING IS NOT SPECIFIC. Screening on "value in 70-100" is a range, not a signature: six offsets are in-band on a majority of records on a real strap, including @17 — byte 2 of the u32 unix timestamp at @15 — and the already-decoded cardiac_flags, step_cadence and two aux thermal channels. A nightly SpO2 cannot have 2 distinct values. The scan now requires >=5 distinct in-band values and stdev >=0.5 before ranking a correlation, reports rejected offsets rather than dropping them silently, and holds @82 to the same floor. The regression test plants a 2-valued byte that correlates BETTER than @82; without the floor it wins the scan and the tool concludes "not specific to @82" from a byte that cannot be a measurement. 3. A STRAP THAT NEVER EMITS @82 IS CLASSIFIED, NOT FAILED. It is now feature_absent, neither PASS nor FAIL, gated behind >=100 records and >=3 nominal periods of scored sleep so a short capture that merely missed the window stays a plain FAIL. KNOWN GAP, follow-up immediately: that evidence bar measures duration, not phase, so it does not catch a capture whose cadence aliases with the duty period. The same working strap at 1200/337/30 over six nights of eight hours scored sleep classifies duty_cycled/fail at 1 Hz and absent/feature_absent at one sample per five minutes — a 300 s cadence occupies only 4 residues mod 1200, so it either always hits the window or never does. This is finding 1 applied to finding 3. It matters because it is asymmetric: feature_absent removes the device from `eligible`, so all_pass is computed without it and promotion gets easier. record_interval_s is already computed when the mode is absent, so the gate is a few lines. PRIVACY. --postable still carries no raw SpO2 values — verified with distinctive per-night values, none appear. Added columns are device timing and a classification; the phase is the useful thing to share, being what another owner needs to align a capture. TESTS. tools/linux-capture 178 -> 194, validator file 16 -> 32, all 16 pre-existing unchanged and still passing; confirmed on both branches with unittest discover. SCOPE. Python capture tooling plus two doc sections. No app code, no decoder, no schema, no Swift/Kotlin — spo2_candidate_82 remains instrumentation-only and this does not move it. Serves the ryanbr#103 promote gate, where until now a device could fail for having captured the wrong 30 seconds out of every 20 minutes. Related: ryanbr#845. Corpus credit to @digitalerdude (ryanbr#103).

ryanbr added 2 commits July 25, 2026 21:21

ryanbr changed the title ~~tools(spo2): let the @82 validator read a NOOP store, not only capture.json (#103)~~ tools(spo2): let the @82 validator read a whoop_sync.py capture DB directly (#103) Jul 26, 2026

ryanbr merged commit 095779d into main Jul 26, 2026
1 check passed

vishk23 mentioned this pull request Jul 27, 2026

tools(spo2): account for @82's duty cycle, screen for variance, and classify a strap that never emits it #860

Merged

ryanbr deleted the tools/spo2-validate-from-db branch August 2, 2026 07:41

Navigation Menu

Repository navigation

tools(spo2): let the @82 validator read a whoop_sync.py capture DB directly (#103) - #819

tools(spo2): let the @82 validator read a whoop_sync.py capture DB directly (#103)#819
ryanbr merged 3 commits into
mainfrom
tools/spo2-validate-from-db

ryanbr commented Jul 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ryanbr commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ryanbr commented Jul 26, 2026 •

edited

Loading