From 188e2726985886a31f8ee6c6010abd5e5435bb31 Mon Sep 17 00:00:00 2001 From: Fanboynz Date: Wed, 9 Sep 2026 19:21:05 +1200 Subject: [PATCH 1/3] protocol: keep the v26 optical window's absolute base, not just its deltas The WHOOP 5/MG v26 record encodes a 25-sample optical window as ONE absolute ADC code followed by 24 deltas. Both platforms read only the deltas, stored them as if they were the samples, and discarded the base. The evidence is in a frame already in the test fixtures. At frame-abs 23 sits 378,307, a valid 20-bit optical code, in the four bytes immediately before the 24 values we do read. Every one of those 24 is NEGATIVE, which no absolute optical reading can be. Reconstructed as base plus running sum they give 25 samples, all inside the 20-bit domain, with an excursion of 4.2% of the base: an ordinary PPG perfusion index. 24 deltas plus a base is also 25 samples, matching the 25-sample optical block v20 carries. No shipped metric is wrong. Nothing in analytics reads `ppgWaveformSample`, and the strap's own PPG-derived heart rate comes from `ppgHrSample`. What was wrong is the corpus being banked for the work this stream exists for: a derivative stored as a signal, and the DC level thrown away, which is precisely what an SpO2 ratio-of-ratios needs. The base is now read and stored beside the deltas rather than folded into them, because the sample blob is little-endian i16 and a real code does not fit in one, and because base-plus-deltas is the shape the strap actually sends. `ppgWaveformAbsolute` reconstructs, with a Kotlin twin, and returns null for a row whose base is null rather than inventing a window from a fabricated zero. That null is the honest part. Every v26 record ingested before this is unrecoverable: a delta series cannot be inverted without its starting point, and no backfill can supply one. Only future records can be right, which is the whole argument for landing it rather than waiting. Schema on both stores, which needed five things moved in step, not the three I had written down: Room's `@Database(version)`, `SCHEMA_VERSION`, the migration chain, and BOTH copies of `schema_oracle.json`, which pins the room version AND the per-table column set and is byte-compared across the platforms. That last guard is what forced the Swift store to gain the column too, which is correct: the two stores are meant to match. Caveat carried in the code: the deltas are SATURATED, clamped at the i16 bounds, so a window holding a clamped delta reconstructs only approximately. The captured fixture's largest magnitude is 1,833, nowhere near 32,768. Found by comparing against OpenStrap's Dart decoder (MIT), which documents the encoding. No code taken; the offset it names was verified against our own captured frame, and both platforms now independently decode 378,307 from it. Android 5,744 green; WhoopProtocol 685 green. --- .../WhoopProtocol/HistoricalStreams.swift | 31 ++++++++++- .../Sources/WhoopProtocol/Interpreter.swift | 12 ++++- .../Sources/WhoopProtocol/Streams.swift | 12 ++++- .../Whoop5PpgWaveformTests.swift | 27 +++++++++- .../Sources/WhoopStore/Database.swift | 12 +++++ .../Sources/WhoopStore/StreamStore.swift | 5 +- .../Resources/schema_oracle.json | 20 +++++--- .../src/main/java/com/noop/data/Entities.kt | 16 +++++- .../main/java/com/noop/data/WhoopDatabase.kt | 21 ++++++-- .../java/com/noop/data/WhoopRepository.kt | 11 +++- .../com/noop/protocol/HistoricalStreams.kt | 51 +++++++++++++++++-- .../protocol/Whoop5PpgWaveformStreamTest.kt | 43 +++++++++++++++- .../app/src/test/resources/schema_oracle.json | 20 +++++--- 13 files changed, 249 insertions(+), 32 deletions(-) diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift index 398cf40094..0204ecd28e 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift @@ -258,7 +258,8 @@ public func extractHistoricalStreams(_ parsed: [ParsedFrame], if let samples = p["ppg_waveform"]?.intArrayValue, !samples.isEmpty { ppgRecords.append((ts: ts, samples: samples)) out.ppgWaveform.append(PpgWaveformSample(ts: ts, samples: samples, - burstIndex: p["burst_index"]?.intValue)) + burstIndex: p["burst_index"]?.intValue, + baseCode: p["ppg_base_code"]?.intValue)) } if let bpm = p["heart_rate"]?.intValue, bpm != 0 { // skip startup hr=0 out.hr.append(HRSample(ts: ts, bpm: bpm)) @@ -433,3 +434,31 @@ public func extractHistoricalStreams(_ parsed: [ParsedFrame], out.droppedImplausibleNewestTs = droppedNewest return out } + +/// Reconstruct a WHOOP 5/MG v26 optical window from its stored parts. #2019. +/// +/// The strap sends a 25-sample window as one absolute ADC code plus 24 deltas, so this is the only way to +/// get back the signal the strap measured: `sample[0] = baseCode`, `sample[i+1] = sample[i] + delta[i]`. +/// NOOP stores the two as they arrive rather than folding them together, because the delta blob is +/// little-endian i16 and a real code (about 378,000 on the captured fixture) does not fit in one. +/// +/// nil when `baseCode` is nil, which is the honest answer for a row written before the base was read: a +/// delta series cannot be inverted without its starting point, and returning the deltas, or a window +/// built from a fabricated zero, would present a signal nobody measured. +/// +/// CAVEAT: the deltas are SATURATED, clamped at the i16 bounds by the encoder, so a window containing a +/// clamped delta reconstructs only approximately. Nothing here can detect that after the fact; a delta at +/// exactly ±32,768 is the signal to distrust, and the captured fixture's largest magnitude is 1,833. +/// +/// Mirror EXACTLY in Kotlin (`ppgWaveformAbsolute`). +public func ppgWaveformAbsolute(baseCode: Int?, deltas: [Int]) -> [Int]? { + guard let baseCode else { return nil } + var out: [Int] = [baseCode] + out.reserveCapacity(deltas.count + 1) + var acc = baseCode + for d in deltas { + acc += d + out.append(acc) + } + return out +} diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift index d6568bd17b..752efb5cd8 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift @@ -643,6 +643,16 @@ private func decodeWhoop5HistoricalV26(_ frame: [UInt8], fb: FieldBuilder) { if let unix = readDType(frame, 15, "u32") { fb.add(15, 4, "unix", "time", value: .int(unix), note: "real unix seconds") } + // #2019: the ABSOLUTE optical code the 24 values below are deltas FROM. The window is 25 samples, + // not 24: sample 0 is this code and delta i produces sample i+1. Reading only the deltas and calling + // them the waveform stored a derivative as if it were a signal, and discarded the DC level, which is + // the half an SpO2 ratio-of-ratios needs. Verified on the captured frame in the waveform tests: + // 378,307 here, a valid 20-bit code, against 24 deltas that are every one NEGATIVE, which no + // absolute optical reading can be. + if let base = readDType(frame, 23, "u32") { + fb.add(23, 4, "ppg_base_code", "ppg", value: .int(base), + note: "absolute optical ADC code; ppg_waveform holds the deltas from it") + } var samples: [Int] = [] for off in stride(from: 27, to: 75, by: 2) { guard let v = readI16(frame, off) else { break } @@ -650,7 +660,7 @@ private func decodeWhoop5HistoricalV26(_ frame: [UInt8], fb: FieldBuilder) { } if !samples.isEmpty { fb.add(27, samples.count * 2, "ppg_waveform", "ppg", value: .intArray(samples), - note: "optical PPG @24 Hz, LE-i16 ADC counts") + note: "optical PPG DELTAS, LE-i16; absolute sample i+1 = base + running sum") fb.parsed["ppg_sample_count"] = .int(samples.count) } // PR#563: the remaining per-record v26 bytes, surfaced as RAW NEUTRAL fields — read off the real diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Streams.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Streams.swift index a556bf99ed..77dae0561c 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Streams.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Streams.swift @@ -373,12 +373,20 @@ public struct SleepStateSample: Equatable, Codable { /// `decodeWhoop5HistoricalV26`); a truncated frame can yield fewer than 24. public struct PpgWaveformSample: Equatable, Codable, Sendable { public let ts: Int // wall-clock unix seconds (one record per second) - public let samples: [Int] // raw i16 ADC counts @24 Hz, verbatim from `ppg_waveform` (usually 24) + /// The window's 24 i16 DELTAS, verbatim from `ppg_waveform`. #2019: these are NOT absolute samples. + /// The strap sends a 25-sample window as one absolute code plus 24 deltas; reconstruct with + /// `ppgWaveformAbsolute(baseCode:deltas:)`. + public let samples: [Int] public let burstIndex: Int? // raw per-burst counter @21; nil for legacy archives - public init(ts: Int, samples: [Int], burstIndex: Int? = nil) { + /// #2019: the absolute optical ADC code `samples` are deltas from, at frame-abs 23. nil on a row + /// written before it was read, and that nil is TRUE rather than merely missing: a delta series + /// cannot be inverted without its starting point, so those windows have no recoverable level. + public let baseCode: Int? + public init(ts: Int, samples: [Int], burstIndex: Int? = nil, baseCode: Int? = nil) { self.ts = ts self.samples = samples self.burstIndex = burstIndex + self.baseCode = baseCode } } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift index 1441e12826..b151ae7527 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift @@ -101,12 +101,37 @@ final class Whoop5PpgWaveformTests: XCTestCase { let f = parseFrame(bytes(v26Hex), family: .whoop5) let streams = extractHistoricalStreams([f], deviceClockRef: 1_780_917_232, wallClockRef: 1_780_917_232) XCTAssertEqual(streams.ppgWaveform, - [PpgWaveformSample(ts: 1_780_917_232, samples: expectedWaveform, burstIndex: 1)]) + [PpgWaveformSample(ts: 1_780_917_232, samples: expectedWaveform, burstIndex: 1, + baseCode: expectedBaseCode)]) XCTAssertTrue(streams.ppgHr.isEmpty, "a lone 1 s record is too short for a confident HR estimate") // Not "no rows at all" — the Backfiller's silent-data-loss diagnostic must see this as decoded. XCTAssertFalse(streams.isEmpty) } + /// #2019: the absolute optical code the stored values are deltas FROM, at frame-abs 23. It is what + /// makes the window reconstructable and what used to be discarded. + private let expectedBaseCode = 378_307 + + /// The reconstruction, on the real captured window. Every absolute sample must land inside the + /// 20-bit ADC domain, which is the check that says the base and the deltas belong together: read as + /// absolute values the stored deltas are all NEGATIVE, and no optical reading can be. The excursion + /// is 4.2% of the base, an ordinary PPG perfusion index. Mirrored in Kotlin. + func testTheWindowReconstructsIntoTheOpticalDomain() { + let f = parseFrame(bytes(v26Hex), family: .whoop5) + let streams = extractHistoricalStreams([f], deviceClockRef: 1_780_917_232, + wallClockRef: 1_780_917_232) + let row = try! XCTUnwrap(streams.ppgWaveform.first) + let absolute = try! XCTUnwrap(ppgWaveformAbsolute(baseCode: row.baseCode, deltas: row.samples)) + XCTAssertEqual(absolute.count, 25, "one absolute code plus 24 deltas is a 25-sample window") + XCTAssertEqual(absolute.first, expectedBaseCode) + XCTAssertEqual(absolute.last, 362_532) + XCTAssertTrue(absolute.allSatisfy { $0 >= 0 && $0 < (1 << 20) }, + "every sample must sit in the 20-bit ADC domain") + // A legacy row, whose base was never stored, is honestly unreconstructable rather than silently + // reconstructed from a fabricated zero. + XCTAssertNil(ppgWaveformAbsolute(baseCode: nil, deltas: row.samples)) + } + func testExtractHistoricalStreamsCarriesEachBurstIndexWithItsWaveform() { let parsed = [v26Hex, v26HexChannel46].map { parseFrame(bytes($0), family: .whoop5) } let streams = extractHistoricalStreams(parsed, diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 21a049c2d6..43f09a627a 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -919,6 +919,18 @@ extension WhoopStore { // No index: the table is capped at maxStoredMessages (40 rows), so a full scan + sort on // read is negligible and an index buys nothing worth the extra Room<->GRDB parity surface. } + // #2019: carry the v26 optical window's ABSOLUTE base code beside its deltas. + // + // The 25-sample window is one absolute ADC code plus 24 deltas, and only the deltas were read, so + // the stored `samples` blob is a derivative and the DC level was thrown away. Nullable and + // additive: an existing row keeps its deltas and gets a null base, which is the true statement + // about it. A delta series cannot be inverted without the base, so those windows have no + // recoverable absolute level and no backfill can invent one. Twin of Room's MIGRATION_37_38. + migrator.registerMigration("v44-ppg-waveform-base-code") { db in + try db.alter(table: "ppgWaveformSample") { t in + t.add(column: "baseCode", .integer) + } + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift index eacc9f463f..8d947cc538 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift @@ -360,12 +360,13 @@ extension WhoopStore { // `packPpgSamples`) rather than 24 scalar rows, so this insert is O(records), not O(samples). if !streams.ppgWaveform.isEmpty { let stmt = try db.cachedStatement(sql: """ - INSERT INTO ppgWaveformSample (deviceId, ts, samples, burstIndex) VALUES (?, ?, ?, ?) + INSERT INTO ppgWaveformSample (deviceId, ts, samples, burstIndex, baseCode) + VALUES (?, ?, ?, ?, ?) ON CONFLICT(deviceId, ts) DO NOTHING """) for s in streams.ppgWaveform { try stmt.execute(arguments: [deviceId, s.ts, WhoopStore.packPpgSamples(s.samples), - s.burstIndex]) + s.burstIndex, s.baseCode]) ppgWaveformWritten += 1 } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json index 743485063f..f2d915e6b0 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json @@ -1,6 +1,6 @@ { "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here.", - "roomVersion": 37, + "roomVersion": 38, "grdbMigrations": [ "v1", "v2", @@ -47,14 +47,14 @@ "v43-coach-messages" ], "divergenceReasons": { - "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android — the per-row upload flag belongs to an upload path that does not exist there — so it is dead width, not divergent data. Removing it needs a Room table rebuild.", - "battery-alter-append-order": "REAL COLUMN-ORDER DRIFT (CLAUDE.md: 'column order in a Room CREATE TABLE must match the entity field order'). GRDB appended `synced` (v5) then `charging` (v6) with ALTER TABLE, which can only append; the Room entity declares `charging` before `synced`. Same columns, same types, different positions — so any positional copy (INSERT INTO battery SELECT * FROM …) swaps them. Fixing it means reordering the Room entity's fields, which changes Room's identity hash and therefore needs a version bump plus a no-op migration.", + "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", + "battery-alter-append-order": "REAL COLUMN-ORDER DRIFT (CLAUDE.md: 'column order in a Room CREATE TABLE must match the entity field order'). GRDB appended `synced` (v5) then `charging` (v6) with ALTER TABLE, which can only append; the Room entity declares `charging` before `synced`. Same columns, same types, different positions \u2014 so any positional copy (INSERT INTO battery SELECT * FROM \u2026) swaps them. Fixing it means reordering the Room entity's fields, which changes Room's identity hash and therefore needs a version bump plus a no-op migration.", "grdb-boolean-affinity": "GRDB's .boolean column type is declared BOOLEAN, which SQLite gives NUMERIC affinity; Room maps Kotlin Boolean to INTEGER. For the 0/1 values both sides write, NUMERIC and INTEGER affinity store bit-identical INTEGERs, so a .noopbak round-trips. Closing it means a GRDB table rebuild.", "paired-device-alter-append-order": "REAL COLUMN-ORDER DRIFT, same shape as battery. GRDB's v16-paired-device-peripheral appended `peripheralId` at the end; the Room entity declares it fifth, between `nickname` and `sourceKind`.", - "ppg-hr-bpm-integer-on-android": "AFFINITY DRIFT, and NOT a data consequence: ppgHrSample.bpm is declared REAL on iOS and INTEGER on Android — but only the SQLITE COLUMN differs. The ROW type is `Int` on both sides: Swift's `PpgHrSample.bpm` is `Int`, carrying the comment 'the same Int domain as the measured HRSample.bpm and the Android PpgHr.Estimate.bpm, so the stored value and type match across platforms (#219)'. Both platforms also round to whole bpm before that — Swift `(fsD * 60 / refinedLag).rounded()` then `Int(est.bpm)`, Kotlin `Math.round(fsD * 60 / refinedLag).toInt()` — and the read path (`CAST(ROUND(p.bpm) AS INTEGER)`) rounds again, a no-op on a whole number. There is one writer and one construction site on each side. So REAL holds the same whole numbers INTEGER does and NO value differs. Recorded because the DECLARED affinity genuinely diverges, and a bpm that ever became fractional would then diverge silently; closing it means a GRDB table rebuild. Deliberately NOT the decoder-oracle (#869) class — that caught a real value divergence.", - "room-omits-sql-default": "GRDB declares these columns NOT NULL DEFAULT 0; Room emits NOT NULL with no SQL DEFAULT, because a Kotlin constructor default never reaches the schema (only @ColumnInfo(defaultValue=) does). Both sides are NOT NULL and both always name the column on insert, so no row differs; an INSERT that omitted the column would fail on Android and get 0 on iOS. Closing it means adding @ColumnInfo(defaultValue = \"0\") plus a Room version bump, since the generated CREATE TABLE — and therefore Room's identity hash — changes.", + "ppg-hr-bpm-integer-on-android": "AFFINITY DRIFT, and NOT a data consequence: ppgHrSample.bpm is declared REAL on iOS and INTEGER on Android \u2014 but only the SQLITE COLUMN differs. The ROW type is `Int` on both sides: Swift's `PpgHrSample.bpm` is `Int`, carrying the comment 'the same Int domain as the measured HRSample.bpm and the Android PpgHr.Estimate.bpm, so the stored value and type match across platforms (#219)'. Both platforms also round to whole bpm before that \u2014 Swift `(fsD * 60 / refinedLag).rounded()` then `Int(est.bpm)`, Kotlin `Math.round(fsD * 60 / refinedLag).toInt()` \u2014 and the read path (`CAST(ROUND(p.bpm) AS INTEGER)`) rounds again, a no-op on a whole number. There is one writer and one construction site on each side. So REAL holds the same whole numbers INTEGER does and NO value differs. Recorded because the DECLARED affinity genuinely diverges, and a bpm that ever became fractional would then diverge silently; closing it means a GRDB table rebuild. Deliberately NOT the decoder-oracle (#869) class \u2014 that caught a real value divergence.", + "room-omits-sql-default": "GRDB declares these columns NOT NULL DEFAULT 0; Room emits NOT NULL with no SQL DEFAULT, because a Kotlin constructor default never reaches the schema (only @ColumnInfo(defaultValue=) does). Both sides are NOT NULL and both always name the column on insert, so no row differs; an INSERT that omitted the column would fail on Android and get 0 on iOS. Closing it means adding @ColumnInfo(defaultValue = \"0\") plus a Room version bump, since the generated CREATE TABLE \u2014 and therefore Room's identity hash \u2014 changes.", "sqlite-text-pk-nullable": "GRDB writes `id TEXT PRIMARY KEY`; SQLite's legacy quirk leaves a non-INTEGER PRIMARY KEY column NULLable, so PRAGMA reports notnull=0. Room writes `id TEXT NOT NULL, PRIMARY KEY(id)`. Every writer on both platforms supplies a non-null id, so no stored row differs. Closing it means a GRDB table rebuild.", - "workout-route-polyline-android-only": "REAL COLUMN DRIFT — literally the case #775 was filed about: `workout.routePolyline` (encoded GPS route) was added by Room MIGRATION_3_4 and has no GRDB twin. A .noopbak exported from Android carries a column iOS's `workout` table cannot receive." + "workout-route-polyline-android-only": "REAL COLUMN DRIFT \u2014 literally the case #775 was filed about: `workout.routePolyline` (encoded GPS route) was added by Room MIGRATION_3_4 and has no GRDB twin. A .noopbak exported from Android carries a column iOS's `workout` table cannot receive." }, "tables": { "appleDaily": { @@ -552,7 +552,7 @@ }, "dismissedWorkout": { "platform": "android_only", - "reason": "Durable 'this detected bout is not a workout' marker (#107, Room MIGRATION_4_5). macOS persists the equivalent as a UserDefaults span list because its read model cannot add a column to the shared workout struct — documented on the entity, not drift.", + "reason": "Durable 'this detected bout is not a workout' marker (#107, Room MIGRATION_4_5). macOS persists the equivalent as a UserDefaults span list because its read model cannot add a column to the shared workout struct \u2014 documented on the entity, not drift.", "columns": [ { "name": "deviceId", @@ -1237,6 +1237,12 @@ "affinity": "INTEGER", "notNull": false, "default": null + }, + { + "name": "baseCode", + "affinity": "INTEGER", + "notNull": false, + "default": null } ], "primaryKey": [ diff --git a/android/app/src/main/java/com/noop/data/Entities.kt b/android/app/src/main/java/com/noop/data/Entities.kt index 17a11e53f6..3fcd5bdbc7 100644 --- a/android/app/src/main/java/com/noop/data/Entities.kt +++ b/android/app/src/main/java/com/noop/data/Entities.kt @@ -692,13 +692,26 @@ data class PpgWaveformSampleEntity( val ts: Long, val samples: ByteArray, val burstIndex: Int? = null, + /** + * #2019: the absolute optical ADC code that [samples] are DELTAS from. The v26 window is 25 samples + * encoded as this code plus 24 deltas; sample 0 is the code and delta i produces sample i+1. + * + * Stored beside the deltas rather than folded into them because [samples] is a little-endian i16 blob + * and a real code (about 378,000 on the captured fixture) does not fit in an i16. Keeping the wire's + * own shape is also the honest one: the strap sends a base and deltas. + * + * Null on a row written before this was read, and that null is TRUE rather than merely missing: the + * base was discarded at decode, and a delta series cannot be inverted without it, so those rows have + * no recoverable absolute level. Use it to tell a reconstructable window from one that never can be. + */ + val baseCode: Long? = null, ) { // ByteArray needs structural equals/hashCode (the generated identity ones break round-trip asserts). override fun equals(other: Any?): Boolean { if (this === other) return true if (other !is PpgWaveformSampleEntity) return false return deviceId == other.deviceId && ts == other.ts && samples.contentEquals(other.samples) && - burstIndex == other.burstIndex + burstIndex == other.burstIndex && baseCode == other.baseCode } override fun hashCode(): Int { @@ -706,6 +719,7 @@ data class PpgWaveformSampleEntity( result = 31 * result + ts.hashCode() result = 31 * result + samples.contentHashCode() result = 31 * result + (burstIndex ?: 0) + result = 31 * result + (baseCode?.hashCode() ?: 0) return result } } diff --git a/android/app/src/main/java/com/noop/data/WhoopDatabase.kt b/android/app/src/main/java/com/noop/data/WhoopDatabase.kt index d548503982..1ed0b45a16 100644 --- a/android/app/src/main/java/com/noop/data/WhoopDatabase.kt +++ b/android/app/src/main/java/com/noop/data/WhoopDatabase.kt @@ -54,7 +54,7 @@ import androidx.sqlite.db.SupportSQLiteDatabase AppleStepHour::class, CoachMessageRow::class, ], - version = 37, + version = 38, // #775: ON so Room's KSP processor writes the generated schema (every table's exact `CREATE TABLE`, // columns in declaration order with affinity/NOT NULL/default, PK and indices) as JSON. That export // is what lets a plain JVM test — no device, no Robolectric — read Android's REAL schema and compare @@ -74,7 +74,7 @@ abstract class WhoopDatabase : RoomDatabase() { const val DB_NAME = "noop_whoop.db" /** Room schema version — MUST equal the `@Database(version = …)` above. Surfaced in the backup * manifest (#1410) so an export states its schema. Bump both together on a migration. */ - const val SCHEMA_VERSION = 37 + const val SCHEMA_VERSION = 38 @Volatile private var instance: WhoopDatabase? = null @@ -954,6 +954,21 @@ abstract class WhoopDatabase : RoomDatabase() { } } + /** + * #2019: carry the v26 optical window's ABSOLUTE base code beside its deltas. + * + * The 25-sample window is one absolute ADC code plus 24 deltas, and only the deltas were read, + * so the stored `samples` blob is a derivative and the DC level was thrown away. Nullable and + * additive: an existing row keeps its deltas and gets a null base, which is the true statement + * about it. A delta series cannot be inverted without the base, so those windows have no + * recoverable absolute level and no backfill can invent one. + */ + internal val MIGRATION_37_38 = object : Migration(37, 38) { + override fun migrate(db: SupportSQLiteDatabase) { + db.execSQL("ALTER TABLE `ppgWaveformSample` ADD COLUMN `baseCode` INTEGER") + } + } + /** PRD-K2: persisted Coach conversation history (schema v37). Swift twin: WhoopStore * Database.swift `v43-coach-messages` migration. Column order matches [CoachMessageRow] * field declaration order so Room's generated `CREATE TABLE` shape agrees. */ @@ -999,7 +1014,7 @@ abstract class WhoopDatabase : RoomDatabase() { MIGRATION_22_23, MIGRATION_23_24, MIGRATION_24_25, MIGRATION_25_26, MIGRATION_26_27, MIGRATION_27_28, MIGRATION_28_29, MIGRATION_29_30, MIGRATION_30_31, MIGRATION_31_32, MIGRATION_32_33, MIGRATION_33_34, MIGRATION_34_35, MIGRATION_35_36, - MIGRATION_36_37, + MIGRATION_36_37, MIGRATION_37_38, ) private fun build(appContext: Context): WhoopDatabase = diff --git a/android/app/src/main/java/com/noop/data/WhoopRepository.kt b/android/app/src/main/java/com/noop/data/WhoopRepository.kt index 7b8ab55487..d99a7747dd 100644 --- a/android/app/src/main/java/com/noop/data/WhoopRepository.kt +++ b/android/app/src/main/java/com/noop/data/WhoopRepository.kt @@ -312,7 +312,14 @@ data class PpgHrRow(val ts: Long, val bpm: Int, val conf: Double) * unix second, [samples] the raw i16 ADC counts (usually 24, fewer on a truncated frame). deviceId is * attached on insert; the samples are packed to a little-endian i16 BLOB by [StreamPersistence.packPpgSamples]. */ -data class PpgWaveformRow(val ts: Long, val samples: List, val burstIndex: Int? = null) +data class PpgWaveformRow( + val ts: Long, + val samples: List, + val burstIndex: Int? = null, + /** #2019: the absolute optical code [samples] are deltas from; null on a legacy row, whose absolute + * level is gone for good because a delta series cannot be inverted without it. */ + val baseCode: Long? = null, +) /** Count of rows ACTUALLY inserted per stream (mirrors WhoopStore.insert return tuple). */ data class InsertCounts( @@ -612,7 +619,7 @@ class WhoopRepository( dao.insertPpgWaveform( streams.ppgWaveform.map { PpgWaveformSampleEntity(deviceId, it.ts, StreamPersistence.packPpgSamples(it.samples), - it.burstIndex) + it.burstIndex, it.baseCode) }, ) // #1911 rolling retention, amortised and best-effort on exactly the same terms as the v18-aux diff --git a/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt b/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt index 5aa79ec3b8..dc57bc94ab 100644 --- a/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt +++ b/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt @@ -534,7 +534,12 @@ private fun decodeWhoop5Historical(frame: ByteArray): Map? { * (`burst_index`, NOT a channel id; PR#553) is carried beside the waveform * so durable rows retain their burst boundaries. The footer after [75] remains intentionally unmapped. */ -private data class V26Record(val unix: Long, val samples: List, val burstIndex: Int?) +private data class V26Record( + val unix: Long, + val samples: List, + val burstIndex: Int?, + val baseCode: Long?, +) private fun decodeWhoop5HistoricalV26(frame: ByteArray): V26Record? { if (frame.histU8(8) != PacketType.HISTORICAL_DATA.rawValue) return null @@ -542,6 +547,14 @@ private fun decodeWhoop5HistoricalV26(frame: ByteArray): V26Record? { // Long, not Int — see [histU32]. This path was never actually wrong (its one consumer re-widened // with `and 0xFFFFFFFFL`), but carrying the reader's own type removes the mask and the trap. val unix = frame.histU32(15) ?: return null + // #2019: the ABSOLUTE optical code the 24 values below are deltas FROM. The window is 25 samples, + // not 24: sample 0 is this code and delta i produces sample i+1. Reading only the deltas and calling + // them the waveform stored a derivative as if it were a signal, and threw away the DC level, which + // is the half an SpO2 ratio-of-ratios needs. Verified on the captured frame in + // `Whoop5PpgWaveformStreamTest`: 378,307 here, a valid 20-bit code, against 24 deltas that are every + // one NEGATIVE, which no absolute optical reading can be. Long, not Int: unsigned 32-bit, see + // [histU32]. + val baseCode = frame.histU32(23) val samples = ArrayList(24) var off = 27 while (off < 75) { @@ -552,7 +565,7 @@ private fun decodeWhoop5HistoricalV26(frame: ByteArray): V26Record? { if (samples.isEmpty()) return null val rawBurstIndex = frame.histU8(21) return V26Record(unix = unix, samples = samples, - burstIndex = rawBurstIndex?.takeIf { it > 0 }) + burstIndex = rawBurstIndex?.takeIf { it > 0 }, baseCode = baseCode) } /** @@ -835,7 +848,9 @@ fun extractHistoricalStreams( // corrected wall-second. Guard on non-empty so a truncated frame that decoded // zero samples never banks an empty row (mirrors the Swift `!samples.isEmpty`). if (rec.samples.isNotEmpty()) { - ppgWaveform.add(PpgWaveformRow(baseTs, rec.samples, rec.burstIndex)) + ppgWaveform.add( + PpgWaveformRow(baseTs, rec.samples, rec.burstIndex, rec.baseCode), + ) } } } @@ -1093,3 +1108,33 @@ private fun appendHistBattery(out: MutableList, ts: Long, p: Map): List? { + if (baseCode == null) return null + val out = ArrayList(deltas.size + 1) + var acc = baseCode + out.add(acc) + for (d in deltas) { + acc += d + out.add(acc) + } + return out +} diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5PpgWaveformStreamTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5PpgWaveformStreamTest.kt index 237b00d0b6..d717f5de86 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5PpgWaveformStreamTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5PpgWaveformStreamTest.kt @@ -3,6 +3,7 @@ package com.noop.protocol import com.noop.data.PpgWaveformRow import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull import org.junit.Assert.assertTrue import org.junit.Test @@ -21,12 +22,22 @@ class Whoop5PpgWaveformStreamTest { private fun bytes(s: String): ByteArray = ByteArray(s.length / 2) { ((Character.digit(s[it * 2], 16) shl 4) + Character.digit(s[it * 2 + 1], 16)).toByte() } - // Real captured WHOOP 5.0 v26 optical-PPG frame (unix @15 == 1780917232), 24 i16 samples @ [27:75]. + // Real captured WHOOP 5.0 v26 optical-PPG frame (unix @15 == 1780917232), the absolute base code + // @[23:27] and 24 i16 DELTAS @[27:75]. #2019: the window is 25 samples, one absolute code plus 24 + // deltas, and only the deltas used to be read. private val v26Hex = "aa015000010035412f1a80ad418401f0a3266aae470100c3c5050068faccfa8dfb46fc8bfd4c" + "febafedafe6dff56ffd5fffbff37ff6afce5f9d7f8dffa5efc98fddbfe5afe84fe15ff5cff40" + "5fb33c50080101006cb67c17" + /** + * #2019: the absolute optical code the deltas below are measured FROM, at frame-abs 23. It is what + * makes the window reconstructable, and what was thrown away before: 378,307 is a valid 20-bit code, + * while every delta is NEGATIVE, which no absolute optical reading can be. + */ + private val expectedBaseCode = 378_307L + + /** The 24 DELTAS, not samples. Reconstruct with [expectedBaseCode]; see `ppgWaveformAbsolute`. */ private val expectedWaveform = listOf( -1432, -1332, -1139, -954, -629, -436, -326, -294, -147, -170, -43, -5, -201, -918, -1563, -1833, -1313, -930, -616, -293, -422, -380, -235, -164, @@ -46,7 +57,10 @@ class Whoop5PpgWaveformStreamTest { family = DeviceFamily.WHOOP5, ) assertEquals( - listOf(PpgWaveformRow(ts = 1_780_917_232L, samples = expectedWaveform, burstIndex = 1)), + listOf( + PpgWaveformRow(ts = 1_780_917_232L, samples = expectedWaveform, burstIndex = 1, + baseCode = expectedBaseCode), + ), streams.ppgWaveform, ) assertTrue("a lone 1 s record is too short for a confident HR estimate", streams.ppgHr.isEmpty()) @@ -55,6 +69,31 @@ class Whoop5PpgWaveformStreamTest { assertFalse(streams.isEmpty) } + /** + * The reconstruction, on the real captured window. Every absolute sample must land inside the 20-bit + * ADC domain, which is the check that says the base and the deltas belong together: read as absolute + * values the stored deltas are all NEGATIVE, and no optical reading can be. The excursion is 4.2% of + * the base, an ordinary PPG perfusion index. + */ + @Test + fun theWindowReconstructsIntoTheOpticalDomain() { + val row = extractHistoricalStreams( + listOf(bytes(v26Hex)), + deviceClockRef = 1_780_917_232, + wallClockRef = 1_780_917_232, + family = DeviceFamily.WHOOP5, + ).ppgWaveform.single() + val absolute = ppgWaveformAbsolute(row.baseCode, row.samples)!! + assertEquals("one absolute code plus 24 deltas is a 25-sample window", 25, absolute.size) + assertEquals(expectedBaseCode, absolute.first()) + assertTrue("every sample must sit in the 20-bit ADC domain", + absolute.all { it in 0 until (1L shl 20) }) + assertEquals(362_532L, absolute.last()) + // And a legacy row, whose base was never stored, is honestly unreconstructable rather than + // silently reconstructed from a fabricated zero. + assertNull(ppgWaveformAbsolute(null, row.samples)) + } + @Test fun zeroBurstIndexIsAbsentLikeSwift() { val frame = bytes(v26Hex) diff --git a/android/app/src/test/resources/schema_oracle.json b/android/app/src/test/resources/schema_oracle.json index 743485063f..f2d915e6b0 100644 --- a/android/app/src/test/resources/schema_oracle.json +++ b/android/app/src/test/resources/schema_oracle.json @@ -1,6 +1,6 @@ { "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here.", - "roomVersion": 37, + "roomVersion": 38, "grdbMigrations": [ "v1", "v2", @@ -47,14 +47,14 @@ "v43-coach-messages" ], "divergenceReasons": { - "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android — the per-row upload flag belongs to an upload path that does not exist there — so it is dead width, not divergent data. Removing it needs a Room table rebuild.", - "battery-alter-append-order": "REAL COLUMN-ORDER DRIFT (CLAUDE.md: 'column order in a Room CREATE TABLE must match the entity field order'). GRDB appended `synced` (v5) then `charging` (v6) with ALTER TABLE, which can only append; the Room entity declares `charging` before `synced`. Same columns, same types, different positions — so any positional copy (INSERT INTO battery SELECT * FROM …) swaps them. Fixing it means reordering the Room entity's fields, which changes Room's identity hash and therefore needs a version bump plus a no-op migration.", + "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", + "battery-alter-append-order": "REAL COLUMN-ORDER DRIFT (CLAUDE.md: 'column order in a Room CREATE TABLE must match the entity field order'). GRDB appended `synced` (v5) then `charging` (v6) with ALTER TABLE, which can only append; the Room entity declares `charging` before `synced`. Same columns, same types, different positions \u2014 so any positional copy (INSERT INTO battery SELECT * FROM \u2026) swaps them. Fixing it means reordering the Room entity's fields, which changes Room's identity hash and therefore needs a version bump plus a no-op migration.", "grdb-boolean-affinity": "GRDB's .boolean column type is declared BOOLEAN, which SQLite gives NUMERIC affinity; Room maps Kotlin Boolean to INTEGER. For the 0/1 values both sides write, NUMERIC and INTEGER affinity store bit-identical INTEGERs, so a .noopbak round-trips. Closing it means a GRDB table rebuild.", "paired-device-alter-append-order": "REAL COLUMN-ORDER DRIFT, same shape as battery. GRDB's v16-paired-device-peripheral appended `peripheralId` at the end; the Room entity declares it fifth, between `nickname` and `sourceKind`.", - "ppg-hr-bpm-integer-on-android": "AFFINITY DRIFT, and NOT a data consequence: ppgHrSample.bpm is declared REAL on iOS and INTEGER on Android — but only the SQLITE COLUMN differs. The ROW type is `Int` on both sides: Swift's `PpgHrSample.bpm` is `Int`, carrying the comment 'the same Int domain as the measured HRSample.bpm and the Android PpgHr.Estimate.bpm, so the stored value and type match across platforms (#219)'. Both platforms also round to whole bpm before that — Swift `(fsD * 60 / refinedLag).rounded()` then `Int(est.bpm)`, Kotlin `Math.round(fsD * 60 / refinedLag).toInt()` — and the read path (`CAST(ROUND(p.bpm) AS INTEGER)`) rounds again, a no-op on a whole number. There is one writer and one construction site on each side. So REAL holds the same whole numbers INTEGER does and NO value differs. Recorded because the DECLARED affinity genuinely diverges, and a bpm that ever became fractional would then diverge silently; closing it means a GRDB table rebuild. Deliberately NOT the decoder-oracle (#869) class — that caught a real value divergence.", - "room-omits-sql-default": "GRDB declares these columns NOT NULL DEFAULT 0; Room emits NOT NULL with no SQL DEFAULT, because a Kotlin constructor default never reaches the schema (only @ColumnInfo(defaultValue=) does). Both sides are NOT NULL and both always name the column on insert, so no row differs; an INSERT that omitted the column would fail on Android and get 0 on iOS. Closing it means adding @ColumnInfo(defaultValue = \"0\") plus a Room version bump, since the generated CREATE TABLE — and therefore Room's identity hash — changes.", + "ppg-hr-bpm-integer-on-android": "AFFINITY DRIFT, and NOT a data consequence: ppgHrSample.bpm is declared REAL on iOS and INTEGER on Android \u2014 but only the SQLITE COLUMN differs. The ROW type is `Int` on both sides: Swift's `PpgHrSample.bpm` is `Int`, carrying the comment 'the same Int domain as the measured HRSample.bpm and the Android PpgHr.Estimate.bpm, so the stored value and type match across platforms (#219)'. Both platforms also round to whole bpm before that \u2014 Swift `(fsD * 60 / refinedLag).rounded()` then `Int(est.bpm)`, Kotlin `Math.round(fsD * 60 / refinedLag).toInt()` \u2014 and the read path (`CAST(ROUND(p.bpm) AS INTEGER)`) rounds again, a no-op on a whole number. There is one writer and one construction site on each side. So REAL holds the same whole numbers INTEGER does and NO value differs. Recorded because the DECLARED affinity genuinely diverges, and a bpm that ever became fractional would then diverge silently; closing it means a GRDB table rebuild. Deliberately NOT the decoder-oracle (#869) class \u2014 that caught a real value divergence.", + "room-omits-sql-default": "GRDB declares these columns NOT NULL DEFAULT 0; Room emits NOT NULL with no SQL DEFAULT, because a Kotlin constructor default never reaches the schema (only @ColumnInfo(defaultValue=) does). Both sides are NOT NULL and both always name the column on insert, so no row differs; an INSERT that omitted the column would fail on Android and get 0 on iOS. Closing it means adding @ColumnInfo(defaultValue = \"0\") plus a Room version bump, since the generated CREATE TABLE \u2014 and therefore Room's identity hash \u2014 changes.", "sqlite-text-pk-nullable": "GRDB writes `id TEXT PRIMARY KEY`; SQLite's legacy quirk leaves a non-INTEGER PRIMARY KEY column NULLable, so PRAGMA reports notnull=0. Room writes `id TEXT NOT NULL, PRIMARY KEY(id)`. Every writer on both platforms supplies a non-null id, so no stored row differs. Closing it means a GRDB table rebuild.", - "workout-route-polyline-android-only": "REAL COLUMN DRIFT — literally the case #775 was filed about: `workout.routePolyline` (encoded GPS route) was added by Room MIGRATION_3_4 and has no GRDB twin. A .noopbak exported from Android carries a column iOS's `workout` table cannot receive." + "workout-route-polyline-android-only": "REAL COLUMN DRIFT \u2014 literally the case #775 was filed about: `workout.routePolyline` (encoded GPS route) was added by Room MIGRATION_3_4 and has no GRDB twin. A .noopbak exported from Android carries a column iOS's `workout` table cannot receive." }, "tables": { "appleDaily": { @@ -552,7 +552,7 @@ }, "dismissedWorkout": { "platform": "android_only", - "reason": "Durable 'this detected bout is not a workout' marker (#107, Room MIGRATION_4_5). macOS persists the equivalent as a UserDefaults span list because its read model cannot add a column to the shared workout struct — documented on the entity, not drift.", + "reason": "Durable 'this detected bout is not a workout' marker (#107, Room MIGRATION_4_5). macOS persists the equivalent as a UserDefaults span list because its read model cannot add a column to the shared workout struct \u2014 documented on the entity, not drift.", "columns": [ { "name": "deviceId", @@ -1237,6 +1237,12 @@ "affinity": "INTEGER", "notNull": false, "default": null + }, + { + "name": "baseCode", + "affinity": "INTEGER", + "notNull": false, + "default": null } ], "primaryKey": [ From 2af59436157e1f4ec262fb5402094de54736eeab Mon Sep 17 00:00:00 2001 From: Fanboynz Date: Wed, 9 Sep 2026 19:28:31 +1200 Subject: [PATCH 2/3] diag: say whether the v26 windows this session banked can be reconstructed at all The previous commit documented that a SATURATED delta makes a window reconstruct only approximately, and then shipped no way to see whether one ever occurred. It also made a null base mean two different things with no way to tell them apart: a legacy row, or a record that arrived truncated, or a firmware that does not carry the base at frame-abs 23. All three would have banked silently. One line per session, beside the R-R census, and null when the session carried no v26 windows so a 4.0 or a strap that banked none stays quiet: Backfill: v26 optical census: 412 window(s), 412 with a base, 0 saturated, base 361204..379881 Each number earns its place. The window count is the volume. `with a base` below it is the signal that matters most, because those windows can NEVER be reconstructed and nothing else would say so. The saturated count is the caveat made observable rather than left in a doc comment. The base range is the session's DC level, which is the quantity the whole stream is banked for and the one that used to be discarded outright, so a log now carries enough to tell a plausible optical run from a rail- pinned one without pulling the store. Counted on what the DECODER produced rather than on what the store kept: whether a window carries a base is a fact about the wire, not about the insert. Pure and mirrored, pinned on both platforms against the same literals including the empty case, the missing-base case, the saturation caveat and an absent range reading as absent rather than as a fabricated zero. Android 5,750 green; WhoopProtocol 691 green. --- .../WhoopProtocol/HistoricalStreams.swift | 35 +++++++++ .../Whoop5PpgWaveformTests.swift | 54 +++++++++++++ .../src/main/java/com/noop/ble/Backfiller.kt | 29 +++++++ .../main/java/com/noop/ble/WhoopBleClient.kt | 7 ++ .../com/noop/protocol/HistoricalStreams.kt | 37 +++++++++ .../noop/protocol/PpgWaveformCensusTest.kt | 75 +++++++++++++++++++ 6 files changed, 237 insertions(+) create mode 100644 android/app/src/test/java/com/noop/protocol/PpgWaveformCensusTest.kt diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift index 0204ecd28e..ffc336003f 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift @@ -462,3 +462,38 @@ public func ppgWaveformAbsolute(baseCode: Int?, deltas: [Int]) -> [Int]? { } return out } + +/// A delta at either i16 rail: the encoder clamped it, so the window reconstructs only approximately. +public func isSaturatedPpgDelta(_ delta: Int) -> Bool { + delta == Int(Int16.min) || delta == Int(Int16.max) +} + +/// The per-session v26 optical census, or nil when the session carried no v26 windows (a 4.0, or a 5/MG +/// that banked none) so a log with nothing to say stays quiet. #2019. +/// +/// Three things a strap log could not previously answer, all of which decide whether the banked windows +/// are usable for the channel-mapping work this stream exists for: +/// +/// - how many windows arrived at all; +/// - how many carried the absolute base. A window without one cannot be reconstructed, ever. On a +/// well-formed record the base is always readable, so `withBase` below the window count means +/// TRUNCATED records or a firmware that does not carry it at frame-abs 23, and either is worth knowing +/// rather than silently banking un-reconstructable windows; +/// - how many windows hold a SATURATED delta. Those reconstruct only approximately, and the caveat is +/// worthless without a way to see whether it ever fires. +/// +/// The base range is carried because it is the DC level over the session, which is the quantity the whole +/// stream is banked for and the one that used to be discarded entirely. +/// +/// Mirror EXACTLY in Kotlin (`ppgWaveformCensusLine`). +public func ppgWaveformCensusLine(windows: Int, withBase: Int, saturatedWindows: Int, + baseMin: Int?, baseMax: Int?) -> String? { + guard windows > 0 else { return nil } + let range: String = { + guard let baseMin, let baseMax else { return " base n/a" } + return " base \(baseMin)..\(baseMax)" + }() + let note = saturatedWindows > 0 ? " (a saturated window reconstructs only approximately)" : "" + return "Backfill: v26 optical census: \(windows) window(s), \(withBase) with a base, " + + "\(saturatedWindows) saturated,\(range)\(note)" +} diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift index b151ae7527..b58c418d10 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5PpgWaveformTests.swift @@ -171,3 +171,57 @@ final class Whoop5PpgWaveformTests: XCTestCase { XCTAssertEqual(round.ppgWaveform, withBurst.ppgWaveform) } } + +/// #2019: the per-session optical census. Mirrored by the Kotlin `PpgWaveformCensusTest` against the +/// SAME literals, so the two platforms cannot report the same offload differently. +final class PpgWaveformCensusTests: XCTestCase { + + /// A session with no v26 windows says nothing at all: a 4.0, or a 5/MG that banked none, must not + /// print a census of zero. + func testNoWindowsIsSilent() { + XCTAssertNil(ppgWaveformCensusLine(windows: 0, withBase: 0, saturatedWindows: 0, + baseMin: nil, baseMax: nil)) + } + + /// The ordinary healthy session: every window carried a base, none saturated. + func testEveryWindowReconstructable() { + XCTAssertEqual( + ppgWaveformCensusLine(windows: 412, withBase: 412, saturatedWindows: 0, + baseMin: 361_204, baseMax: 379_881), + "Backfill: v26 optical census: 412 window(s), 412 with a base, 0 saturated, base 361204..379881") + } + + /// `withBase` below the window count is the signal that matters: those windows can never be + /// reconstructed, and before this line existed they were banked with nothing said about it. + func testWindowsMissingABaseAreVisible() { + XCTAssertEqual( + ppgWaveformCensusLine(windows: 10, withBase: 7, saturatedWindows: 0, + baseMin: 100, baseMax: 200), + "Backfill: v26 optical census: 10 window(s), 7 with a base, 0 saturated, base 100..200") + } + + /// A saturated window earns the caveat inline, because the caveat is the reason to distrust the + /// reconstruction and it is worthless if it only lives in a doc comment. + func testSaturationCarriesItsCaveat() { + let line = ppgWaveformCensusLine(windows: 5, withBase: 5, saturatedWindows: 2, + baseMin: 1, baseMax: 2) + XCTAssertEqual(line, "Backfill: v26 optical census: 5 window(s), 5 with a base, 2 saturated, " + + "base 1..2 (a saturated window reconstructs only approximately)") + } + + /// An absent range reads as absent rather than as a fabricated zero. + func testAbsentBaseRangeSaysSo() { + XCTAssertEqual( + ppgWaveformCensusLine(windows: 3, withBase: 0, saturatedWindows: 0, + baseMin: nil, baseMax: nil), + "Backfill: v26 optical census: 3 window(s), 0 with a base, 0 saturated, base n/a") + } + + /// Both i16 rails count, and an ordinary delta does not. + func testSaturationIsBothRails() { + XCTAssertTrue(isSaturatedPpgDelta(-32_768)) + XCTAssertTrue(isSaturatedPpgDelta(32_767)) + XCTAssertFalse(isSaturatedPpgDelta(-1_833)) + XCTAssertFalse(isSaturatedPpgDelta(0)) + } +} diff --git a/android/app/src/main/java/com/noop/ble/Backfiller.kt b/android/app/src/main/java/com/noop/ble/Backfiller.kt index ccd04e6d4d..e85832f57d 100644 --- a/android/app/src/main/java/com/noop/ble/Backfiller.kt +++ b/android/app/src/main/java/com/noop/ble/Backfiller.kt @@ -229,6 +229,18 @@ class Backfiller( */ var sessionSkinTempRows = 0 private set + + /** #2019: this session's v26 optical windows, and what they carried. See [ppgWaveformCensusLine]. */ + var sessionPpgWindows = 0 + private set + var sessionPpgWithBase = 0 + private set + var sessionPpgSaturated = 0 + private set + var sessionPpgBaseMin: Long? = null + private set + var sessionPpgBaseMax: Long? = null + private set private val sessionNightKeys = HashSet() val sessionNights: Int get() = sessionNightKeys.size @@ -343,6 +355,11 @@ class Backfiller( this.continuedAfterRows = continuedAfterRows isBackfilling = true sessionRowsPersisted = 0 + sessionPpgWindows = 0 + sessionPpgWithBase = 0 + sessionPpgSaturated = 0 + sessionPpgBaseMin = null + sessionPpgBaseMax = null sessionRrOffered = 0 sessionRrInserted = 0 sessionRrSumMs = 0 @@ -644,6 +661,18 @@ class Backfiller( // "persisted N rows (M with motion) across K night(s)" — the win-rate signal we never logged. val (rows, motion, nights) = chunkTally(counts, decoded.gravity.map { it.ts } + decoded.hr.map { it.ts }) sessionRowsPersisted += rows + // #2019: the v26 optical census, folded per chunk. Counted on what the DECODER produced + // rather than on what the store kept, because an un-reconstructable window is a decode + // fact: the base is either on the wire or it is not. + for (w in decoded.ppgWaveform) { + sessionPpgWindows += 1 + w.baseCode?.let { b -> + sessionPpgWithBase += 1 + sessionPpgBaseMin = minOf(sessionPpgBaseMin ?: b, b) + sessionPpgBaseMax = maxOf(sessionPpgBaseMax ?: b, b) + } + if (w.samples.any { com.noop.protocol.isSaturatedPpgDelta(it) }) sessionPpgSaturated += 1 + } // #1008/#1118 census accumulation (pre-storage offered vs post-key inserted). sessionRrOffered += rrCensus.intervals sessionRrInserted += counts.rr diff --git a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt index a3b7ee05f4..22fda60d81 100644 --- a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt +++ b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt @@ -10448,6 +10448,13 @@ class WhoopBleClient( // #1008/#1118: the pre-storage R-R census for this offload, next to the persisted tally so one // line pair says what the decoder OFFERED and what the store KEPT. Twin of the Swift emit. backfiller.sessionRrEmissionLine()?.let { rrLine -> log(rrLine) } + // #2019: and the v26 optical census, in the same place, so one block says what the offload + // banked AND whether those optical windows can be reconstructed at all. + com.noop.protocol.ppgWaveformCensusLine( + backfiller.sessionPpgWindows, backfiller.sessionPpgWithBase, + backfiller.sessionPpgSaturated, backfiller.sessionPpgBaseMin, + backfiller.sessionPpgBaseMax, + )?.let { ppgLine -> log(ppgLine) } // #990: fold this session's drained rows into the persisted ALL-TIME tally at the single // summary emit point, so the Connection readout can show install-lifetime progress beside // the per-session count (which resets on every reconnect). Unconditional, like the summary diff --git a/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt b/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt index dc57bc94ab..d6787230ee 100644 --- a/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt +++ b/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt @@ -1138,3 +1138,40 @@ fun ppgWaveformAbsolute(baseCode: Long?, deltas: List): List? { } return out } + +/** A delta at either i16 rail: the encoder clamped it, so the window reconstructs only approximately. */ +fun isSaturatedPpgDelta(delta: Int): Boolean = delta == Short.MIN_VALUE.toInt() || delta == Short.MAX_VALUE.toInt() + +/** + * The per-session v26 optical census, or null when the session carried no v26 windows (a 4.0, or a + * 5/MG that banked none) so a log that has nothing to say stays quiet. #2019. + * + * Three things a strap log could not previously answer, all of which decide whether the banked windows + * are usable for the channel-mapping work this stream exists for: + * + * - how many windows arrived at all; + * - how many carried the absolute base. A window without one cannot be reconstructed, ever. On a + * well-formed record the base is always readable, so `withBase` below the window count means + * TRUNCATED records or a firmware that does not carry it at frame-abs 23, and either is worth + * knowing rather than silently banking un-reconstructable windows; + * - how many windows hold a SATURATED delta. Those reconstruct only approximately, and the caveat is + * worthless without a way to see whether it ever fires. + * + * The base range is carried because it is the DC level over the session, which is the quantity the + * whole stream is banked for and the one that used to be discarded entirely. + * + * Twin of the Swift `ppgWaveformCensusLine`. + */ +fun ppgWaveformCensusLine( + windows: Int, + withBase: Int, + saturatedWindows: Int, + baseMin: Long?, + baseMax: Long?, +): String? { + if (windows <= 0) return null + val range = if (baseMin != null && baseMax != null) " base $baseMin..$baseMax" else " base n/a" + val note = if (saturatedWindows > 0) " (a saturated window reconstructs only approximately)" else "" + return "Backfill: v26 optical census: $windows window(s), $withBase with a base, " + + "$saturatedWindows saturated,$range$note" +} diff --git a/android/app/src/test/java/com/noop/protocol/PpgWaveformCensusTest.kt b/android/app/src/test/java/com/noop/protocol/PpgWaveformCensusTest.kt new file mode 100644 index 0000000000..2d6d8286cd --- /dev/null +++ b/android/app/src/test/java/com/noop/protocol/PpgWaveformCensusTest.kt @@ -0,0 +1,75 @@ +package com.noop.protocol + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * #2019: the per-session optical census. Twin of the Swift `PpgWaveformCensusTests` against the SAME + * literals, so the two platforms cannot report the same offload differently. + */ +class PpgWaveformCensusTest { + + /** + * A session with no v26 windows says nothing at all: a 4.0, or a 5/MG that banked none, must not + * print a census of zero. + */ + @Test + fun noWindowsIsSilent() { + assertNull(ppgWaveformCensusLine(0, 0, 0, null, null)) + } + + /** The ordinary healthy session: every window carried a base, none saturated. */ + @Test + fun everyWindowReconstructable() { + assertEquals( + "Backfill: v26 optical census: 412 window(s), 412 with a base, 0 saturated, base 361204..379881", + ppgWaveformCensusLine(412, 412, 0, 361_204L, 379_881L), + ) + } + + /** + * `withBase` below the window count is the signal that matters: those windows can never be + * reconstructed, and before this line existed they were banked with nothing said about it. + */ + @Test + fun windowsMissingABaseAreVisible() { + assertEquals( + "Backfill: v26 optical census: 10 window(s), 7 with a base, 0 saturated, base 100..200", + ppgWaveformCensusLine(10, 7, 0, 100L, 200L), + ) + } + + /** + * A saturated window earns the caveat inline, because the caveat is the reason to distrust the + * reconstruction and it is worthless if it only lives in a doc comment. + */ + @Test + fun saturationCarriesItsCaveat() { + assertEquals( + "Backfill: v26 optical census: 5 window(s), 5 with a base, 2 saturated, base 1..2 " + + "(a saturated window reconstructs only approximately)", + ppgWaveformCensusLine(5, 5, 2, 1L, 2L), + ) + } + + /** An absent range reads as absent rather than as a fabricated zero. */ + @Test + fun absentBaseRangeSaysSo() { + assertEquals( + "Backfill: v26 optical census: 3 window(s), 0 with a base, 0 saturated, base n/a", + ppgWaveformCensusLine(3, 0, 0, null, null), + ) + } + + /** Both i16 rails count, and an ordinary delta does not. */ + @Test + fun saturationIsBothRails() { + assertTrue(isSaturatedPpgDelta(-32_768)) + assertTrue(isSaturatedPpgDelta(32_767)) + assertFalse(isSaturatedPpgDelta(-1_833)) + assertFalse(isSaturatedPpgDelta(0)) + } +} From 79728be92512bb90d31ec2b8eeb7b41a905309ec Mon Sep 17 00:00:00 2001 From: Fanboynz Date: Wed, 9 Sep 2026 19:35:26 +1200 Subject: [PATCH 3/3] fix(store): read the v26 base back on Swift, and finish the schema lockstep CI caught what a machine without SQLite headers cannot: the WhoopStore suite does not build here, so three things went out unverified and one of them was a real defect. The defect: Swift's `ppgWaveformSamples` is a NAMED projection, `SELECT ts, samples, burstIndex`. A new column is invisible to it until it is listed, so the base was banked on every write and every read handed back nil. Nil is also exactly what a legacy row reads as, so the column would have looked inert rather than broken, on the platform where nothing yet reads the stream to notice. Kotlin uses `SELECT *` into the entity and was already correct, which is why the Android suites were green throughout. The round-trip test now carries a base and asserts it survives, so the projection cannot silently drop a column again. The other two were lockstep entries I had not found: `schema_oracle.json` pins the GRDB MIGRATION IDENTIFIER list as well as the room version and the per-table columns, and `PpgWaveformSampleTests` pins the table's column set on the Swift side. Both repinned. That makes SEVEN things a schema change moves in this repo, not the three I had written down and not the five I corrected it to an hour ago: the Room version, SCHEMA_VERSION, the Room migration chain, the GRDB migration, and three separate pins inside a schema_oracle.json that exists in two byte-compared copies. The guard is doing its job each time; my note was the thing that was wrong. Android 5,750 green; WhoopProtocol 691 green. The WhoopStore suite rests on CI, which is the point. --- .../Sources/WhoopStore/StreamStore.swift | 9 +++++++-- .../WhoopStoreTests/PpgWaveformSampleTests.swift | 16 ++++++++++++---- .../WhoopStoreTests/Resources/schema_oracle.json | 3 ++- .../app/src/test/resources/schema_oracle.json | 3 ++- 4 files changed, 23 insertions(+), 8 deletions(-) diff --git a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift index 8d947cc538..cd3f81aa48 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift @@ -691,13 +691,18 @@ extension WhoopStore { -> [PpgWaveformSample] { try syncRead { db in try Row.fetchAll(db, sql: """ - SELECT ts, samples, burstIndex FROM ppgWaveformSample + SELECT ts, samples, burstIndex, baseCode FROM ppgWaveformSample WHERE deviceId = ? AND ts >= ? AND ts <= ? ORDER BY ts LIMIT ? """, arguments: [deviceId, from, to, limit]) + // #2019: baseCode is SELECTed explicitly. This projection names its columns, so a new one + // is invisible to it until it is listed — the write would have banked the base and every + // read would have handed back nil, which is the same answer a legacy row gives and would + // have looked like the column doing nothing. .map { PpgWaveformSample(ts: $0["ts"], samples: WhoopStore.unpackPpgSamples($0["samples"]), - burstIndex: $0["burstIndex"]) } + burstIndex: $0["burstIndex"], + baseCode: $0["baseCode"]) } } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/PpgWaveformSampleTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/PpgWaveformSampleTests.swift index 59281e0c14..97972c9379 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/PpgWaveformSampleTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/PpgWaveformSampleTests.swift @@ -31,19 +31,27 @@ final class PpgWaveformSampleTests: XCTestCase { func testPpgWaveformTableShape() async throws { let store = try await WhoopStore.inMemory() let cols = try await store.columnNamesForTest(table: "ppgWaveformSample") - XCTAssertEqual(Set(cols), ["deviceId", "ts", "samples", "burstIndex"]) + // #2019: `baseCode` is the absolute optical code `samples` are deltas from. Nullable and + // additive, so an existing row keeps its deltas and reads null, which is the true statement + // about it: a delta series cannot be inverted without its starting point. + XCTAssertEqual(Set(cols), ["deviceId", "ts", "samples", "burstIndex", "baseCode"]) } func testPpgWaveformInsertRoundTripAndDedup() async throws { let store = try await WhoopStore.inMemory() - let streams = Streams(ppgWaveform: [PpgWaveformSample(ts: 1_780_917_232, - samples: realSamples, burstIndex: 7)]) + // #2019: with a baseCode, so the round trip proves the column is WRITTEN and READ. The read is a + // named projection, so a new column is invisible to it until it is listed, and a base that banked + // but never came back would read as nil, which is exactly what a legacy row reads as. + let streams = Streams(ppgWaveform: [PpgWaveformSample(ts: 1_780_917_232, samples: realSamples, + burstIndex: 7, baseCode: 378_307)]) _ = try await store.insert(streams, deviceId: "my-whoop") let n1 = try await store.ppgWaveformCountForTest() XCTAssertEqual(n1, 1) let read = try await store.ppgWaveformSamples(deviceId: "my-whoop", from: 1_780_917_232, to: 1_780_917_232) - XCTAssertEqual(read, [PpgWaveformSample(ts: 1_780_917_232, samples: realSamples, burstIndex: 7)]) + XCTAssertEqual(read, [PpgWaveformSample(ts: 1_780_917_232, samples: realSamples, burstIndex: 7, + baseCode: 378_307)]) + XCTAssertEqual(read.first?.baseCode, 378_307, "the absolute base must survive the round trip") // Re-inserting the same (deviceId, ts) is idempotent, ON CONFLICT DO NOTHING (mirrors every // other per-second stream's dedupe rule). _ = try await store.insert(streams, deviceId: "my-whoop") diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json index f2d915e6b0..51f12b9ed8 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json @@ -44,7 +44,8 @@ "v40-daily-skin-temp-absolute", "v41-drop-raw-imu-sample", "v42-daily-sleep-hr-only", - "v43-coach-messages" + "v43-coach-messages", + "v44-ppg-waveform-base-code" ], "divergenceReasons": { "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", diff --git a/android/app/src/test/resources/schema_oracle.json b/android/app/src/test/resources/schema_oracle.json index f2d915e6b0..51f12b9ed8 100644 --- a/android/app/src/test/resources/schema_oracle.json +++ b/android/app/src/test/resources/schema_oracle.json @@ -44,7 +44,8 @@ "v40-daily-skin-temp-absolute", "v41-drop-raw-imu-sample", "v42-daily-sleep-hr-only", - "v43-coach-messages" + "v43-coach-messages", + "v44-ppg-waveform-base-code" ], "divergenceReasons": { "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.",