diag: carry the bytes for an unmapped packet type, and sample every layout by ryanbr · Pull Request #1998 · ryanbr/noop

ryanbr · 2026-09-08T07:26:49Z

Two gaps in what a strap log carries, both on the paths whose entire job is making an unmapped firmware
mappable. Prompted by looking at what a 5/MG actually gives us for #1992 and the v20/v21 decode gaps.

1. #891 names an unmapped record type and withholds its bytes

The census line says:

the strap sent N record(s) of packet type X, which this decoder has no rows for ... please report it on
#891 with the strap model and firmware build

and carries no hex. The frame is not recoverable anywhere else either, which the code comment already
says out loud: the decoder's fallthrough drops it, and rejectedHistoricalRecords archives only type-47.
So the single line that asks for a report is the one that makes the report unactionable.

Each unmapped type now dumps its first frame, full length, once per session. First sighting only, so a
30k-record offload of one unmapped type still costs exactly one dump, matching the census line's own
first-sighting rule.

2. The RE dump samples arrival order, not layout

Spo2ReTrace took the first 8 decodable records in arrival order. On a strap emitting a dominant layout
beside a rarer one, the budget is spent inside the first chunk on the layout already at parity, and the
rare one, which is the one still unmapped, never gets a frame.

The failure mode is specific and bad: the log announces "historical records use layout vN" from
loggedLayoutVersions while carrying zero bytes of vN. Proof a thing exists, with no material to work on
it. This bites v20/v21 in particular, because they decode, so they never reach the reject path that would
otherwise have dumped them.

The budget is now per layout version: 3 each, 12 a session. Guarantees samples for a layout at 1% of
traffic, and stops re-dumping one already understood.

A deliberate asymmetry

Sampling sits in the extractor on Android and in the Backfiller on Apple.

Android's extractor is handed raw ByteArrays, so it samples in place, next to the family-aware type
naming that decides what the sample is keyed by (duplicating that naming in the Backfiller is exactly the
kind of thing that drifts). Swift's extractor only sees ParsedFrames, and ParsedFrame.rawHex is
deliberately empty on the ingest fast path (collectFields: false, D#969). Reading it there would have
logged an empty dump on every real offload while passing any test that built its own ParsedFrame -- so
the bytes are zipped back in at the Backfiller, which is the level that actually has them. The emitted
line is identical on both platforms and pinned by tests on both sides.

Tests

+7 Kotlin, +4 Swift. The selection is pinned, not just the formatting: that a sample matches the frame
that earned it, that only the first frame of a type is taken, that the excluded-by-design types
(METADATA, CONSOLE_LOGS) are not sampled, that no frame of that type yields no line rather than an
empty dump, and that neither dump truncates. The two budget tests that pinned MAX_SAMPLES == 8 are
repinned to the new shape rather than deleted, and gained an assertion that the per-layout cap is
strictly smaller than the session cap, since stratifying does nothing if it is not.

Android: 670 classes / 5666 tests, 0 failures. Swift is CI-verified only.

Privacy

Both dumps go through the existing log seam, so they inherit the redactHexDumpPii scrub that runs at
the log-sanitiser layer. Nothing new is written to a file directly.

…ayout Two gaps in what the strap log dumps, both on the paths that exist to make an unmapped firmware mappable. #891 asks a reporter to raise an issue about a record type NOOP has no rows for, and carries none of its bytes. The frame is not recoverable anywhere else either: the decoder's fallthrough drops it, and the reject archive only ever holds type-47. So the one line that says "please report this" was the one that made the report unactionable. Each unmapped type now dumps its first frame, full length, once per session. The SpO2 RE dump took the first 8 decodable records in arrival order, so on a strap emitting a dominant layout beside a rarer one the whole budget went to the layout already at parity, inside the first chunk. A log would then announce "historical records use layout vN" while carrying no bytes of vN at all. The budget is now per layout version (3 each, 12 a session), which guarantees samples for a layout at 1% of traffic and stops re-dumping one we already read. The sampling sits in the extractor on Android and in the Backfiller on Apple, for a real reason rather than drift: Swift's extractor only sees ParsedFrames, whose rawHex is empty on the ingest fast path, so reading it there would have logged an empty dump on every real offload while passing any test that built its own ParsedFrame. The emitted line is identical on both platforms.

… time Two costs in the diagnostic paths, one of them a regression I introduced in #1998. Stratifying the SpO2 RE dump per layout version broke the only thing bounding its work. The loop's outer guard counts DUMPS, so once the per-version cap holds dumps back the count sticks below the session budget and never trips it again. On a strap emitting a single layout, which is the common case, the cap is reached after three records and every later chunk then re-decodes every frame to rediscover a version already at cap. That is a second full decode of the whole offload, on top of the extractor's, for a dump that can never fire again. The search is now bounded by what it EXAMINES rather than only by what it dumps. 512 records is several chunks' worth, so a layout at a few percent of traffic is still found, and the cost is fixed instead of proportional to offload length. The budget goes on both platforms even though Apple reads the already-parsed record and pays far less: examining different frames would dump different records, and that is the one thing this line's byte-identical promise cannot survive. Separately, the two capture paths encoded hex with String.format PER BYTE, which parses the format string and allocates a Formatter for each one. They write a line per frame for a whole offload and a deep-buffer frame is 2140 bytes. A nibble lookup into one preallocated CharArray does the same job with a single allocation, and a test pins the output identical across the whole byte range because these dumps are read by existing decode tooling. The helper is deliberately not called toHexString: the stdlib has an experimental ByteArray.toHexString() of the same arity since Kotlin 1.9, and while an explicit import outranks a default one, both return lowercase unseparated hex, so the day that one stabilises a silent swap would not fail a single test.

… time Two costs in the diagnostic paths, one of them a regression I introduced in #1998. Stratifying the SpO2 RE dump per layout version broke the only thing bounding its work. The loop's outer guard counts DUMPS, so once the per-version cap holds dumps back the count sticks below the session budget and never trips it again. On a strap emitting a single layout, which is the common case, the cap is reached after three records and every later chunk then re-decodes every frame to rediscover a version already at cap. That is a second full decode of the whole offload, on top of the extractor's, for a dump that can never fire again. The search is now bounded by what it EXAMINES rather than only by what it dumps. 512 records is several chunks' worth, so a layout at a few percent of traffic is still found, and the cost is fixed instead of proportional to offload length. The budget goes on both platforms even though Apple reads the already-parsed record and pays far less: examining different frames would dump different records, and that is the one thing this line's byte-identical promise cannot survive. Separately, the capture paths on BOTH platforms encoded hex with a formatter PER BYTE, which parses the format string and allocates a Formatter for each one. They write a line per frame for a whole offload and a deep-buffer frame is 2140 bytes. A nibble lookup into one preallocated CharArray does the same job with a single allocation, and a test pins the output identical across the whole byte range, on each platform against that platform's old implementation, because these dumps are read by existing decode tooling. Apple had the same cost on the same paths: the deep-buffer capture writes a line per captured frame and the reject archive one per rejected frame, which on an unmapped-layout strap is every frame of every chunk. String(format:) is if anything worse there, since it bridges to NSString formatting and parses the format string on every call. The one-off sites on both platforms are left alone. The helper is deliberately not called toHexString: the stdlib has an experimental ByteArray.toHexString() of the same arity since Kotlin 1.9, and while an explicit import outranks a default one, both return lowercase unseparated hex, so the day that one stabilises a silent swap would not fail a single test.

… time (#2001) Two costs in the diagnostic paths, one of them a regression from #1998. Stratifying the SpO2 RE dump per layout version broke the only thing bounding its work. The loop's outer guard counts DUMPS, so once the per-version cap holds dumps back the count sticks below the session budget and never trips it again. On a strap emitting a single layout, which is the common case, the cap is reached after three records and every later chunk then re-decodes every frame to rediscover a version already at cap: a second full decode of the whole offload, on top of the extractor's, for a dump that can never fire again. The search is now bounded by what it EXAMINES rather than only by what it dumps. 512 records is several chunks' worth, so a layout at a few percent of traffic is still found, and the cost is fixed instead of proportional to offload length. The budget goes on both platforms even though Apple reads the already-parsed record and pays far less, because examining different frames would dump different records and that is the one thing this line's byte-identical promise cannot survive. Separately, the capture paths on both platforms encoded hex with a formatter PER BYTE. They write a line per frame for a whole offload and a deep-buffer frame is 2140 bytes, so one frame cost ~2140 formatter allocations. A nibble lookup into one preallocated buffer does the same job with a single allocation. The one-off sites are left alone, as is FeatureFlagProbe.hex, which looks similar but joins with a separator and elides long payloads. Each encoder is pinned against its own platform's previous implementation across the whole byte range, because these dumps are read by existing decode tooling.

Thirty real conflicts. The Android tree is dropped again per FORK_GUIDE; every other hunk was resolved by asking which mechanism this fork actually runs, not by preferring a side. Taken from upstream: • Protocol: ryanbr#2017/ryanbr#2018/ryanbr#2020 — the 5/MG historical optical and raw-IMU layouts, and the v26 window's ABSOLUTE base code beside its deltas. Its migration lands as `v57-ppg-waveform-base-code` (upstream's v44), since this fork's sequence is nine ahead of theirs. • BLE diagnostics ryanbr#1998-ryanbr#2003: a bond end that says WHY, named BLE integers, a session-budgeted hex dump, and "we hold the link" only when the OS says so. • Battery: ryanbr#1951 stop asking a 5/MG for a reading it refuses; ryanbr#1959 poll 0x2A19 for an unbonded one. • Scoring: ryanbr#1978/ryanbr#1982/ryanbr#1990 the RHR/HRV baseline gates, ryanbr#1977 the strain denominator's domain, RMSSD withheld on an over-counted night, ryanbr#1943 the resting-HR bin gate, the closed final HR/HRV window, HR-recovery requiring one CONTINUOUS effort, ryanbr#1984 nights bridged under minSleepMin, ryanbr#2015 motion re-folded only when that day's gravity moved, ryanbr#2016 a provisional Rest. • Charts ryanbr#2004/ryanbr#2005/ryanbr#2007/ryanbr#2009/ryanbr#2011 and ryanbr#1995's hero-ring detail route, ryanbr#1164's "Pending sync" caption under Rest. • Widgets: the heart-rate widget with its trace (ryanbr#1957/ryanbr#1964) and the widget cost counters. • Health: ryanbr#1503 — the Apple Health external UUID drops the strap id, so a re-pair no longer strands every prior record, plus the one-off sweep that reclaims the stranded ones; ryanbr#1205 GPS routes read back from Apple Health. • Storage: rolling retention for ppgWaveformSample, the skin-temp absolute backfill outside the 21-night window, per-table byte estimates. • Units split (body vs exercise distance), per-day wake overrides, the wind-down nudge pinned to its own day, and the German Sie→du conversion (29 entries, merged per LOCALE so the fork's own de edits stayed). Kept on the fork's side, with the reason: • ryanbr#1005's in-memory `dayScanCache` stays out, again: the persisted `DayScanFingerprint` survives relaunch and its key folds the per-day weight and max HR upstream's does not. • The steps-motion cache stays PERSISTED for the same reason. Upstream's failed-read-vs-empty-day distinction IS adopted into it — on a persisted cache a stale zero from a read error would outlive the process that made it. • Every stream read in the day loop keeps `limit: Int.max`. All five are uncapped together; taking StreamReadCap for three of them would leave the loop half-capped, which is worse than either choice. • The coach engine. Upstream's K1-K15 layer is not adopted: its single 40-row `coachMessage` table, its tool-less pre-baked context and its own brief are behind what this fork already ships. Its ADDITIVE half is kept — the on-device voice input, the Siri "Ask Coach" intent, the brief scheduler and brief widget, and streaming for the OpenAI-compatible providers this fork's Anthropic-only StreamingToolClient does not cover. They are wired up in the next commit; `CoachBriefWidget` is deliberately not registered until then. • Liquid's `DomainTheme` labels, value-sampled Charge tint, `LiquidScoreGauge` and `TodayCardSurface`; classic Today's ring order and `LiveEffort.today`. • RootTabView's four-tab shell and `MoreCatalog`: Coach hangs on content here, not on a tab. Two duplicate mechanisms collapsed rather than carried: • `ChargeBreakdownWiring` (upstream) and `ChargeBreakdownFormat` (fork) are the same extraction done twice. The fork's survives, Coupled now reads it, and upstream's TEST is kept — it is coverage this path never had. • Upstream's `CoachMessageStore` is dropped; `CoachConversationStore` already autosaves the whole multi-conversation history. One defect found while resolving, fixed here: `LiveEffort.today` read HR with no explicit limit, so `hrSamples`' 8000-row default truncated it — at ~18k rows a day that covered about ten hours and the live Effort silently stopped climbing. It failed safe (`effectiveEffort` takes the max), which is why it went unseen. `Live5RestFrozenTests` is removed by upstream, which replaces it with `Live5NoGravityRestChainTests`. `local_day_windows_oracle.json` moves out of upstream's android path into `Tests/StrandAnalyticsTests/Oracles/` — NOT `Fixtures/`, which this repo's .gitignore excludes everywhere. Verified: StrandAnalytics 2233, WhoopStore 627, WhoopProtocol 700, StrandImport 293, StrandDesign 120, NoopLocalAccess 14, SemanticMemory 19, OuraProtocol 209 and PolarProtocol 25 all green; macOS and iOS both build; doc_comment_lint clean; `i18n_audit --ci` byte-identical to its pre-merge output (it is red on main for reasons that predate this, and the seven new widget literals were given all nine locales so the gate did not get worse). Six StrandTests were ALREADY failing on main before this merge and still are — `CoachMyLogsAndZonesTests` (the tool census pin was not bumped when the Hevy tools landed) and five in `RawPhysiologyUnionTests`. Every input to them — CoachTools.swift, ToolConsent.swift, Repository.swift and both test files — is byte-identical to main, so this merge neither caused nor fixed them. They went unnoticed because app-build.yml last ran on 2026-08-25. Analysis migration required: YES. The scoring changes above make stored derived values stale; the recipe bump and its resumable migration are the next commit.

ryanbr force-pushed the diag/unmapped-type-hex branch from e6dbb2b to 93d4e49 Compare September 8, 2026 07:32

ryanbr force-pushed the diag/unmapped-type-hex branch from 93d4e49 to 2534992 Compare September 8, 2026 07:34

ryanbr merged commit c45daa4 into main Sep 8, 2026
16 checks passed

ryanbr deleted the diag/unmapped-type-hex branch September 8, 2026 08:04

ryanbr mentioned this pull request Sep 8, 2026

perf: bound the RE dump's search, and stop formatting hex a byte at a time #2001

Merged

Navigation Menu

Repository navigation

diag: carry the bytes for an unmapped packet type, and sample every layout - #1998

diag: carry the bytes for an unmapped packet type, and sample every layout#1998
ryanbr merged 1 commit into
mainfrom
diag/unmapped-type-hex

ryanbr commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant