Repository navigation
Android import: bound peak RAM with a per-import total-byte budget (#70 parity) - #125
Merged
Merged
Conversation
… parity) Android twin of the iOS StrandImport change in PR #70. WhoopCsvImporter and WearableExportImporter each collect every recognised entry into an in-memory Map<String, ByteArray> guarded only by a per-entry cap (256 MB) and, for Wearable, a per-COUNT cap (200k). Nothing bounded the SUM of the retained set -- a large or crafted export (the Wearable path can carry up to 200k entries) could accumulate unbounded ByteArray in the map before parsing ever runs. Add an aggregate MAX_TOTAL_BYTES budget (1 GB) enforced in both importers' zip loaders: once adding the next retained entry would push the running total past the budget, collection stops. Real exports (a few MB Whoop, tens of MB Wearable) never approach it, so this is behaviour-preserving. Matches the Swift maxTotalBytes. Improvement over #70: not silent. #70 (like the pre-existing MAX_FILES cap) stops collecting with a bare break. Here the loaders return a `truncated` flag and the ImportSummary message says "(partial -- export exceeded the 1 GB import memory budget)" when it trips, so a truncated import is diagnosable instead of reading as a clean import over incomplete data. Also: the Wearable zip loader now dedups first-wins (was last-wins overwrite), which keeps the running total exact and matches the Swift zip loader's guard. Testability: extracted WearableExportImporter.collectZipEntries as an InputStream seam (same pattern as AppleHealthImporter.parseStreamForTest) so a JVM unit test drives the budget from an in-memory zip -- no ContentResolver needed. Verified locally: :app:compileFullDebugKotlin + the new ImportByteBudgetTest + the full com.noop.ingest test package all pass.
digitalerdude
pushed a commit
to digitalerdude/noop
that referenced
this pull request
Jul 9, 2026
New raw-SpO2 readout (ryanbr#93), unit/display fixes (skin temp degF ryanbr#101, Effort scale ryanbr#45), the workout-in-progress card restored on Home (ryanbr#105), Apple Health steps counted (ryanbr#89), steadier battery alerts (ryanbr#80/ryanbr#99/ryanbr#114), clearer overnight-HRV copy (ryanbr#92), a lighter/faster app (ryanbr#128/ryanbr#129/ryanbr#70/ryanbr#125), and honest diagnostics (Oura anchor ryanbr#91, Test Centre sleep capture ryanbr#127). - docs/releases/v8.5.0.md (release notes) - android build.gradle.kts: versionCode 267->268, versionName 8.4.0->8.5.0 - project.yml: MARKETING_VERSION 8.4.0->8.5.0, CURRENT_PROJECT_VERSION 178->179 - in-app What's New: AppChangelog.swift + AppChangelog.kt 8.5.0 entry (parallel) Android :app:compileFullDebugKotlin passes.
DX23876
pushed a commit
to DX23876/noop
that referenced
this pull request
Jul 22, 2026
Android import: bound peak RAM with a per-import total-byte budget (ryanbr#70 parity)
DX23876
pushed a commit
to DX23876/noop
that referenced
this pull request
Jul 22, 2026
New raw-SpO2 readout (ryanbr#93), unit/display fixes (skin temp degF ryanbr#101, Effort scale #45), the workout-in-progress card restored on Home (ryanbr#105), Apple Health steps counted (ryanbr#89), steadier battery alerts (ryanbr#80/ryanbr#99/ryanbr#114), clearer overnight-HRV copy (ryanbr#92), a lighter/faster app (ryanbr#128/ryanbr#129/ryanbr#70/ryanbr#125), and honest diagnostics (Oura anchor ryanbr#91, Test Centre sleep capture ryanbr#127). - docs/releases/v8.5.0.md (release notes) - android build.gradle.kts: versionCode 267->268, versionName 8.4.0->8.5.0 - project.yml: MARKETING_VERSION 8.4.0->8.5.0, CURRENT_PROJECT_VERSION 178->179 - in-app What's New: AppChangelog.swift + AppChangelog.kt 8.5.0 entry (parallel) Android :app:compileFullDebugKotlin passes.
ryanbr
added a commit
that referenced
this pull request
Jul 27, 2026
… batch (#888) #848 landed rolling retention for v18AuxSample by copying the insertRawImu shape (#423). The shape is right; the constant it depends on is 168x larger, and the cost scales with it. The delete finds the Nth-newest row by RANK, and there is no shortcut — SQLite walks N index entries to know which one is Nth. insertRawImu keeps 3,600 rows so that is free. This keeps 604,800, and Backfiller calls insert() once per offload CHUNK, so every chunk of every sync paid a walk. Swept once per V18_AUX_PRUNE_EVERY_ROWS (10,000) banked rows instead. Measured at ~75ns/entry, a 50-chunk sync against a full table goes from ~2.3s of CPU to ~0.05s; against a 100k-row table, from ~0.38s to ~0.01s. The frequency reduction applies from day one — even a small table goes from 50 walks per sync to one. RETENTION SEMANTICS UNCHANGED. Still newest-N-rows, not a time window. A window would be cheaper still and wrong: the cap's own doc notes v18 seconds are a fraction of a day, so a sporadically-worn strap's 604,800 rows span far more than a week, and the census this table exists for wants that history. The table may sit up to the threshold plus the crossing batch above the cap — well under a MB against its ~50 MB ceiling. The counter is keyed by DEVICE because the delete is; a shared one would let one strap's inserts spend another's budget and the overshoot bound would not hold. The sweep is BEST-EFFORT, and has to be. It is now its own transaction rather than riding the insert's, so the rows are already committed by the time it runs: a throw would surface as an insert failure and make Backfiller re-send a chunk it has already banked. The budget is cleared only on success, so a failed sweep simply retries on the next batch. On Kotlin the CancellationException is rethrown rather than swallowed by runCatching — pruneV18Aux is a suspend call, and the same rethrow AppViewModel (#125) and HealthConnectWriter already use. Swift's syncWrite is synchronous, so `try?` there can only swallow GRDB errors, which is the intent. TESTS. The threshold is injectable because the existing retention tests insert ten rows against a cap of three and assert three survive; under amortisation they would have seen ten. Those tests pin the retention SQL, so they now pass v18AuxPruneEveryRows: 1 and prove exactly what they did before. Four new tests cover the amortisation: deferred below the threshold, enforced once crossed, the counter resetting so a long offload sweeps repeatedly, and the budget not being shared between devices — the last built to fail against a shared counter. SCOPE. Preventive rather than urgent: LIMIT walks min(rowcount, cap), and this table was created in #848, so no user has a full one yet. Two earlier revisions also swapped the counter for a ConcurrentHashMap and switched it to count rows written rather than attempted; both were dropped as unreachable and immaterial respectively, and the diff went from 202 lines to 142.
simoncad7
pushed a commit
to simoncad7/noop
that referenced
this pull request
Aug 17, 2026
… batch (ryanbr#888) ryanbr#848 landed rolling retention for v18AuxSample by copying the insertRawImu shape (ryanbr#423). The shape is right; the constant it depends on is 168x larger, and the cost scales with it. The delete finds the Nth-newest row by RANK, and there is no shortcut — SQLite walks N index entries to know which one is Nth. insertRawImu keeps 3,600 rows so that is free. This keeps 604,800, and Backfiller calls insert() once per offload CHUNK, so every chunk of every sync paid a walk. Swept once per V18_AUX_PRUNE_EVERY_ROWS (10,000) banked rows instead. Measured at ~75ns/entry, a 50-chunk sync against a full table goes from ~2.3s of CPU to ~0.05s; against a 100k-row table, from ~0.38s to ~0.01s. The frequency reduction applies from day one — even a small table goes from 50 walks per sync to one. RETENTION SEMANTICS UNCHANGED. Still newest-N-rows, not a time window. A window would be cheaper still and wrong: the cap's own doc notes v18 seconds are a fraction of a day, so a sporadically-worn strap's 604,800 rows span far more than a week, and the census this table exists for wants that history. The table may sit up to the threshold plus the crossing batch above the cap — well under a MB against its ~50 MB ceiling. The counter is keyed by DEVICE because the delete is; a shared one would let one strap's inserts spend another's budget and the overshoot bound would not hold. The sweep is BEST-EFFORT, and has to be. It is now its own transaction rather than riding the insert's, so the rows are already committed by the time it runs: a throw would surface as an insert failure and make Backfiller re-send a chunk it has already banked. The budget is cleared only on success, so a failed sweep simply retries on the next batch. On Kotlin the CancellationException is rethrown rather than swallowed by runCatching — pruneV18Aux is a suspend call, and the same rethrow AppViewModel (ryanbr#125) and HealthConnectWriter already use. Swift's syncWrite is synchronous, so `try?` there can only swallow GRDB errors, which is the intent. TESTS. The threshold is injectable because the existing retention tests insert ten rows against a cap of three and assert three survive; under amortisation they would have seen ten. Those tests pin the retention SQL, so they now pass v18AuxPruneEveryRows: 1 and prove exactly what they did before. Four new tests cover the amortisation: deferred below the threshold, enforced once crossed, the counter resetting so a long offload sweeps repeatedly, and the budget not being shared between devices — the last built to fail against a shared counter. SCOPE. Preventive rather than urgent: LIMIT walks min(rowcount, cap), and this table was created in ryanbr#848, so no user has a full one yet. Two earlier revisions also swapped the counter for a ConcurrentHashMap and switched it to count rows written rather than attempted; both were dropped as unreachable and immaterial respectively, and the diff went from 202 lines to 142.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Android twin of the iOS
StrandImportchange in #70 — the two importers are deliberate ports of each other, and both had the same gap.Problem (same as #70, on Android)
WhoopCsvImporterandWearableExportImportereach collect every recognised entry into an in-memoryMap<String, ByteArray>guarded only by a per-entry cap (256 MB) and, for Wearable, a per-count cap (200k). Nothing bounded the sum of the retained set — a large or crafted export (Wearable can carry up to 200k entries) could accumulate unboundedByteArrayin RAM before parsing.Change
Add an aggregate
MAX_TOTAL_BYTES(1 GB) enforced in both importers' zip loaders: once adding the next retained entry would push the running total past the budget, collection stops. Real exports (a few MB Whoop, tens of MB Wearable) never approach it → behavior-preserving. Matches the SwiftmaxTotalBytes.Improvement over #70: not silent
#70 (like the pre-existing
MAX_FILEScap) stops with a barebreak. Here the loaders return atruncatedflag and theImportSummarymessage says "(partial — export exceeded the 1 GB import memory budget)" when it trips — so a truncated import is diagnosable instead of reading as a clean import over incomplete data. (This is the log-on-trip nicety noted in the #70 review.)Also: the Wearable zip loader now dedups first-wins (was last-wins overwrite), which keeps the running total exact and matches the Swift zip loader's guard.
Testability + verification
Extracted
WearableExportImporter.collectZipEntriesas anInputStreamseam (same pattern asAppleHealthImporter.parseStreamForTest), so a JVM unit test drives the budget from an in-memory zip — noContentResolver. Verified locally::app:compileFullDebugKotlin+ the newImportByteBudgetTest(4 cases) + the fullcom.noop.ingestpackage all pass — so unlike the iOS side (blocked by the CSQLite wall on WSL), this half is actually test-proven here.Pairs with #70 (iOS); merge either order.