Repository navigation
Importers: one capped stream reader instead of five - #1017
Merged
Merged
Conversation
readCapped is the memory-exhaustion guard on user-supplied import files - a content:// URI or a zip entry can claim any size, and every importer reads the whole thing before parsing. It existed FIVE times, with identical bodies, under two names (readCapped / readCappedBytes) and two shapes (extension / member). The duplication was known, not accidental: each copy carried a comment saying the twins elsewhere "are not visible here". That is the wrong thing to accept for a guard - a hardening or a fix lands on one and misses four. The CAP is deliberately NOT centralised. It is policy and legitimately differs by file type: 32 MB for a lab-marker CSV, 512 MB for a Xiaomi zip entry. Each importer keeps its constant and passes it; only the mechanism moves. The failure wording is preserved exactly via a `what` parameter - everything says "Input exceeds ..." except the zip-entry path, which says "Entry" - so no user-visible message changes. Seven tests, where the five copies had none between them: content under the cap returned byte-for-byte, multi-chunk reads, exactly-at-cap allowed, one-over throws, empty allowed, both failure messages, and that it throws BEFORE buffering an oversize stream, which is the property the guard exists for rather than merely returning an error. Nine orphaned java.io imports removed with the bodies. Kotlin only warns on those and the ad-hoc compile runs -nowarn, so they would have sat unnoticed.
All five originals were file-private. Extracting them produced a PUBLIC extension on InputStream, which is a wider API than the refactor asked for - it would surface on every InputStream in the module's autocomplete and be callable from anywhere. Nothing outside com.noop.ingest calls it, and Kotlin has no package-private, so internal is the tightest scope that still reaches the five importers and the unit tests. Precedent in the tree: HealthConnectImporter.maxSourceLong is internal and tested the same way. Widening a private helper to public API is the kind of change that should be asked for, not arrive as a side effect of moving code.
simoncad7
pushed a commit
to simoncad7/noop
that referenced
this pull request
Aug 17, 2026
* Importers: one capped stream reader instead of five readCapped is the memory-exhaustion guard on user-supplied import files - a content:// URI or a zip entry can claim any size, and every importer reads the whole thing before parsing. It existed FIVE times, with identical bodies, under two names (readCapped / readCappedBytes) and two shapes (extension / member). The duplication was known, not accidental: each copy carried a comment saying the twins elsewhere "are not visible here". That is the wrong thing to accept for a guard - a hardening or a fix lands on one and misses four. The CAP is deliberately NOT centralised. It is policy and legitimately differs by file type: 32 MB for a lab-marker CSV, 512 MB for a Xiaomi zip entry. Each importer keeps its constant and passes it; only the mechanism moves. The failure wording is preserved exactly via a `what` parameter - everything says "Input exceeds ..." except the zip-entry path, which says "Entry" - so no user-visible message changes. Seven tests, where the five copies had none between them: content under the cap returned byte-for-byte, multi-chunk reads, exactly-at-cap allowed, one-over throws, empty allowed, both failure messages, and that it throws BEFORE buffering an oversize stream, which is the property the guard exists for rather than merely returning an error. Nine orphaned java.io imports removed with the bodies. Kotlin only warns on those and the ad-hoc compile runs -nowarn, so they would have sat unnoticed. * Keep the shared reader internal, as every copy it replaces was private All five originals were file-private. Extracting them produced a PUBLIC extension on InputStream, which is a wider API than the refactor asked for - it would surface on every InputStream in the module's autocomplete and be callable from anywhere. Nothing outside com.noop.ingest calls it, and Kotlin has no package-private, so internal is the tightest scope that still reaches the five importers and the unit tests. Precedent in the tree: HealthConnectImporter.maxSourceLong is internal and tested the same way. Widening a private helper to public API is the kind of change that should be asked for, not arrive as a side effect of moving code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five copies of a safety guard become one, with the tests none of them had.
What
readCappedisThe memory-exhaustion guard on user-supplied import files. A
content://URI or a zip entry can claim any size it likes, and every importer reads the whole thing into memory before parsing — so without a ceiling a malicious or merely broken file takes the process down with an OOM.It existed five times, with identical bodies, under two names (
readCapped/readCappedBytes) and two shapes (extension function / member function):ActivityFileImporterreadCappedNutritionCsvImporterreadCappedLiftingImporterreadCappedLabMarkerCsvImportreadCappedBytesWearableExportImporterreadCappedThe duplication was known, not accidental — each copy carried a comment noting that the twins elsewhere "are not visible here". That is the wrong thing to accept for a guard: a hardening or a fix lands on one and misses four.
What is deliberately NOT shared
The cap. It is policy and it legitimately differs by file type — 32 MB for a lab-marker CSV against 512 MB for a Xiaomi zip entry. Each importer keeps its own constant and passes it in. Only the mechanism moves.
The failure wording, preserved exactly via a
whatparameter: everything says"Input exceeds …"except the zip-entry path, which says"Entry exceeds …". Those messages reach users through an import failure, so none of them change.Tests
Seven, where the five copies had none between them:
> cap, not>=, and nothing pinned thatInputStream, because "returns an error eventually" is not the property this exists for; bounded peak allocation isVerification
src/main/javacompile against amainworktree — 2517 errors both sides, error sets byte-identical, 0 in any of the six touched files.OK (7 tests).Tools/doc_comment_lint.pyclean.java.ioimports removed with the bodies. Kotlin only warns on unused imports and the ad-hoc compile runs-nowarn, so these would have sat there unnoticed — worth flagging as a gap in how I verify, not just a tidy-up.Net: 81 lines deleted, 7 added across the importers, plus one 40-line helper and its tests.
Parity
Android-only. The Swift importers take
Datafrom the document picker rather than streaming anInputStream, so there is no equivalent read loop to consolidate. No stored value or analytic changes.Re-review: the extraction silently widened visibility
All five originals were file-private. Extracting them produced a public extension on
InputStream— a wider API than the refactor asked for, surfacing on everyInputStreamin the module's autocomplete and callable from anywhere.Nothing outside
com.noop.ingestcalls it, and Kotlin has no package-private, so it is nowinternal: the tightest scope that still reaches the five importers and the unit tests. Precedent in the tree —HealthConnectImporter.maxSourceLongisinternaland tested the same way.Worth naming because it is a whole class of quiet change: moving code between scopes alters its visibility, and the compiler never complains about getting more access than you had. Widening a private helper to public API should be asked for, not arrive as a side effect of a refactor.
Also re-verified this pass:
zinis aZipInputStream, so the extension resolves on the subclass andread()still returns -1 at the end of the entry rather than the stream — the behaviour the removed member function relied on is preserved. AndWearableExportImporterlegitimately still importsInputStream, which is why only itsByteArrayOutputStreamimport was removed.