Add source-aware delta OTA extraction - #1
Open
gotenksIN wants to merge 42 commits into
Open
Conversation
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Compile the pinned Android 17 Zucchini and libchrome source closure in process. Add a safe Rust buffer/status wrapper and licensed golden fixtures for every Android executable type. Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Signed-off-by: Omkar Chandorkar <gotenksIN@aospa.co>
Add an independent, C++/libchrome-free port of the Zucchini apply path. - Ensemble patch parsing (varints, streams, validation) matching patch_reader.cc - OffsetMapper, TargetPool, equivalence/extra-data/raw-delta application - ELF x86/x86-64/AArch32/AArch64 disassembler: ELF parsing, address translation, abs32/rel32 relocation and ARM instruction codecs - Byte-exact on the noop, elf-x86, elf, elf-arm32 and elf-arm64 fixtures DEX remains a documented follow-up.
Port the full DisassemblerDex apply path: header/map parsing, code-item and item-offset parsing, all 42 reference groups (readers and writers), instruction parsing, and the 16/32-bit index and relative-code codecs. All seven zucchini fixtures now reconstruct byte-exactly with no C++.
- tests/zucchini_pure.rs replays every committed fixture byte-for-byte - docs/zucchini-rust-port.md records scope, the Android preflight gap, and the smallest remaining C++ boundary - silence style-only clippy lints that come from the faithful C++ port
Add the FFI-only validations that upstream Zucchini lacks: - ValidateReferenceBoundaries - ValidateDexWriterWidths - ValidateDexReferenceTargets Reject dex-large-unsafe16.zuc with the native "android executable preflight failed" message. With this, the apply path no longer needs C++/libchrome at all for Android formats.
- patch: malformed trailing reference-delta varints no longer decode; the cursor is left unconsumed on failure (DecodeVarUInt semantics) and apply now honors reference_deltas_done like native Done() - dex: widen opcode table arithmetic so opcode 0xFF (const-method-type) is reachable instead of wrapping to 0 - dex: payload offsets are bounded by remaining instruction units, not the whole code item - dex: checked multiplication for map-array sizing; try_reserve code-item offsets - elf: Thumb2 scanning advances by the decoded instruction size (4 for an unmatched 32-bit instruction) and does not fetch past the region - elf: validate 64-bit program-header p_offset/p_filesz before narrowing - engine: preflight checks the new disassembler size too Adds unit regressions for opcodes/payloads/Thumb2/ELF64 bounds and an integration regression for the trailing reference-delta case.
- add apply_with_cancel without changing apply(); poll between elements, pools, and equivalence units, returning Status::Cancelled - use try_reserve for DEX code-item offsets and ELF section tables - regression test for cancellation
- src/zucchini.rs now delegates to zucchini_pure and maps status/message - build.rs builds the vendored C++/libchrome only with OTADUMP_NATIVE_ZUCCHINI=1 (differential testing); the default build and full test suite no longer require C++ - align preflight/old-file and new-file messages with the native FFI - add a cfg-gated native-vs-pure differential fixture test
…DEX apply Differential testing against C++ Zucchini found a divergence in DEX TypeId string reference reading: group 0 (TypeIdItem::descriptor_idx -> StringId) was reading the index using string_index(2) instead of string_index(4). This truncated 32-bit descriptor string IDs to 16 bits, ignoring corrupted upper bytes.
Builds on the group-0 (TypeId::descriptor_idx) differential fix: every index-based DEX mapper's read width equals its group's declared reference width, so derive it centrally instead of hardcoding per group. This fixes the annotations-directory 32-bit ids (groups 19/22/23), which were still being read as 16-bit and could silently truncate. Adds a unit test showing 16-bit vs 32-bit index reads.
- preflight no longer folds Status::Cancelled into the preflight ApplyError - route extraction's run_op_zucchini through apply_with_cancel using the extraction token and surface typed cancellation for clean unwinding - poll cancellation in the old/new CRC chunk loops, raw-delta application, boundary validation, DEX target validation, and per-reference correction - Disassembler::read is now fallible so allocation failures can surface - OTADUMP_NATIVE_ZUCCHINI now requires the exact value "1" - add cancellation-through-preflight and CRC-chunk regression tests
- patch streams (equivalences, raw deltas, reference deltas, extra targets, extra data, elements) reserve with try_reserve; bound pool/element counts - DEX item, code-reference, annotation and instruction vectors use try_reserve; Disassembler::read is fallible end to end - ELF section/reloc/abs32/rel32/gap vectors and address-translator units use try_reserve instead of infallible clone/with_capacity - preflight boundary vector reserves fallibly; OffsetMapper clone and TargetPool inserts are fallible - add absurd stream/element/pool count regression tests
- decode each DEX code item's instructions once at parse time into a flat, location-sorted list; all 42 groups filter it instead of re-decoding - analyze_element materialises each reference group once and shares it between preflight and apply via GroupRefCache, removing repeated groups x equivalences x items scans - add group_ref_cache_matches_ranged_reads equivalence test - document measured before/after (~30% on dex, ~6% on dex-large)
Add unit tests for zucchini::apply_with_cancel preserving Status::Cancelled and for the non-cancelled malformed-patch path.
Rebased onto the GroupRefCache work. Two divergences from the C++
reference broke real, generator-produced patches (collection-jvm,
kotlin-stdlib, material-icons DEX and x86-64 ELF); the committed
fixtures did not exercise either.
1. DEX instruction filters compared the raw opcode byte. Native
dex::Instruction::opcode is the canonical range-start opcode, so a
family entry like {0x52, 2, c, 14} matches every iget/iput opcode
0x52..=0x5F. The port under-emitted groups 17 (CodeToFieldId), 20
(CodeToMethodId) and 35 (CodeToRelCode16). find_instruction now
carries the start opcode and parse_instructions stores it; the
GroupRefCache therefore caches the full stream too.
2. PruneEquivalencesAndSortBySource calls std::sort, which is not
stable; equivalences sharing a src_offset are reordered by
libstdc++ introsort, and the resulting order determines which
equivalence a projection picks. A stable Rust sort produced different
target pools, so validate_dex_reference_targets rejected valid
patches and ELF reference deltas misaligned. Port libstdc++ std::sort
exactly (introsort + final insertion sort, with the __partial_sort
heap fallback).
Verified byte-exact against the C++ FFI, and reference streams are
identical, for collection-jvm (6,757 equivalences), kotlin-stdlib
(16,793), material-icons (138,880) and x86-64 ELF (1,930). Adds a
compact real-world differential fixture (annotation-jvm 1.8.1 ->
1.10.0, 161 equivalences) that requires both fixes, plus unit tests for
the opcode families and the native sort permutation. Full cargo test
(25+37+13) and cargo clippy --all-targets --all-features -- -D warnings
pass.
…on profiles Build-only optimization prototype on top of delta-ota-support. No behavior changes: all 48 Rust tests and the Python smoke pass. - build.rs: drop xxhash.c (only lz4frame.c uses it; not built) and the dead libchrome debug/metrics/activity stack (17 TUs, ~7.8k upstream lines); keep callback_internal.cc + stringprintf.cc and add a ~90-line libchrome_shim.cc providing the 5 logging/StackTrace symbols the Zucchini closure actually references. Fixes the as-committed link failure (missing thread_id_name_manager/process_handle/command_line TUs). - build.rs: -fvisibility=hidden [-inlines-hidden] for native code, unwind tables dropped for C, --exclude-libs,ALL on final links. - Cargo: drop bsdiff-android; local src/bsdiff.rs replicates parse_bsdf2_header (BSDIFF40/BSDF2 magic, offtin lengths, None/Bz2/Brotli streams) using existing bzip2/brotli deps. - Cargo: add release-cli profile (panic=abort) for the static CLI; release stays unwind so Python wheels raise panics as exceptions. - CI: build --profile release-cli --bin otadump (no more pointless cdylib in CLI artifacts); packaging paths updated. Measured (linux x86_64, rustc 1.98): CLI 2,892,080 B (panic=abort, -131,760 B vs unwind); cdylib 2,754,352 B exporting exactly one symbol (PyInit_otadump); zero XXH symbols in either artifact.
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Remove the in-repo native/lz4 tree (AOSP-vendored liblz4 1.10.0 sources, the custom otadump_lz4_* C shim, build.rs compilation, and the source inventory) in favor of the pinned lz4-sys =1.11.1 crate (published as 1.11.1+lz4-1.10.0, enforced by the lockfile checksum). The crate bundles liblz4 1.10.0 C sources that are byte-identical (SHA-256 verified) to AOSP platform/external/lz4 @ 734e07032602, which Android lz4diff uses, so compressed blocks stay byte-for-byte compatible with AOSP. The crate's bindings omit the block-level destSize/partial APIs, so src/lz4.rs declares LZ4_compress_destSize, LZ4_compress_HC_destSize, LZ4_decompress_safe_partial, and the HC stream lifecycle directly; signatures match lz4.h/lz4hc.h 1.10.0. Static linkage is unchanged: lz4-sys compiles the archive with cc and final artifacts pull only lz4.o/lz4hc.o (lz4frame.o/xxhash.o excluded), with no dynamic liblz4 dependency. LZ4DIFF now also works off Linux x86-64 GNU hosts. The BSD-2-Clause liblz4 notice moves to licenses/LICENSE.lz4 and keeps shipping in release archives and wheels. Verification: - cargo test / cargo test --release: 66 passed (11 lz4 unit tests byte-comparing frozen AOSP fixtures, 2 lz4diff integration tests) - differential harness vs the base C-shim build: byte-identical output for fast mode, HC levels 2-12, zero-padding, and partial destSize across fixture and synthetic inputs - cargo clippy --all-targets --locked -- -D warnings: clean - ABI3 wheel builds, imports, and passes tests/python_smoke.py; the wheel ships licenses/LICENSE.lz4 - ldd/nm -D: no dynamic liblz4 in the CLI binary or the wheel .so - release-cli binary: 2222512 -> 2223584 bytes (+1072) Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
cargo test: 48 passed.cargo checkandcargo check --features pythonpassed.cargo clippy --all-targets --all-features -- -D warningspassed.Public test inputs
These Pong builds provide a reproducible full-OTA and raw-image source pair for the same delta.
Base images plus delta
Pong_B4.1-260818-1726: https://android.googleapis.com/packages/ota-api/package/0fabd802d4714166f47aed0104d2eb4e8fcaad9f.zipDownload the boot, firmware, and all multipart logical image assets from the base release.
Extract them into one source directory and pass that directory with
--source-dir.Full OTA plus delta
Pong_B4.1-260414-1749: https://archive.org/download/nothing-archive/spike0en/fullota/pong/Pong_B4.1-260414-1749.zipPong_B4.1-260818-1726: https://android.googleapis.com/packages/ota-api/package/0fabd802d4714166f47aed0104d2eb4e8fcaad9f.zipExtract the full base OTA first, then use its reconstructed images as the source directory for the delta.