Pack and unpack words struct cannot describe - #721
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Re-keyed from Measured configuration: a detached worktree of cle at the revision named, this workspace's pinned Python 3.12 environment with archinfo at
What the fourth commit is for, stated plainly. On this fixture the thread lands at Behaviour outside the reported failure, compared against baseline by running the same matrix of sizes, signedness and endness under both revisions: Scope: 14 of the 183 p-code languages pypcode 4.0.1.dev0 exposes report a three-byte word, and Caveats: Hosted CI at head The sections below were measured on earlier heads whose Corpus measurement of the open queue, 2026-08-15 — 1 of 24 alone; a prerequisite, not an independent recoveryCorrecting the record. This change clears almost none of its own class on its own. The open pull-request queue was scored against 733 objects drawn from a sweep's own failing units (35 error classes, 49 architectures, 16 containers), with each repository's current master as the baseline rather than the revisions the sweep pinned. Each object is loaded with Four of the narrow-p-code-architecture changes form a chain. Applied alone, each one clears part of its own class and leaves the rest of it standing on the next change's defect:
Applied together — angr#6793 + cle#721 + cle#717 + archinfo#363 — the One qualification about cle#717, which is included in the combination above: since its rebase onto cle#735 it carries no production change, only regression coverage, so nothing in the movement here is attributable to it. It guards the placement rule the other three depend on rather than supplying it. For this PR specifically: of the 24 measured objects in the The loading claim is untouched: loading is what this PR fixes, and the 23 objects above do load. It is the corpus-recovery reading that has to be qualified — merged alone, this change converts one failure into a completed CFG, and 23 into a different failure. Merge dependency: angr/angr#6932 and #721 are required togetherMeasured 2026-08-28, one variable at a time, on a private corpus; objects are cited by architecture, container and sha256 only. The population is 3,452 objects — PIC-18, AVR and HCS12 firmware images loaded as blobs under a 24-bit SLEIGH language — all failing at A/B over a stratified sample of 107 of those objects, 15 per SLEIGH language, one object per process under the invocation the failures came from —
With both applied the sample recovers 7,868 blocks and 909 functions, against 273 blocks and 89 functions from the 39 objects the lifter change alone gets through, and nothing at all on master. So the lifter change alone moves 64% of the sample one layer down rather than closing it. The next failure is the read side of the same 3-byte word: Neither order is destructive and neither change breaks the other, but the population stays broken until both land, at whichever of the two defects is left. Example objects, each angr/archinfo#364 is not required, and should not be added to this pair. With both changes applied and archinfo left at master — Caveat: the sample is stratified by SLEIGH language rather than uniform, so the per-build counts describe the sample, and the population claim is the class the two changes close between them rather than a projection of block counts onto all 3,452 objects. Note on the cle baseline named above. Corpus evidence, 2026-08-28Load-only measurement over a private corpus, so objects are cited by architecture, container format and sha256 only. Each object is loaded in its own process as 1,810 distinct objects terminate in None of them is fixed on master: 45 of a 45-object sample still terminate there. On this head, |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_721 |
acdccc0 to
1571db8
Compare
1571db8 to
df42c72
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS This came out of assembling the angr/vibr preview snapshot (every green open PR merged together, then each component's full test suite run against the result with an With #765 applied, this PR's |
79eb856 to
cc251dd
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Full word-width report before and after this change. Rows A-E pack and unpack words of widths Before — every width cle at the merge base, d2ecea0After — every width round-trips in both endiannesses, a short access raises with this change, cc251dd |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Interaction with #765, and the assertion it movedThis branch and #765 are each green alone and cannot both be in a rollup: the The test read the DTV pointer back out of the new thread and compared it against
d124129 compares against Verificationmaster + #765 + #730 + #721, all three at the heads below, cle's whole suite: Heads: #765 session: sharpen |
Clemory.unpack() and pack() catch struct.error to tell an access that ran off the end of its backer, which is a KeyError, from an operation that genuinely failed. They made that decision by calling struct.calcsize() on the same format string, which raises the same error when the format is one struct cannot parse. That exception escaped from inside the handler, so neither answer was reported and callers saw a bare struct.error from a line that was only meant to be measuring. ClemoryReadOnlyView.unpack() carries two more copies of that handler, one for the cached backer and one for the general lookup. Work out which exception to report first and raise it afterwards, outside the handler, so a short access no longer carries the struct error as its context either.
struct has integer format characters for 1, 2, 4 and 8 bytes only, so pack_word had no way to write the 3-byte word of a 24-bit architecture, and unpack_word covered wider words with a halving recursion that rejected every size that was not a power of two. ELF TLS setup writes the DTV pointer with pack_word, so opening any ELF on a 24-bit architecture failed with "bad char in struct format" before analysis began. Build a word of any width struct cannot name from its bytes instead. The write checks that the whole word is backed first, because store() writes the bytes that fit before it reports the overrun.
test_tls_24bit_arch assembled its own ELF32 header with struct.pack to reach an architecture whose word is three bytes wide, so it exercised a container shape no toolchain emits. Load tests/avr/isqrt_atmega128.o from angr/binaries instead. Its e_flags name the extended-address AVR variant, which Ghidra maps to avr8:LE:16:extended, a 24-bit language; name that language explicitly because cle picks the 16-bit default for every EM_AVR ELF today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The regression read the relocated DTV pointer out of the new thread and compared it against thread.dtv_offset, which only agrees while the thread object is mapped at 0. That is the loader's placement policy rather than anything this test is about: cle#765 moves an invented object above the image instead of onto the null page, and the assertion then reads 0x504006 against 0x4006 and fails on a thread that was set up correctly. Compare against thread.mapped_base + thread.dtv_offset, which is what the InternalTLSRelocation writes and what the pointer has to hold for the DTV to be reachable. The assertion is unchanged where mapped_base is 0, and it still fails on this branch's merge base with struct.error: bad char in struct format, from cle/memory.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DqcAcuGLrNJViJrpdtFYCS
d124129 to
b5a0326
Compare
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
Opening any ELF on a 24-bit architecture fails before analysis begins. Building the ELF TLS thread for
tests/avr/isqrt_atmega128.ounderavr8:LE:16:extended, whose word is three bytes:ELFThreadManagerwrites the DTV pointer withpack_word, so no thread can be created and the load never completes. The same call fails for any widthstructhas no format character for:Root cause
ClemoryBase.unpack_wordandpack_wordare the only path a word of arbitrary width takes, and both handed the width straight to archinfo'sstruct_fmt. Only widths above 8 were handled at all, by halving:structhas integer format characters for 1, 2, 4 and 8 bytes only, so everything at or below 8 was assumed to have one and 3 does not;pack_wordhad no recursion at all. Separately, both error handlers classified the failure withstruct.calcsize(fmt)inside theexcept struct.errorblock, so a format stringstructcannot parse raised a secondstruct.errorfrom within the handler, replacing the original and chaining the classification failure on as__context__— and a word running off the end of a backer was reported asstruct.errorrather than theKeyErrorthat means "not mapped".Fix
Widths outside
frozenset((1, 2, 4, 8))are read and written as bytes and recombined withint.from_bytes/to_bytes, so any positive size works, signed or unsigned, in either endianness.pack_wordchecks the whole word is backed before writing anything, so a short write leaves memory untouched. Classification moves to_classify_struct_error, which returns the exception for the caller to raise outside the handler and returns the original unchanged whencalcsizeitself fails.Testing
tests/test_tls_resiliency.py::TestTlsResiliency::test_tls_24bit_archloadstests/avr/isqrt_atmega128.ounder the 24-bit p-code language and assertsthread.memory.unpack_word(offset + thread.tcb_offset) == thread.mapped_base + thread.dtv_offset, which is the address the relocation writes;tests/test_unpackword.pyround-trips odd widths and pins theKeyError;tests/test_clemory.pypins the unchainedstruct.error. The fixture is onangr/binariesmaster. The test names the p-code language explicitly rather than relying on autodetection.angr/archinfo#364 is the other half of this: each alone leaves the same objects failing at the other's defect, so the two have to land together, in either order.
Validation: #721 (comment)
sync: angr/archinfo#364
session: sharpen