Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head The regression, both ways
Branch, env Master: the same test file with Two controls, each of which had to move the number:
Fixture censusOver the 940 tracked paths whose content is an ELF at The
|
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
import cle
ld = cle.Loader("binaries/tests/x86_64/vmprotect_sample1.vmp.bin", auto_load_libs=False)
obj = ld.main_object
print(obj)
print(sorted({s.version for s in obj.symbols if s.version}))
print(len([s for s in obj.symbols if s.version not in (None, "*local*", "*global*")]))
for name in ("printf", "__gxx_personality_v0", "_ZNSs6appendEPKcm"):
print(name, sorted({s.version for s in obj.symbols if s.name == name}))Before — the RVA cle master 0e77adeAfter — the version tables are read from the image, 112 symbols get a library version back, with this change |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_836 |
__register_dyn builds seven pyelftools section objects out of DT_ tags, each
with an sh_offset taken through AT.from_lva(...).to_rva() -- an
image-relative address, not a file offset. Five of them are then pointed at
self.memory so that address means something: dynsym, versym, and the reloc,
jmprel and relr tables. The verneed and verdef tables were the two
exceptions, so pyelftools kept the ELFFile's file stream and seeked an RVA in
it.
Nothing shows on an ordinary binary, where the segment holding the version
tables is laid out at the image base plus its own file offset and the two
numbers coincide. On the new fixture
binaries/tests/x86_64/vmprotect_sample1.vmp.bin they do not: DT_VERNEED is
0xCE3808 against an image base of 0x400000, so the seek goes to 0x8E3808 in
a 0x2E847B-byte file -- 0x5FB38D past the end -- and the two-byte read of
vn_version comes back empty:
ELFParseError: expected 2, found 0
The table really begins at file offset 0x2E3808, 0x600000 lower. The
exception escapes __register_segments, so the object does not load at all and
nothing downstream runs.
Replace the stream on both sections, which is what the other five already do.
Symbol versions now decode: 112 of the fixture's symbols get a library
version.
The two assignments carry "# type: ignore" because pyright cannot see that
pyelftools reads these attributes back through duck typing, and Typecheck
budgets errors per file: the five older sections predate the budget, so their
identical assignments are grandfathered and two new ones would fail the check.
Diagnosed by @fkil in #383, who also proposed this fix. Wrong since 99379eb
added symbol versioning in 2022, which gave the versym table a memory stream
and these two none.
Fixes #383
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
a3815ec to
19f2125
Compare
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
cle.Loaderrefuses any ELF whose symbol-version tables sit at a virtual address that is notalso a valid file offset. On
binaries/tests/x86_64/vmprotect_sample1.vmp.bin, the sample@fkil attached to #383 on 2023-04-11, which has had no reply since:
Nothing is salvaged: the exception escapes
__register_segments, so the object does not loadand no CFG, symbol table or decompilation follows.
Root cause
__register_dynbuilds seven pyelftools section objects out ofDT_tags, each with ansh_offsettaken throughAT.from_lva(...).to_rva()— an image-relative address, not a fileoffset. Five of them are then pointed at the image so that address means something (line numbers
here are master's,
0e77ade3):readelf_verneedandreadelf_verdefare the two exceptions, so pyelftools keeps theELFFile's file stream and seeks an RVA in it. In the sampleDT_VERNEEDis0xCE3808against an image base of
0x400000, so the seek goes to0x8E3808in a0x2E847B-byte file —0x5FB38Dpast the end — and the two-byte read ofvn_versioncomes back empty. The tablereally begins at file offset
0x2E3808,0x600000lower.Nothing shows on an ordinary binary because the segment holding the version tables is laid out
at the image base plus its own file offset, so the two numbers coincide. Both lines have been
wrong since 99379eb (#324) added symbol versioning in April 2022 — it gave the versym table a
memory stream and these two none.
Fix
Replace the stream on both sections, which is @fkil's second suggestion and what the other five
already do. The four assignments carry
# type: ignorebecause pyelftools declaresstreamandelffilewith types cle deliberately violates here, andTypecheckbudgets pyright errors perfile: the five older sections predate the budget, so their identical assignments are
grandfathered while new ones fail the check. His first suggestion, converting the offset with
to_raw()instead, would leavethese two as the only tables in
__register_dynread out of the file, and he flagged himselfthat it may be wrong when the stream is a memory dump.
Symbol versions now decode: 112 of the sample's symbols get a library version, including
printf@GLIBC_2.2.5,__gxx_personality_v0@CXXABI_1.3and_ZNSs6appendEPKcm@GLIBCXX_3.4.The
verdefhalf gets no fixture with a mismatched offset — see Testing — but it is the samemistake and is fixed with it rather than left half-repaired. It is not untested: on this branch
GNUVerDefSection.iter_versionsis called with aClemorystream and decodes all 23 versionsof
binaries/tests/x86_64/libc.so.6, which cle's own suite already loads.Testing
tests/test_symbol_versions.pyloads the sample, checks thatDT_VERNEED's file offset isinside the file while its RVA is past the end — so the test fails loudly if the fixture is
ever swapped for an ordinary binary — and asserts three versioned symbols by name. On master
it fails with the
ELFParseErrorabove; on this branch it passes.No tracked fixture had the offsets apart: over the 940 tracked paths whose content is an ELF in
angr/binariesat0166109ethere are 578DT_VERNEEDand 40DT_VERDEFentries, and all 618put the table at an RVA that is also its file offset. Hence the new one.
Depends on angr/binaries#230 for the fixture, which
cle/ci.ymlresolves throughangr/ci-settings/actions/binaries-ref.Expected CI, recorded before the push: all 20 checks green — the 18 Actions runs
ci / Build,ci / Lint,ci / Typecheck,ci / Test (0..9),ci / Decompiler Snapshot Testing (0),ci / Publish Unit Tests Results,Test (Pyodide),Test macos-15andTest windows-2022,plus the two commit statuses
docs/readthedocs.org:cleandpre-commit.ci - pr. The previoushead got 19 of those and failed
ci / Typecheckon the four assignments now marked# type: ignore; every other check passed, includingci / Build, so the fixture reachedevery job.
Fixes #383. Validation: #836 (comment)
🤖 Generated with Claude Code
session: sharpen