COFF: Give a section with no file bytes an address of its own - #764
COFF: Give a section with no file bytes an address of its own#764zardus wants to merge 1 commit into
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head The COFF A/B below was measured at Method251 x86 and x86-64 COFF objects, one object per process and one A/B side per The population: 242 objects sampled across every shard of a private COFF corpus Result
The audit checker's The objects the map change reachesOver the 157, 6 objects lose blocks to that check and 903 blocks are refused
The last two keep their block sets bit-identical: the 36 refusals on the first
Test
Full CI
This change is independent of #761 and #724, which touch the same function but not Corpus evidence added 2026-08-26, measured on head The same sweep that motivated #739 counts basic blocks landing inside a region the loader reports as data. The COFF share of that count is this defect: Measured over the affected COFF objects, master against this head:
The count goes to zero, and the change is not merely cosmetic for the metric: giving Across the whole sweep this accounts for 1,683 of 48,658 instances, about 3.5%; the remaining 96% is the ELF note-section case in #739. Reproducing the counts on the pinned toolchain the sweep used gives the same figures as master, so this has not drifted upstream — only this branch moves it. Rebased 2026-08-27. Correcting the SHA this paragraph previously gave: the corpus figures were measured at head Corpus attribution, 2026-08-28, keyed to head A category sweep, deduplicated by object digest, scores 460,618 distinct objects and asks whether a recovered block overlaps a section the loader reports as non-executable data. 14,503 objects carry at least one, 48,658 blocks in total. This mechanism owns 13 of those objects and 1,683 of the blocks, 3.5% of the category — small in objects, disproportionate in blocks, because a single shadowed Read out of the COFF section table directly rather than through CLE, all ten of the sampled objects agree: the overlapping section is Measured on 69 stratified objects of the category, of which 10 are COFF:
63 of the 69 go to exactly zero; the six survivors are Xtensa objects and are a p-code read-ahead defect, angr/angr#6816, not this. The COFF objects gain recovery rather than losing it — two of them add 93 and 83 blocks and 7 and 12 functions once Nothing upstream has fixed this: cle master scores the same 1,134 on those ten objects as the sweep's own revisions did. The flag condition and the bound, added 2026-09-02Two mechanisms. A section with no bytes in the file gets zero-filled space of its
Acceptance, measured rather than arguedTwo arms, each printing
The 230 are distinct by The same harness, unchanged, on the three fixtures, so that the empty diff is an
What the flag condition costsNothing found. Every tracked regular file in
Public availability and tracked-ness are separate axes here. Four limits of that enumeration, none of which changes the answer: the Nix store Correction, 2026-09-06. This section first reported 731,580 objects, Objects that stop failingFour distinct MSVC objects, present as five copies, between 1.16 MB and Merge orderBytes mapped by
The second is the tracked 108-byte Both #804 and #806 conflict with this branch in |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_764 |
7c1c288 to
45b1e78
Compare
45b1e78 to
09fcf86
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS The three fixtures this change turns on, before and after. the reproducer, run once per objectimport logging, os, resource, sys
logging.getLogger("cle").setLevel(logging.CRITICAL)
import cle
# One object per process, so the peak RSS below belongs to this object and is not a
# high-water mark left by an earlier load.
name = sys.argv[1]
path = os.path.join(os.environ["BINARIES"], "tests", "x86", name)
before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024
ld = cle.Loader(path, auto_load_libs=False)
obj = ld.main_object
after = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024
bss = next(s for s in obj.sections if s.name == ".bss")
text = next(s for s in obj.sections if s.name == ".text")
image = sum(len(backer) for _, backer in obj.memory.backers())
flag = "set" if bss.only_contains_uninitialized_data else "clear"
print(f"{name} ({os.path.getsize(path)} bytes on disk)")
print(f" .bss SizeOfRawData {bss.memsize:#x} uninitialized-data flag {flag}")
print(f" .text at {text.vaddr:#x} .bss placed at {bss.vaddr:#x}")
print(f" image {image} bytes, peak RSS {before} MB -> {after} MB (+{after - before} MB)")Before — every cle master at
|
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS What this changes on a real corpus, measured. This is a memory-layout fix, not a Sample. 12,000 objects drawn uniformly at random, from a seeded permutation, Method. All 468 are loaded with the catalogue's declared recipe against What changes. 398 of the 432 loaded objects (92%) map a section What does not change. Recovered functions and CFG nodes are identical on Note on a false result I nearly published. An earlier run showed four objects Overlap with the other open COFF changes. #724, #775 and #761 each touch The corpus is not redistributable, so its objects are described by architecture, session: sharpen |
09fcf86 to
51c1f24
Compare
The backend maps the object at its own file offsets and gives each section vaddr = PointerToRawData. A section holding no bytes in the file states that field as 0 -- that is what .bss is -- while SizeOfRawData still states its length, so it lands on the file header and, once it is longer than the header and section table, over the sections that follow. .text begins at file offset 0x104 in a six-section mingw object, so a 0x1300-byte .bss covers its first 0x11fc bytes. find_section_containing() then answers .bss for real code, and CFGFast._generate_cfgnode drops any block whose section is not executable, so those functions are never recovered; uninitialized data reads as the file header rather than zeros, and every .bss symbol is given an address inside the code. A section that states PointerToRawData 0 and marks itself IMAGE_SCN_CNT_UNINITIALIZED_DATA now gets zero-filled space of its own past the image, at the alignment its IMAGE_SCN_ALIGN_* states. Relocation patch offsets and symbol addresses read the same layout, so they follow it. The flag is the condition rather than the zero pointer alone, because the zero pointer alone is what a file controls: a 120-byte object can state PointerToRawData 0 with SizeOfRawData 0x4000000 on a section marked code, and zero-filling that is 64 MiB of allocation bought with one header field. Across 1,480 sections with no bytes in the file, in 64 distinct shapes, every one sets the flag, so requiring it costs nothing real. A section without it keeps the address its header states, which is what master does with it. MAX_IMAGE_SIZE bounds what the flag still admits. SizeOfRawData is 32 bits wide and a section that does set the flag can still state close to 4 GiB, so past 0x10000000 the section is placed and reports its stated size but no zero fill is allocated for it and a warning names it -- the outcome pe.py reaches through max_virtual_address. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
51c1f24 to
9f84165
Compare
Coff._add_relocs took the patch address as section.PointerToRawData plus reloc.VirtualAddress and registered a relocation there without checking it. Neither bound was tested, and the two fail differently. A field past the end of the file crashes. The backend maps the object as one backer covering the file, so CoffRelocationDIR32.value asks Clemory for four bytes at an address nothing maps, and cle.Loader(..., perform_relocations=True) raises KeyError out of Clemory.load. A field merely past the end of its own section does not crash, and that is the worse half. Every offset in the file is mapped, so the store lands wherever the arithmetic points -- another section's raw data, the relocation table, the symbol table -- and the load returns normally with those bytes rewritten. Check both bounds where the relocation is registered rather than in relocate(). A relocation that cannot be applied should not reach self.relocs at all: it is handed to the symbol resolver, it can produce an extern symbol for a field that will never be written, and it is visible to every consumer that iterates an object's relocations. It is also where the PE backend drops a section whose raw data the file does not hold. The field's width comes from struct.calcsize on the relocation class's PACK_FORMAT -- four bytes normally, eight for ADDR64, two for SECTION -- so a four-byte field starting on the last byte of a section is out of bounds, which a bound on the start offset alone would miss. PACK_FORMAT is declared on CoffRelocation rather than on Relocation, so RELOC_CLASSES is annotated with the class it actually holds. This leaves the section mapping loop alone. Bounding a section's raw data by the size of the file is #806; the two compose, because _add_relocs walks self._coff.sections itself and would still register the relocations of a section that loop has skipped. Two details keep this bound correct against the other open COFF branches, and change nothing on this one. The section comes out of self._coff.sections by index rather than off the loop variable. Both name the same object here, by the definition of enumerate. #764 rewrites this loop to walk indices and drops the variable, and the two branches merge with no textual conflict, so with both applied and the loop variable read _add_relocs raises NameError on the first relocation of a supported type. Of the five COFF objects angr/binaries tracks that this backend loads, four carry such a relocation and stop loading; the fifth has none. #804 is stacked on #764 and carries the same rewrite. The file-size half of the bound is taken against self._image_vmem, the bytes the backend maps, rather than against self._data. Here the two are the same object: _image_vmem is assigned from _data in __init__, never rebound, and cle defines no subclass of Coff. #804 places a section whose file offset does not satisfy its alignment past the end of the file and extends the image to cover it, so a relocation into a moved section is past len(self._data) and inside the image, and bounding on the file would skip it. With both applied and the file used, x86/fauxware.obj keeps 177 of its 225 relocations and x86_64/fauxware.obj 66 of 126, and the test below asserting 225 fails.
cle's tests on master now load tests/aarch64/langdetect_go.macho and tests/aarch64/relocatable_object.macho, which #193 and #224 added after this branch was cut. angr/cle#764 and angr/cle#804 name this pull request in their sync: lines, so CI checks this branch out instead of master; once either is rebased onto current cle master those two files would be missing and the macOS job would fail. Merging master in supplies them and leaves this branch's own three objects and build script untouched.
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
A COFF section with no bytes in the file is mapped over the code. On
binaries/tests/x86/coff_bss.obj, a mingw object whose.bssis 0x1000 long:.bssstarts on the file header and runs over the whole of.text. CFGFast dropsevery block whose section is not executable, so the object's code is unreachable;
uninitialized data reads back as the COFF header (
4c 01isMachineIMAGE_FILE_MACHINE_I386,04 00the section count); and_buffer, a.bsssymbol, gets an address inside the header rather than in its own storage.
It also loses whole objects. Four MSVC objects, in five copies, between 1.16 MB
and 4.86 MB do not load at all on master:
The
.bssat the image base stretches the main object's span over the address theloader then places the extern object at. All four are in a working tree that is not
public, and no public object reproduces it: all 1,097 objects with this shape that
come from publicly downloadable packages load on master. So the reproducer cannot
be published.
Root cause
The backend maps the object at its file offsets and gives every section
vaddr = PointerToRawData:A section with no file bytes states that offset as 0, while still stating its
length elsewhere. The fixture's header says exactly that:
so
.bssis placed at 0 and, being longer than the header and section table,extends over everything that follows.
Fix
A section marked
IMAGE_SCN_CNT_UNINITIALIZED_DATAgets zero-filled space of itsown past the image, at the alignment its
IMAGE_SCN_ALIGN_*states. Relocationpatch offsets and symbol addresses come from a per-section address list rather
than from
PointerToRawData, so they follow the new layout.The flag, not the zero pointer, is the condition, because the zero pointer alone
is something a file controls: a 120-byte object can state
PointerToRawData0with
SizeOfRawData0x4000000on a section marked code, and honouring that is64 MiB of zero fill bought with one header field. The format already says which
sections have no bytes in the file, and
CoffSectionalready exposes it asonly_contains_uninitialized_data.A section without the flag is not refused. It keeps the address its header
states, which is exactly what master does with it, and cle stops inventing storage
for a shape nothing in the survey below produced. Nothing stops loading.
That is measured, not assumed. Across
angr/binaries, four dataset trees andevery
.o,.obj,.lib,.aand.sysounder a 365,293-file Nix store --731,573 COFF objects, 730,821 of them archive members -- 1,480 sections state
PointerToRawData0 with a non-zero size, in 64 distinct shapes, and every onesets the flag. 1,097 of those sections, one per object, are in packages anyone can download, the
largest a 16,116,128-byte
.bssin Go 1.26.7'srace_windows.syso.MAX_IMAGE_SIZE,0x10000000, is the second line, for a section that does setthe flag and still states an absurd size -- the field is 32 bits wide, so it can
ask for close to 4 GiB. Past the bound the section is placed and reports its
stated size, but no zero fill is allocated and a warning names it, which is what
_get_memory_mapped_imagedoes pastmax_virtual_addressincle/backends/pe/pe.py. That constant is0x100000000; this one is smallerbecause PE bounds an address whose data the file still has to hold, while nothing
in a file bounds this.
A relocation that resolves into a section left unbacked that way raises
KeyError. That is out of scope here: the ceiling fires on 0 of the 731,573objects surveyed above, whose largest image is 16,679,072 bytes against the
ceiling's 268,435,456.
Testing
Three tests in
tests/test_coff.py, one per path.coff_bss_no_flag.objcarriesthe fix's own condition: its
.bssstates0x4000000with the flag clear, andthe test asserts it keeps
mapped_baseand that the image is the file's 120bytes.
coff_huge_bss.objstates0x20000000with the flag set and asserts theimage stays at 580 bytes.
coff_bss.objasserts the ordinary case still gets itsspace. Remove only the flag condition and the first fails at
4194432 == 4194304;remove only the bound and the second fails at
536871520 == 580.All three fixtures are in angr/binaries#184, which every job here resolves, so it
merges first. It conflicts with binaries 223 in one place, the builder's module
docstring, where 223 rewrites the paragraph into two; land 223 first and rebase.
Merge this and #806 in either order, then #804. Both conflict with this branch in
cle/backends/coff.pyandtests/test_coff.py, textually and already at51c1f241. #804 carries a copy of this commit and adds a second unboundedallocation through its realignment path, which #806's file bound closes.
Validation: #764 (comment)
sync: angr/binaries#184
session: sharpen