Skip to content

Stop indexing PE and ELF header tables past their declared length - #732

Open
zardus wants to merge 2 commits into
masterfrom
feature/fix-cle-header-indices
Open

Stop indexing PE and ELF header tables past their declared length#732
zardus wants to merge 2 commits into
masterfrom
feature/fix-cle-header-indices

Conversation

@zardus

@zardus zardus commented Aug 10, 2026

Copy link
Copy Markdown
Member

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

cle.Loader aborts with IndexError on two numbers a container's own header controls, and in both cases the whole object is lost over one field that could have been ignored.

tests/x86_64/large_common.o, built -mcmodel=medium, places big_buffer at SHN_X86_64_LCOMMON (0xFF02):

  File "cle/backends/elf/symbol.py", line 36, in __init__
    value += owner.sections[sec_ndx].remap_offset
  IndexError: list index out of range

tests/x86_64/efi_short_data_directory.efi declares 6 data directories rather than 16:

  File "cle/backends/pe/pe.py", line 486, in _meta_dd
    dd = self._pe.OPTIONAL_HEADER.DATA_DIRECTORY[idx]
  IndexError: list index out of range

Root cause

ELFSymbol.__init__ treated any integer st_shndx as a section header table index:

if owner.is_relocatable and isinstance(sec_ndx, int):
    value += owner.sections[sec_ndx].remap_offset

An st_shndx from SHN_LORESERVE up is a reserved tag, not an index, but pyelftools decodes only SHN_UNDEF, SHN_ABS and SHN_COMMON to strings and hands the rest back as plain ints, so 0xFF02 subscripts a list of a dozen sections.

PE._meta_dd has the same "the index is always valid" assumption in self._pe.OPTIONAL_HEADER.DATA_DIRECTORY[idx], over a list pefile sizes from NumberOfRvaAndSizes. The optional header may declare fewer than the 16 the format defines, and EFI-stub images commonly declare 6, so the IAT and .NET descriptor lookups index past the end.

Fix

Each backend checks its table before indexing it. A reserved st_shndx names no section, so its symbol gets none and no remap offset while staying an export, exactly as a SHN_COMMON symbol does; the reserved tags are recognised as "not a section index" and not decoded into individual meanings. A directory past the end of a short DATA_DIRECTORY is absent, the way a zero VirtualAddress already is, and is_dotnet goes through _meta_dd instead of indexing directly. That also makes is_dotnet require the nonzero Size the helper has always required, which is how _meta_com_descriptor already reads directory 14, so cle stops answering "this image is .NET" and "this image has no CLR header region" about the same file. CFGFast is the only consumer; it reads the flag to choose between the .NET and the native scanning defaults.

large_common.o: symbol='big_buffer' section=None is_common=False is_export=True size=2097152
decompiler/gzip.o: symbol='ofname' section=None is_common=True is_export=True size=1024
efi_short_data_directory.efi: loaded PE sections=['.text', '.data'] entry=0x1000 is_dotnet=False

Testing

tests/test_elf_symbols.py::test_large_common_symbol asserts symbol.section is None with symbol.is_export guarding the semantics, ::test_common_symbol pins the SHN_COMMON control on tests/x86_64/decompiler/gzip.o, and tests/test_pe.py::TestPEBackend::test_short_data_directory loads the EFI image. Both regressions load a fixture with the header shape they exercise rather than rewriting a field in a copy of one that has a different shape; both fixtures are real toolchain output.

sync: angr/binaries#205

Validation: #732 (comment)

session: sharpen

@zardus

zardus commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head cf834d8543b9fde456143b7beaf52403a35730ae against baseline eac0e5540516b9199dd6a91933e80dc774ea3eac. Python 3.12.13, pefile 2024.8.26, pyelftools 0.33.

Reproducers, both through cle.Loader alone:

  • PE: load an image whose NumberOfRvaAndSizes is 6 rather than 16. tests/test_pe.py::TestPEBackend::test_short_data_directory loads binaries/tests/x86_64/efi_short_data_directory.efi (sha256 dd3d110220f9ab53653be746166f9b5a051b2a93c0998788843e8373a8cc37ab, added by Add a short-data-directory PE and a large-common ELF object binaries#205). On the baseline it raises IndexError: list index out of range at cle/backends/pe/pe.py:513, in _meta_dd for IMAGE_DIRECTORY_ENTRY_IAT (index 12). Guarding only _meta_dd moves the failure to the is_dotnet lookup (index 14), which is why both now go through the guard.

  • ELF: load a relocatable object carrying a symbol tagged SHN_X86_64_LCOMMON (0xff02), which is what gcc emits for a common symbol outside the small code model. tests/test_elf_symbols.py::test_large_common_symbol loads binaries/tests/x86_64/large_common.o (sha256 6d62b5250b2bb62896e1d06fe89c1510bd87881839fa6237c1ef2df7aebc0fce, built with -mcmodel=medium, added by the same fixture branch) and reads big_buffer. On the baseline it raises IndexError at cle/backends/elf/symbol.py:36, via cle/backends/regions.py:52.

  • Regression: pytest tests/test_elf_symbols.py tests/test_pe.py — 18 passed on head; with only cle/backends/elf/symbol.py and cle/backends/pe/pe.py reverted to the merge base, test_large_common_symbol and test_short_data_directory fail with the IndexErrors above and the other 16 pass.

  • Full suite: pytest tests/ — 247 passed, 9 skipped on head; the baseline is 244 passed, 9 skipped. The skips are pre-existing unittest.skip("TODO") markers in tests/test_macho_bindinghelper.py.

  • Lint/type: pylint and pyright against the merge base — cle/backends/elf/symbol.py and cle/backends/pe/pe.py both 10.00 unchanged, with pyright badness slightly lower on each; tests/test_elf_symbols.py new at 10.00 and badness 0.

  • Hooks: pre-commit run --all-files — 22 hooks pass, 2 skip for want of matching files, tree unchanged.

  • Hosted CI: see the CI disposition at the end of this record. The decompiler corpus snapshot for this PR was identical to master's at c857b4edangr/dec-snapshots reported zero changed files — and the production diff has not moved since.

Behavior deltas beyond the crash, both checked:

  • ELFSymbol.section is now None for a reserved st_shndx instead of the raw tag, which was never a valid index into sections. is_export is deliberately unchanged: the baseline made section truthy and the symbol an export, and it still is, which matches how CLE already exports SHN_COMMON. Confirmed on a non-relocatable object, where the baseline does not crash — retagging an exported function in binaries/tests/x86_64/cpp_qualified_symbols.so gives section=65282, is_export=True on the baseline and section=None, is_export=True on head.
  • PE.is_dotnet now goes through _meta_dd, so it also requires a nonzero Size. That is the test _meta_com_descriptor has applied to directory 14 since PE: Parse more metadata. #666, and it is what the two readings of that directory disagreed about; the only consumer anywhere in the ecosystem is CFGFast.__init__ in angr, which reads the flag to default function_prologues and force_smart_scan. Of the 96 PE images under binaries/tests at angr/binaries 879075612e2d571d0b8156515b410ed59fecfbea, 95 declare 16 data directories and the one that does not is the fixture this branch adds — but none of the 96 carries a nonzero IMAGE_DIRECTORY_ENTRY_COM_DESCRIPTOR.VirtualAddress at all, so that population holds no case that could tell the two spellings apart and its zero disagreements say nothing about them. The positive control is the one real .NET image in flight: tests/x86_64/readytorun_linux_x64.dll on the open Add ARM64, ARMNT and ReadyToRun PE fixtures binaries#183, sha256 fe42accda919bccda1b4b7cfbf15020fc9e1252bef16b9dab4ded5106ba9066e, declares that directory at VirtualAddress=0x10530, Size=0x48 — 72 bytes, sizeof(IMAGE_COR20_HEADER) — so both spellings answer True on it. The only shape they can disagree on is a nonzero VirtualAddress with a zero Size, an image declaring a CLR header of no length. pefile has no convention to inherit here: its parse_data_directories table covers ten directories and 14 is not among them, so it never parses a COM descriptor at all. Measured over a private corpus of 24,717 PE images: 2,269 carry a COM descriptor, and every one of them declares Size 72 — the minimum and the maximum are both sizeof(IMAGE_COR20_HEADER) — so the two spellings both answer True 2,269 times and disagree zero times. A CLR header directory's size is a constant, which is why the shape they differ on does not arise.
  • Neither hunk touches any existing fixture. Of the 834 ELF objects under binaries/tests at angr/binaries 879075612e2d571d0b8156515b410ed59fecfbea, exactly one carries a symbol whose st_shndx is an int at or above SHN_LORESERVEtests/x86_64/large_common.o, which this branch adds — and below that value both branches compute exactly what they computed before. That is why both regressions need new fixtures rather than an object already in the repository.
  • Blast radius, same private corpus: 20 of the 24,717 declare fewer than sixteen data directories — NumberOfRvaAndSizes 6 in all twenty, every one an EFI subsystem image. At the baseline 18 of them raise IndexError at cle/backends/pe/pe.py:513 in _meta_dd, reached from _meta_iat, and the other 2 fail earlier in archinfo with ArchNotFound for LoongArch64. At this head all 18 load — 9 AMD64, 6 AArch64, 2 ARMEL, 1 X86, each os='uefi' and is_dotnet=False — and the 2 LoongArch64 still fail on ArchNotFound, a separate gap this change does not touch. The images are not published here; the public reproducer for the same header shape is the OpenWrt EFI-stub kernel named below.

pyelftools>=0.29, CLE's floor, already defines SHN_INDICES.SHN_LORESERVE, so no dependency change.

Caveats: two jobs needed a rerun for reasons outside this change, and both pass on the rerun. Test windows-2022 failed at collection in pyvex/native.py::_parse_ffi_str, where os.replace onto the shared pyvex_ffi_parser_cache path raises WinError 5 when a concurrent job on the same runner holds the destination; cle PR #727 failed identically in the same window. ci / Test (5) failed with a SIGSEGV in angr/tests/engines/test_java.py::TestJava::test_jni_array_operations, inside the JVM.

The images that first showed this came from a corpus sweep and are not published here. The PE side is reproducible on public material — OpenWrt 24.10.2 EFI-stub release kernels for x86-64, aarch64 and loongarch64 all declare 6 data directories; the x86-64 one is sha256 179d7164f8f0d163c5b11fe34043f25016dfbdcad01fd2b9a2f42f57ef0e26b7. The ELF side came from x86-64 Fortran static libraries whose large common blocks carry 0xff02, and from an illumos object carrying the OS-specific 0xff3f. Both regressions here rebuild the same header state from fixtures already on angr/binaries master.

Re-measured at c857b4e with angr/binaries at bcd112a (the head of angr/binaries#205), CPython 3.12.13:

  • Focused: python -m pytest tests/test_elf_symbols.py tests/test_pe.py — 17 passed
  • Regression: the same command with cle/backends/elf/symbol.py and cle/backends/pe/pe.py reverted to the merge base — 2 failed, 15 passed. test_large_common_symbol fails with IndexError: list index out of range at cle/backends/regions.py:52 and test_short_data_directory with the same error at cle/backends/pe/pe.py:486

Not re-run at this head: the full suite, the lint and type comparison, the pre-commit hooks and the fixture surveys above. Those figures are the ones measured at 27a2c9de, and the production diff they cover has not moved.

Re-keyed 2026-08-28. The opening line named 27a2c9ded36b62cc12dbef738020ea6497ee1a4b until now; the branch is at c857b4ed93bfc08a560b50a0fae72338241589ae on baseline d2ecea068794d20b1f14d90eecc1bc4bc4cfa431. This is not a pure rebase: git range-diff b58ea02a446106647cdaae32bdf91b7062404cc1..27a2c9ded36b62cc12dbef738020ea6497ee1a4b d2ecea068794d20b1f14d90eecc1bc4bc4cfa431..c857b4ed93bfc08a560b50a0fae72338241589ae reports the commit changed, and the change is in the tests — tests/test_elf_symbols.py goes from 67 to 45 added lines and tests/test_pe.py from 62 to 14, the struct.pack header rewriting replaced by the two fixtures. Both production hunks (cle/backends/elf/symbol.py, cle/backends/pe/pe.py, 18 and 17 lines) are byte-identical across the move. That is why the reproducer, the regression and the focused counts were re-measured at c857b4e in the refresh above, and why the full suite, lint, hooks and fixture surveys were not: those cover the production diff, which has not moved. Master also touched tests/test_pe.py between the two baselines, which the re-measured focused run at c857b4e already accounts for.


Re-keyed 2026-08-29, second time. The opening line named c857b4ed93bfc08a560b50a0fae72338241589ae
against baseline d2ecea068794d20b1f14d90eecc1bc4bc4cfa431 until now. The branch was replayed onto cle master
eac0e5540516b9199dd6a91933e80dc774ea3eac and is at cf834d8543b9fde456143b7beaf52403a35730ae. git range-diff d2ecea06..b9bc8de4 eac0e554..cf834d85 reports both commits =
and the two diffs against their merge bases differ only in hunk offsets and blob ids, so this is the same patch on
a new base. Everything above was re-measured at the new head rather than carried over: the focused run, the full
suite, the pre-commit hooks, the merge-base lint and type comparison, and both fixture surveys. Only the block
headed "Re-measured at c857b4e" is historical, and its cle/backends/pe/pe.py:486 was the frame at the old
baseline; at eac0e554 the same statement is line 513.

CI disposition.

The previous head b9bc8de4 was red on three tests across two shards of run 33270379266, and none of the three
is this diff.

The two test_uefi failures were the branch being three commits behind cle master. ci / Test (8) failed
tests/simos/test_uefi.py::TestUefi::test_ia32_image_uses_the_microsoft_convention and
::test_aarch64_image_uses_the_architecture_default on assert isinstance(project.simos, SimUefi). Both load a
Terse Executable, which this diff does not touch. cle master eac0e554 ("PE: read the loading environment from
the optional header") sets TE.os = "uefi"; before it, TE.__init__ never assigns os, so Backend.__init__'s
None survives, and angr's os_mapping is a defaultdict(lambda: SimOS) — an os of None yields a plain
SimOS and raises nothing. cle's CI checks the pull request out at its own head rather than merged with master
(resolve_refs.py is passed refs/pull/732/merge and fetches refs/pull/732/head; the Build log records
HEAD is now at b9bc8de), so the shards ran a cle that predates that commit. Measured directly, angr at
e780d8501fd956f3776677663f8877d863eeab43 running tests/simos/test_uefi.py against four cle trees, each
confirmed by printing cle.__file__ and angr.__file__ from the test process:

cle pytest tests/simos/test_uefi.py
d2ecea06, the old merge base, none of this branch's commits 3 failed, 1 passed
b9bc8de4, the old head 3 failed, 1 passed
eac0e554, cle master 4 passed
cf834d85, this head 4 passed

The old merge base fails identically to the old head, which is what rules the diff out. The third row of each
failing run is test_riscv64_image_does_not_become_windows, which CI skipped for want of the fixture and which
fails locally on the old head with KeyError: 'Win32' raised from angr/simos/windows.py — the failure
eac0e554 exists to prevent.

The third failure was the pinned fixture branch. ci / Test (1) failed
tests/analyses/decompiler/test_block_simplifier.py::TestBlockSimplifier::test_a_long_stack_pointer_chain_reaches_a_fixed_point
with Exception: Not a valid binary file: .../binaries/tests/i386/deep_sp_chain, which is angr/project.py's
os.path.exists branch — the file was not there and cle was never reached. That fixture landed on
angr/binaries master at d4ffa2f (angr/binaries#215, merged 19:12:58Z) sixteen minutes before this run
resolved refs/pull/205/head, and a sync: reference is checked out instead of master, so the pin lost it.
Repaired on the sibling rather than here: angr/binaries#205 replayed onto master, range-diff reports both its
commits unchanged and its diff against the merge base is byte-identical, head now 879075612e2d571d0b8156515b410ed59fecfbea. That tip carries
tests/i386/deep_sp_chain and tests/riscv64/uefi/HighMemDxe.efi as well as this branch's two objects.

Typecheck. ci / Typecheck was red on cle/backends/pe/pe.py at the head before last, on badness
(10*errors + warnings)/lines, and the branch had added no diagnostic: its pyright message multiset was byte-identical to master's 52, over 26 fewer lines. The 39
Cannot access attribute "VirtualAddress"/"Size" for class "Structure" errors in that count come from
PE._meta_dd, whose -> pefile.Structure | None annotation discarded the data-directory entry type the pefile
stub already knows. The second commit drops that annotation and records the reason in the docstring: inference
gives the entry type where the stub is installed and an unknown type where it is not, so the call sites
type-check, none of them moves, and nothing changes at run time. At this head, with the base now equal to
master's tip, run-ci-diff-checks.py reports no lint or type regression on any of the four changed files:
cle/backends/pe/pe.py badness 0.6280193236714976 -> 0.3042433947157726, cle/backends/elf/symbol.py
0.17857142857142858 -> 0.15151515151515152, tests/test_pe.py 0.12121212121212122 -> 0.11627906976744186,
tests/test_elf_symbols.py 0.0 -> 0.0, and pylint 10.00 for all four. Those absolute numbers are this
workspace's; the CI image installs types-pefile and sortedcontainers-stubs, which this environment lacks, so
the verdicts match and the figures do not. tests/test_elf_symbols.py tests/test_pe.py: 18 passed, with cle
imported from the branch tree. The three badness figures this record used to carry for that job
(0.4276315789473684 -> 0.10638297872340426 against a base of 0.41867954911433175) were measured against
master's tip while the branch was three commits behind it, and do not describe this head; they are replaced
rather than restored.

Platform legs. On earlier heads Test windows-2022 failed on the known pyvex FFI parser-cache race —
PermissionError: [WinError 5] in pyvex/native.py _parse_ffi_str, where os.replace publishes the cffi
cache and Windows refuses to overwrite a file another xdist worker holds open, blocked on angr/pyvex#569 — and
Test macos-15 was cancelled rather than failed, killed in the same second by a matrix with no
fail-fast: false. If either recurs at this head it is that and not this diff; confirm at the WinError 5
frame rather than by the module 'pyvex' has no attribute 'vex_ffi' cascade below it.

Hosted checks at this head are reported when the run reaches a terminal state.

Neighbouring pull requests. Two open cle pull requests touch these lines. cle#756 carries a commit with the same
subject and the same _meta_dd annotation change; the two differ only in the docstring's opening line, so
git merge-tree --write-tree cf834d85 <756 head> conflicts on that one line while each merges cleanly with
master today. Whichever lands second takes a one-line rebase and the duplicate commit falls out. cle#757 adds its own index >= len(OPTIONAL_HEADER.DATA_DIRECTORY) check for directory 14 in a helper of its
own — it merges cleanly with master and with this branch, but landing both leaves two copies of one guard, and the
second to land should go through _meta_dd. Neither blocks this change:
git merge-tree --write-tree origin/master cf834d85 is clean.

What did not run. The workspace-wide gate needs a nix develop entry, which reinstalls the shared editable
packages and rewrites the native libraries in place; 79 processes on this host had libpyvex.so mapped when this
was measured, so the runs above were done in isolated trees instead — cle and angr extracted with git archive
and imported through PYTHONPATH, each run printing its own cle.__file__ and angr.__file__ — and no suite
outside cle's own and angr/tests/simos/test_uefi.py was executed locally. The hosted run is the authority for
the rest.

Re-keyed 2026-09-04. Head is now 43c897a6c363151643ab7808fe9c275c404397db, on cle master 3812052df2ad284cd16684fb7b7eb66e8d14dc6d. The branch was rebased to clear a merge conflict. The conflict was one hunk in cle/backends/pe/pe.py: #808 put self.gopclntab = register_gopclntab_symbols(self) immediately above the is_dotnet assignment this branch rewrites, so the two are adjacent and both sides are kept whole. git range-diff marks the second commit unchanged and the first altered only in its hunk header and context. The lines this branch adds and removes against its merge base are byte-identical before and after the rebase, so every figure above describes the same patch on a new base.

The sync: angr/binaries#205 sibling moved too, from 8545e3e to 88bd6404410910d9da33890f14b47aa213981f63, which is that branch with angr/binaries master merged in. cle's CI checks a referenced sibling out at its branch tip rather than merged with master, and master had since gained two Mach-O fixtures that cle master's own tests/test_macho.py now needs, so without that merge this run would have failed for a reason outside this change.

Hosted CI at this head, read live 2026-09-04: 20 terminal checks, every one success, nothing outside success.

@zardus
zardus force-pushed the feature/fix-cle-header-indices branch from 6cfc4d2 to 27a2c9d Compare August 10, 2026 00:48
@angr-bot

Copy link
Copy Markdown
Member

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_732

@zardus

zardus commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Full load report for tests/x86_64/large_common.o and tests/x86_64/efi_short_data_directory.efi from angr/binaries#205, with the ordinary SHN_COMMON symbol of tests/x86_64/decompiler/gzip.o as a control, before and after this change. Each is loaded with cle.Loader(path, auto_load_libs=False); the ELF lines print the named symbol's section, common flag, export flag and size.

Before — both objects abort mid-load with IndexError, one from the section list and one from the data
directory:

cle at the merge base, eac0e55
cle: <cle at the merge base>/cle/__init__.py

-- ELF: large_common.o, built -mcmodel=medium, so big_buffer is SHN_X86_64_LCOMMON (0xFF02)
x86_64/large_common.o: RAISED builtins.IndexError: list index out of range
    | Traceback (most recent call last):
    |   File "<probe>", line 31, in elf
    |     ld = cle.Loader(path, auto_load_libs=False)
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/loader.py", line 187, in __init__
    |     self.initial_load_objects = self._internal_load(
    |                                 ^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/loader.py", line 805, in _internal_load
    |     obj = self._load_object_isolated(main_spec)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/loader.py", line 1017, in _load_object_isolated
    |     result = backend_cls(binary, binary_stream, is_main_bin=self._main_object is None, loader=self, **options)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/backends/elf/elf.py", line 204, in __init__
    |     self.__register_sections()
    |   File "<cle at the merge base>/cle/backends/elf/elf.py", line 1473, in __register_sections
    |     self.__register_relocs(sec_readelf, dynsym=None)
    |   File "<cle at the merge base>/cle/backends/elf/elf.py", line 1351, in __register_relocs
    |     symbol = self.get_symbol(readelf_reloc.entry.r_info_sym, symtab)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/backends/elf/elf.py", line 456, in get_symbol
    |     symbol = ELFSymbol(self, re_sym)
    |              ^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/backends/elf/symbol.py", line 36, in __init__
    |     value += owner.sections[sec_ndx].remap_offset
    |              ~~~~~~~~~~~~~~^^^^^^^^^
    |   File "<cle at the merge base>/cle/backends/regions.py", line 52, in __getitem__
    |     return self._list[idx]
    |            ~~~~~~~~~~^^^^^
    | IndexError: list index out of range

-- ELF control: an ordinary SHN_COMMON symbol
x86_64/decompiler/gzip.o: loaded ELF relocatable=True symbol='ofname' st_shndx-derived section=None is_common=True is_export=True size=1024

-- PE: an EFI stub whose NumberOfRvaAndSizes is 6, not the 16 the format defines
x86_64/efi_short_data_directory.efi: RAISED builtins.IndexError: list index out of range
    | Traceback (most recent call last):
    |   File "<probe>", line 47, in pe
    |     ld = cle.Loader(path, auto_load_libs=False)
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/loader.py", line 187, in __init__
    |     self.initial_load_objects = self._internal_load(
    |                                 ^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/loader.py", line 805, in _internal_load
    |     obj = self._load_object_isolated(main_spec)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/loader.py", line 1017, in _load_object_isolated
    |     result = backend_cls(binary, binary_stream, is_main_bin=self._main_object is None, loader=self, **options)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/backends/pe/pe.py", line 168, in __init__
    |     self._parse_meta_regions()
    |   File "<cle at the merge base>/cle/backends/pe/pe.py", line 489, in _parse_meta_regions
    |     self._meta_iat()
    |   File "<cle at the merge base>/cle/backends/pe/pe.py", line 521, in _meta_iat
    |     iat_dd = self._meta_dd("IMAGE_DIRECTORY_ENTRY_IAT")
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "<cle at the merge base>/cle/backends/pe/pe.py", line 513, in _meta_dd
    |     dd = self._pe.OPTIONAL_HEADER.DATA_DIRECTORY[idx]
    |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
    | IndexError: list index out of range

LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for optarg. What's going on?
LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for optind. What's going on?
LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for stderr. What's going on?
LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for stdout. What's going on?
LOG WARNING cle.loader x1: Symbol imported without a known size; emulation may fail if it is used non-opaqely: Version, __errno_location, __memcpy_chk, __stack_chk_fail, __strcat_chk, __strcpy_chk, _exit, add_envopt, atoi, check_zipfile, clear_bufs, close, closedir, copy, display_ratio, exit, fchmod, fchown, fdatasync, fdopendir, fdutimens, fill_inbuf, fprint_off, free, fstat, fsync, getopt_long, gzip_base_name, gzip_error, header_bytes, isatty, last_component, localtime, memcmp, memmove, open_safer, openat_safer, perror, raise, read_error, rpl_fclose, rpl_fflush, rpl_fprintf, rpl_printf, sigaction, sigaddset, sigemptyset, sigismember, signal, sigprocmask, strcmp, strcpy, strcspn, streamsavedir, strlen, strlwr, strrchr, unlinkat, unlzh, unlzw, unpack, unzip, unzip_crc, updcrc, write_buf, write_error, xstrdup, xunlink, yesno, zip. See https://docs.angr.io/extending-angr/environment#simdata

After — both load, big_buffer comes out as a sectionless export, and the SHN_COMMON control is unchanged:

with this change, cf834d8
cle: <cle with this change>/cle/__init__.py

-- ELF: large_common.o, built -mcmodel=medium, so big_buffer is SHN_X86_64_LCOMMON (0xFF02)
x86_64/large_common.o: loaded ELF relocatable=True symbol='big_buffer' st_shndx-derived section=None is_common=False is_export=True size=2097152

-- ELF control: an ordinary SHN_COMMON symbol
x86_64/decompiler/gzip.o: loaded ELF relocatable=True symbol='ofname' st_shndx-derived section=None is_common=True is_export=True size=1024

-- PE: an EFI stub whose NumberOfRvaAndSizes is 6, not the 16 the format defines
x86_64/efi_short_data_directory.efi: loaded PE sections=['.text', '.data'] entry=0x1000 deps=[] is_dotnet=False

LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for optarg. What's going on?
LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for optind. What's going on?
LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for stderr. What's going on?
LOG WARNING cle.backends.externs x1: Symbol type mismatch between export request and response for stdout. What's going on?
LOG WARNING cle.loader x1: Symbol imported without a known size; emulation may fail if it is used non-opaqely: Version, __errno_location, __memcpy_chk, __stack_chk_fail, __strcat_chk, __strcpy_chk, _exit, add_envopt, atoi, check_zipfile, clear_bufs, close, closedir, copy, display_ratio, exit, fchmod, fchown, fdatasync, fdopendir, fdutimens, fill_inbuf, fprint_off, free, fstat, fsync, getopt_long, gzip_base_name, gzip_error, header_bytes, isatty, last_component, localtime, memcmp, memmove, open_safer, openat_safer, perror, raise, read_error, rpl_fclose, rpl_fflush, rpl_fprintf, rpl_printf, sigaction, sigaddset, sigemptyset, sigismember, signal, sigprocmask, strcmp, strcpy, strcspn, streamsavedir, strlen, strlwr, strrchr, unlinkat, unlzh, unlzw, unpack, unzip, unzip_crc, updcrc, write_buf, write_error, xstrdup, xunlink, yesno, zip. See https://docs.angr.io/extending-angr/environment#simdata

zardus and others added 2 commits September 4, 2026 03:59
Loading aborted with IndexError on two header-controlled numbers that were
used as list indices without being checked against the list.

PE: NumberOfRvaAndSizes states how many data directories an image has, and
fewer than 16 is legal - EFI stub images commonly declare 6 - so pefile parses
a short DATA_DIRECTORY. The IAT (12) and .NET descriptor (14) lookups indexed
it with fixed constants. An index past the end means the image does not have
that directory, which is what a zero VirtualAddress already means, so treat it
the same way. Both lookups now go through _meta_dd, the one place that decides
whether a directory is present, so is_dotnet no longer keeps its own copy of
that decision; it now also wants a nonzero Size, the way every other directory
does.

ELF: st_shndx values from SHN_LORESERVE up are reserved tags, not section
header table indices. pyelftools decodes only SHN_UNDEF, SHN_ABS and
SHN_COMMON into strings and passes the rest through as ints, so a symbol in
the processor-specific SHN_X86_64_LCOMMON indexed the section list. Such a
symbol names no section, so it gets neither a section nor a remap offset. It
stays an export: SHN_X86_64_LCOMMON is what a large common symbol gets
instead of SHN_COMMON, and cle already exports those.

Both cases need an input no binary in angr/binaries had, so both fixtures are
new there: efi_short_data_directory.efi, a UEFI application whose optional
header declares six directories, and large_common.o, built by
gcc -mcmodel=medium. Each reproduces its IndexError on the unfixed loader.

Requires the angr/binaries branch feature/short-data-directory-and-lcommon.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Typecheck job scores each changed file against master's copy of it:
badness is (10*errors + warnings)/lines and must not increase. Its
environment installs types-pefile, whose stub types
OPTIONAL_HEADER.DATA_DIRECTORY as a list of entries carrying VirtualAddress
and Size. _meta_dd declared pefile.Structure, the base class that has
neither, which threw that away for every caller: 39 of pe.py's 52 errors are
a caller reading VirtualAddress or Size off the result.

That is what makes this branch red. It adds no diagnostic of its own - it
scores the same 52 errors as master - but its pe.py is 26 lines shorter than
master's, and the same error count over a smaller line count is a regression
by that measure.

Leaving the return type to inference gives the entry type where the stub is
installed and an unknown one where it is not, so the callers type-check and
nothing about them changes at run time. pe.py goes from 52 errors to 13.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zardus
zardus force-pushed the feature/fix-cle-header-indices branch from cf834d8 to 43c897a Compare September 4, 2026 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants