Read the Go pclntab on Mach-O and PE, not only ELF - #808
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head The branch was rebased onto baseline Suite. The new tests are load-bearing. Reverting the four changed production files Merge-base lint and typecheck. Pylint, the comparison the hosted Lint job The pyright figures first recorded here were the wrong measurement, and the
Corpus A/B. The corpus is every tracked file in
Cost of dropping the gate. The scan is a CFG recovery,
The total function count falls at the same time, 3,816 to 2,496 and 2,342 to Downstream. The three angr tests that decompile these two Go PEs are Language detection. Not covered. Two of the thirteen Go PEs, the session: sharpen |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS What cle reports for the three Go fixtures the new tests use, before and after Before — no pclntab is read on either format, so no Go function has a name cle masterAfter — every pclntab entry becomes a function symbol, and the recovered with this change |
86cceb2 to
9167cb7
Compare
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_808 |
cle/backends/gopclntab.py was written for all three container formats.
PCLNTAB_SECTION_NAMES lists __gopclntab and __go_pclntab beside the ELF
spellings, _EMBEDDING_SECTION_NAMES lists .rdata, .rodata, __rodata, __const and
__DATA_CONST for the formats that bury the table in a generic read-only section,
and the module docstring names ELF, Mach-O and PE. Only the ELF backend ever
called register_gopclntab_symbols, so a Go binary built for darwin or windows
reached the loader with whatever handful of function symbols its symbol table
happened to carry. On tests/x86_64/windows/langdetect_go.exe that is 85 symbols
against 1898 Go functions.
Mach-O and PE were not failing for the same reason:
- Mach-O has a section named __gopclntab, which the name list already matched,
so load_gopclntab found and parsed the table. Nothing called it.
- PE has no section a Go-marker name test can recognise. Thirteen Go PEs in
the corpus load, between them showing six different section-name sets; all
of them carry .text, .rdata, .data, .idata, .reloc and .symtab, and not one
carries a name in _GO_MARKER_SECTION_NAMES or one starting ".go.". The
pclntab sits inside .rdata and has to be found by its magic, and
_looks_like_go gated that scan on exactly those names, so it never ran. Drop
the gate and let GoPclntab.parse decide, which is what validates a candidate
anyway: it checks the header padding, the pointer size, the minimum
instruction length, the five sub-table offsets, that the function entries
increase monotonically, and that every name is terminated inside the table.
Calling register_gopclntab_symbols from the Mach-O backend exposed three places
where that backend narrows Backend's symbols list to AbstractMachOSymbol without
saying so: SymbolList.add indexes every symbol by library_ordinal, get_symbol
reads is_stab, and get_symbol_by_address_fuzzy reads bind_xrefs and
symbol_stubs. Those four attributes are declared on AbstractMachOSymbol and on
nothing else, so any plain Symbol added by shared loader code hits an
AttributeError. Guard the three sites instead of teaching a format-neutral
symbol about Mach-O.
Declare gopclntab on Backend so every object answers the question rather than
only some.
9167cb7 to
bd92d8a
Compare
3 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
cle reads a Go binary's
pclntabonly when the binary is an ELF. On Mach-O andPE it reads nothing, so a Go binary arrives with whatever handful of names its
symbol table happens to carry.
tests/x86_64/windows/langdetect_go.exeis therepository's own cross-compiled Go program, and its pclntab names 1,898
functions:
tests/aarch64/langdetect_go.machobehaves the same way, and it has a sectioncalled
__gopclntab.Two things downstream lose by this.
CFGFaston the stripped Go PEtests/x86_64/windows/131252a8059fdbb12d77cd4711e597c45bb48e6d4bc3ddc808697a5e0488ff2cfinds 1,783 of that binary's 1,821 Go functions and can name six of them. And
Project.is_go_binaryis"go" in self.languages() and self.language_is_certain,and
language_is_certainaccepts onlyNoneor"high". With noruntime.*symbol and no section name
LanguageDetectorrecognises, a Go Mach-O or PE isdetected as Go at
lowconfidence, so the Go calling convention never engageson either format.
Root cause
cle/backends/gopclntab.pywas written for all three formats.PCLNTAB_SECTION_NAMESlists__gopclntaband__go_pclntabbeside the ELFspellings, and
_EMBEDDING_SECTION_NAMESlists.rdata,.rodata,__rodata,__constand__DATA_CONSTfor the formats that bury the table in a genericread-only section. But
register_gopclntab_symbolshas exactly one caller,cle/backends/elf/elf.py:252at2f7657fd.Mach-O and PE do not fail for the same reason, which is why fixing one would not
have fixed the other.
Mach-O has the
__gopclntabsection, soload_gopclntabfinds and parses thetable without complaint. Nothing calls it.
PE carries no Go-named section, so the table has to be found by its magic inside
.rdata. That scan sat behind a gate:_GO_MARKER_SECTION_NAMESis{".gosymtab", ".typelink", ".itablink", ".noptrdata", ".noptrbss"}, and no Go PE has any of them. Thirteen Go PEs inangr/binariesata2eb7cfload, between them showing six differentsection-name sets; all thirteen carry
.text,.rdata,.data,.idata,.relocand.symtab, twelve add.zdebug_*and two add.pdataand.xdata, and not one carries any of those five names or a name starting.go.. So the gate is False for every Go PE and the scan never runs.Fix
Call
register_gopclntab_symbolsfrom the Mach-O and PE backends, where the ELFbackend already calls it, and declare
gopclntabonBackendso every objectanswers the question rather than only some.
elf.pyis left alone.Delete the gate rather than repair it, and let
GoPclntab.parsedecide. That iswhat validates a candidate anyway: it checks the header padding, the pointer
size, the minimum instruction length, the five sub-table offsets, that the
function entries increase monotonically, and that every name is terminated
inside the table. A repaired gate is possible — of the 106 PE paths in
angr/binariesata2eb7cf(105 distinct blobs; the one duplicated pair is notGo), the thirteen with a
.symtabsection are exactly the thirteen Go ones —but that is a Go linker detail rather than a documented marker. What it saves is
a
bytes.findfor two four-byte magics over an object's non-executableread-only sections, and across that whole corpus those sections declare 36.2 MB
on the 862 objects the gate was keeping the scan away from.
Calling this from the Mach-O backend turned up three places where that backend
narrows
Backend.symbolstoAbstractMachOSymbolwithout saying so.SymbolList.addindexes every symbol bylibrary_ordinal,get_symbolreadsis_stab, andget_symbol_by_address_fuzzyreadsbind_xrefsandsymbol_stubs. Those attributes are declared onAbstractMachOSymboland onnothing else, so any plain
Symboladded by shared loader code raisedAttributeError. The three sites are guarded instead of teaching aformat-neutral symbol about Mach-O; they are the only reads of a Mach-O-only
symbol attribute over
self.symbolsanywhere in cle.Not done here: at
2f7657fdno Mach-O symbol is reported as a function, so theaddress-based deduplication in
register_gopclntab_symbolssuppresses nothingon that format and every pclntab entry is added beside the underscore-prefixed
name the Mach-O symbol table already carries. The pull request #796 changes
that. With both applied the dedupe, which compares addresses only, covers all
1,888 pclntab addresses on the Mach-O fixture and keeps
_main.mainovermain.main, so it will need to compare names as well as addresses.Testing
Five new tests in
tests/test_gopclntab.pyassert the recovered table andsymbols on a Go PE, a stripped Go PE and a Go Mach-O, and the existing non-Go
test is widened to three binaries, one per format, each with a read-only section
the scan now reaches. Reverting the four changed production files to
2f7657fdfails exactly those six and leaves the other seven passing.
The Mach-O fixture is new. It arrives in angr/binaries#224, which this
needs.
Validation: #808 (comment)
session: sharpen