Load and relocate DOS MZ executables - #837
Conversation
cle refuses every MS-DOS executable with "Unable to find a loader backend", so a whole class of real-mode binaries is unreachable, and the blob recipe the error suggests cannot apply the header's relocation table: a DOS executable is linked at segment 0 and DOS adds the run-time load segment to every word the table names. Add cle.backends.mz.MZ. It maps the load module in a flat 20-bit space at segment * 16 + offset, takes its architecture from the p-code real-mode language, and turns each header relocation into an MZRelocation that adds the load segment to the word it names. The p-code language already resolves far pointers to flat linear addresses, so no segmented-address machinery is needed. Real mode has 16-bit registers and a 20-bit address space, and cle equated the two: Loader._is_range_free and _find_safe_rebase_addr both bounded placement by 2 ** arch.bits, which confines a real-mode object to 64 KiB and fails to place anything larger. Backend grows a mapped_address_bits property defaulting to arch.bits, the two loader sites read it, and MZ overrides it to 20. Nothing else changes behaviour. Two header fields need care. e_cs and e_ss are relative segments DOS adds to the load segment in 16-bit arithmetic, so a high segment word is ambiguous: a linker writes 0xfff0 for the sixteen paragraphs below the load module, where the Program Segment Prefix sits, and a large enough image reaches a high segment counting upwards. _resolve_relative takes whichever reading lands in range, and the bounds are half-open so at most one of them can. And e_lfanew sits at 0x3c whatever the header's paragraph count says, so the check that refuses PE, NE, LE, LX, W3, W4, P2, P3, PL and PM containers is not gated on the declared header size. .COM files, the extended containers, DOS extenders whose real program lives past the stub, overlay modules, and the Program Segment Prefix are all out of scope.
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Which tree was testedThe gate below was launched one commit earlier, at The workspace gatecle
The suites that bear on this change
The guard was checked in the direction that could fail, not only the one that The built Lint and type comparison against the baseline
That took two rounds of repair to reach: pyright first reported 14 errors in CorpusA private third-party collection of DOS-era binaries, 2,128 distinct sha256 in its On the baseline all 2,123 fail with one message, The 552 that still fail, grouped by the check that refuses each: Independent check on the first class: exactly 475 of the 2,123 do not begin with That 28 of the 1,571 carry a protected-mode label in the corpus's own classification. For 153 objects separate the 1,418 arm measured mid-development from the 1,571 arm.
The sweep was launched from an earlier revision of this branch. Its ControlsNegative control, which is what makes The wrapped-segment resolution was checked for the value it returns, not only for FixturesThe three fixtures come from |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS What Before — every DOS MZ executable is refused, whatever is in it: cle master 0e77adeAfter — both load, in a 20-bit address space, with the entry point taken from with this change
After — the relocation table is applied, which is the part a with this changeWithout the fixups every one of those words still reads its file value, so every |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_837 |
|
@rhelmot Do we want to add 16-bit support? |
|
I would like to have 16-bit support for a personal project I'm doing! |
|
Merged. I guess loading 16-bit executables is not a big burden, but we do not plan to support 16-bit arches in angr proper. |
|
Neither Yan nor I remember why we decided to not support 16-bit architectures in the first place, and after a longer discussion, I think we all believe it's time to abolish that rule, so we should reconsider supporting 16-bit architectures in angr. |
3 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
clecannot load an MS-DOS executable. Every MZ file whose header belongs to a DOSprogram rather than to a stub is refused:
That is angr/angr#48, open since December 2015: "this is a
MS-DOS executable, which is not currently supported by CLE. It wouldn't be too hard
to write a loader for this format".
The
blobload the error suggests is not a substitute, because it cannot apply theheader's relocation table -- it reports no relocations at all. Of 1,648 MZ objects in
a corpus measured for this change, 1,152 declare a non-empty relocation table, and
every word those tables name keeps its link-time value under a blob load.
Root cause
No registered backend claims the container, and two things stand in the way of one.
The first is the relocation table: a DOS executable is linked at segment 0 and DOS
adds the run-time load segment to every word the table names. That is a real
relocation, so it belongs to a backend rather than to a byte-blob recipe.
The second is that CLE equates the address space with the register width.
Loader._is_range_freerejects any placement past2**self.main_object.arch.bits,and
_find_safe_rebase_addrbounds its search the same way:Real mode has 16-bit registers and a 20-bit address space, so that confines a
real-mode object to 64 KiB and refuses to place anything larger.
arch.bitsis theright answer for the register width and the wrong answer for the address space, and
they are not the same question.
Fix
cle.backends.mz.MZ, registered asmzand probed by default. It maps the loadmodule in a flat 20-bit space at
segment * 16 + offset, takes its architecture fromarchinfo.ArchPcode("x86:LE:16:Real Mode"), setsos = "dos", and turns each headerrelocation into an
MZRelocationthat adds the load segment to the word it names.The p-code real-mode language already resolves far pointers to flat linear addresses,
so no segmented-address machinery is needed.
To separate the address space from the register width,
Backendgrows amapped_address_bitsproperty defaulting toself.arch.bits, and the two lines ofcle/loader.pyabove read it instead.MZoverrides it to 20; nothing else changesbehaviour.
Two header fields need care.
e_csande_ssare relative segments that DOS adds tothe load segment in 16-bit arithmetic, so a high segment word is ambiguous: a linker
writes
0xfff0for the sixteen paragraphs below the load module, where the ProgramSegment Prefix sits, and a large enough image reaches a high segment counting
upwards.
_resolve_relativetries both readings and takes the one in range. They areexactly one real-mode address space apart, so a half-open bound admits at most one of
them; with an inclusive bound the literal reading can land on the ceiling while the
wrapped one lands on zero, both pass, and there is nothing to choose between them.
One of the fixtures has that header and a test pins it.
And
e_lfanewsits at offset0x3cwhatever the header's paragraph count says, sothe check that refuses PE, NE, LE, LX, W3, W4, P2, P3, PL and PM containers is not
gated on the declared header size — PE images exist that declare a two-paragraph
header and still carry a real pointer there.
Deliberately not done:
.COMfiles, for whichblobis an adequate recipe; the NE,LE, LX and PE containers this backend refuses; DOS extenders whose real program lives
past the stub; overlay modules; and the Program Segment Prefix, the uninitialised
allocation and trailing overlays, none of which are synthesised.
Testing
tests/test_mz.py, 10 tests over the three fixtures in the sibling pull request.They assert the backend is selected without being named, the header fields against
the file's bytes, the entry point against the instructions at
CS:IP, that each offive relocation sites reads
file_word + load_segmentafter a rebase, that a zeroload segment leaves the image byte-identical to the file, that a wrapped
e_csande_ssresolve into the load module and name the PSP after a rebase, that the wrappedstack whose literal reading sits on the 1 MiB ceiling resolves to the start of the
image rather than past the top of the address space, that a tracked PE still loads as
a PE, and that a
.COMfile is not claimed. The module skips itself where pypcode isabsent, as
tests/test_arch_detect.pyalready does.Over a corpus of 2,123 distinct DOS-candidate binaries,
cle.Loaderloads 0 onmaster and 1,571 with this change; the 552 that still fail are containers this
backend deliberately refuses, chiefly headerless
.COMimages and LE extenders. Thatcorpus is private and cannot be published, so the three fixtures are the public
reproducer. Before this change
angr/binariestracked 106 files beginning with theMZmagic and every one was a PE; all 106 still load, and none is claimed by the newbackend. The validation record has the full breakdown and the controls.
Merge angr/binaries#232 first: these tests load its fixtures.
Fixes angr/angr#48. Validation: #837 (comment)
sync: angr/binaries#232
session: sharpen