Is it possible to have libdwarf use an in-memory object file

[jeremy-rifkin]

I'm looking at supporting JIT compilers in a library of mine which uses libdwarf. From compilers like LLVM I can get in-memory object files (an ELF or mach-o file that just lives in memory and I get a const char* and size_t for it).

I see there is a dwarf_object_init_b utility which can be used for in-memory dwarf but this seems to involve having lots of information about all of the internal structures of the object file, which I'd like to avoid having to parse myself.

Does libdwarf have an interface taking a const char* and size_t to an in-memory object file? If not, this would be very helpful to have.

[davea42]

There is no provision for situations with a complete object file of known format (Elf, MacOS, PE)
that resides in memory. That might not be difficult to add
with a new dwarf_init...() call, but I won't have time to think
about it today (sorry). How about with an argument specifying the object format (Elf, MacOS, PE) in addition
to memory-address and size? I think that would help simplify things for libdwarf.

[jeremy-rifkin]

Thanks for the quick reply, no rush on my end! It'd be great to have a solution at some point in the coming months though for the Jank programming language jeremy-rifkin/cpptrace#226.

How about with an argument specifying the object format (Elf, MacOS, PE) in addition to memory-address and size?

That would be perfect; I'll know the native platform object file type on my end and can just use that. I can also check magic bytes if needed.

[davea42]

First I'll figure out a sensible overview, a very few 'chunks', and make that match
things to minimize disruption of the code.

Right now getting basic elf/Mac/PE object data is distinctly different from
the DWARF section data.

Your request is as if we did mmap of an entire object file at once right at the start.

[davea42]

I figured out a way to allow all the various allocation methods
(malloc,mmap-per-section, single file mmap) such that
it's simple logic (as simple as possible, no simpler)
in libdwarf.

A new header file,
src/lib/libdwarf/dwarf_seekr.h

(not committed yet) has the data details.
The struct there will be in the Dwarf_Debug and
in every section record. And of course it allows
for decompression forcing malloc and
relocations forcing malloc on particular sections.

More later.

[davea42]

Jeremy: A question for you. (or for anone willing to comment)

In your view and your experience, is an object-file-in-memory
the simple object with any DWARF information? Or could it be the executable and
DWARF information is off in another file (or split up)? (As in split-dwarf or MacOS dSYM or Macos Universal Binary
or Elf section-groups)

I would suppose it has to be a simple self-contained object without section-groups or multiple
program objects in the JIT .

And also wondering if there is a general case here that must be considered.

[jeremy-rifkin]

In your view and your experience, is an object-file-in-memory
the simple object with any DWARF information? Or could it be the executable and
DWARF information is off in another file (or split up)?

What I've seen is dwarf info in the in-memory elf file. I would be surprised to see anything like split-dwarf of dSYM, that would defeat most of the point of the in-memory object file approach as I understand it. I'm not sure if mach-o universal binaries or elf section groups could appear.

[davea42]

Given the existence of a block of memory being in an object format your
application could invent a suitable unique name, write the block of
memory to the file, and process the file using libdwarf.

It would also allow validation of the format of the data right now.

And maybe permission could be granted for me to add such a file
to the libdwarf regressiontests.