Skip to content

IRToASTVisitor: preserve native calling conventions in declarations and calls #347

Open

Description

@humdrum00001010

Rellic v0.2.3 creates function declarations using a default Clang
FunctionProtoType::ExtProtoInfo. It does not copy Function::getCallingConv().
Calls through function pointers similarly construct a default C function type
without reading CallInst::getCallingConv().

An LLVM module with x86_stdcallcc, x86_fastcallcc, and x86_thiscallcc
therefore becomes plain C declarations, definitions, and indirect calls. This
changes argument registers and stack cleanup when the result is compiled for
i686-pc-windows-msvc.

A synthetic module contains eight functions: four sum functions with different
conventions, one caller, and three callers through pointers. For example, these
are declarations and a call from the synthetic module:

define x86_stdcallcc i32 @stdcall_sum(i32 %a, i32 %b, i32 %c) {
  %ab = add i32 %a, %b
  %abc = add i32 %ab, %c
  ret i32 %abc
}

define x86_fastcallcc i32 @fastcall_sum(i32 inreg %a, i32 inreg %b, i32 %c) {
  %ab = add i32 %a, %b
  %abc = add i32 %ab, %c
  ret i32 %abc
}

define x86_thiscallcc i32 @thiscall_sum(ptr %receiver, i32 %a, i32 %b) {
  %value = load i32, ptr %receiver
  %ab = add i32 %a, %b
  %abc = add i32 %ab, %value
  ret i32 %abc
}

define i32 @call_indirect_stdcall(ptr %fn, i32 %a, i32 %b, i32 %c) {
  %result = call x86_stdcallcc i32 %fn(i32 %a, i32 %b, i32 %c)
  ret i32 %result
}

The generated declarations are currently ordinary C declarations, and the
pointer call has an ordinary C pointer cast. A static roundtrip can be checked
without executing argument cases:

rellic-decomp --input calling-conventions.ll --output calling-conventions.c
clang --target=i686-pc-windows-msvc -S -emit-llvm -O0 \
  calling-conventions.c -o roundtrip.ll

The return to LLVM drops the three named native conventions. A local fix
preserves both Clang function-type conventions and printed declaration
attributes, and creates convention-bearing typedefs for indirect calls. The
typedefs avoid invalid placement of attributes inside a C cast type-name.
Unsupported LLVM-only conventions produce an explicit error.

Static comparison of the complete synthetic module linked as two DLLs gives
byte-exact code for the four individual sum functions and all three indirect
callers. The combined caller has an instruction-ordering difference after
optimization; that difference is retained in the report.

No third-party binary, machine-code sample, runtime inputs, local identifiers,
or product-specific information is included in this report.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions