Extensible Calling Conventions for native callee functions

[AaronRobinsonMSFT]

Background and Motivation

The UnmanagedCallersOnlyAttribute provides an extensible mechanism to define native calling conventions on a .NET method. This enables users to author functions that are callable in a native environment. A symmetrical need arises for functions that are defined in native code but declared to be called in a .NET environment. These native function declarations are done using DllImportAttribute.

This approach complements UnmanagedCallersOnlyAttribute and aligns with the reuse of the CallConv* types under the System.Runtime.CompilerServices namespace.

Proposed API

If the user specifies the CallingConvention.Winapi (the default) on the DllImportAttribute the following attribute will be queried.

namespace System.Runtime.InteropServices
{
+    /// <summary>
+    /// Provides an equivalent to <see cref="UnmanagedCallersOnlyAttribute"/> for native
+    /// functions declared in .NET.
+    /// </summary>
+    [AttributeUsage(AttributeTargets.Method)]
+    public class UnmanagedCalleeAttribute : Attribute
+    {
+        public UnmanagedCalleeAttribute() { }
+
+        /// <summary>
+        /// Types indicating calling conventions for the unmanaged callee.
+        /// </summary>
+        /// <remarks>
+        /// If <c>null</c>, the semantics are identical to <c>CallingConvention.Winapi</c>.
+        /// </remarks>
+        public Type[]? CallConvs;
+    }
}

Usage Examples

P/Invoke

Using the defaults, all 3 of the following declarations are the same.

[DllImport("nativelib")]
static extern int Func(IntPtr a, nint val);

[DllImport("nativelib", CallingConvention = CallingConvention.Winapi)]
static extern int Func(IntPtr a, nint val);

[UnmanagedCallee]
[DllImport("nativelib")]
static extern int Func(IntPtr a, nint val);

Some calling conventions can be declared in multiple ways.

[DllImport("nativelib", CallingConvention = CallingConvention.Cdecl)]
static extern int Func(IntPtr a, nint val);

[UnmanagedCallee(CallConvs = new[] { typeof(CallConvCdecl) })]
[DllImport("nativelib")]
static extern int Func(IntPtr a, nint val);

[UnmanagedCallee(CallConvs = new[] { typeof(CallConvCdecl) })]
[DllImport("nativelib", CallingConvention = CallingConvention.Winapi)]
static extern int Func(IntPtr a, nint val);

Future calling conventions would be enabled without updating the CallingConvention enum. Examples in C# and VB.NET.

[UnmanagedCallee(CallConvs = new[] { typeof(CallConvVectorCall) })]
[DllImport("nativelib")]
static extern int Func(IntPtr a, nint val);

<UnmanagedCallee(CallConvs := New Type() { GetType(CallConvVectorCall) })>
<DllImport("nativelib")>
Public Shared Function Func(a As IntPtr, val As IntPtr) As Integer
End Function

This proposal provides an alternative mechanism to support Native Varargs in #48796.

COM style interface - Not proposed but example of how the attribute could evolve.

To enable this scenario the attribute would be updated with the following: [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]

Using the defaults, all 3 of the following declarations are the same on Windows.

[ComVisible(true)]
[Guid("C509A1C1-A6C9-4229-89CE-10B53CF8271B")]
interface IFoo
{
    int Call(ref SomeType valuetype);
}

[ComVisible(true)]
[Guid("C509A1C1-A6C9-4229-89CE-10B53CF8271B")]
[UnmanagedCallee]
interface IFoo
{
    int Call(ref SomeType valuetype);
}

[ComVisible(true)]
[Guid("C509A1C1-A6C9-4229-89CE-10B53CF8271B")]
[UnmanagedCallee(CallConvs = new[] {  typeof(CallConvStdCall), typeof(CallConvMemberInstance) })]
interface IFoo
{
    int Call(ref SomeType valuetype);
}

Alternative Designs

Provide an enum to trigger off for reading the new attribute.

namespace System.Runtime.InteropServices
{
    public enum CallingConvention
    {
+        /// <summary>
+        /// Instructs the runtime to consult the <see cref="UnmanagedCalleeAttribute"/>.
+        /// </summary>
+        UnmanagedCallee = 7
    }
}

Alternative name proposed could be CallingConvention.UnmanagedCallConv.

[AaronRobinsonMSFT]

/cc @jkotas @jkoritzinsky @elinor-fung @tannergooding @lambdageek

[tannergooding]

It took me a second to put this together😄
So this is just how we plan on evolving calling conventions for DllImport and similar scenarios?

If I understood right, for future calling conventions, we basically annotate as CallingConvention.UnmanagedCallee and that will cause the runtime to resolve the UnmanagedCallee attribute (looking on method first, then directly containing type). The CallConv* types specified determine what the correct convention is?

What is the behavior if the attribute doesn't exist or contains invalid types?
How far up the type hierarchy will the runtime look?
Have any alternative names to UnmanagedCallee been suggested (such as just UnmanagedCallConv)?

[AaronRobinsonMSFT]

So this is just how we plan on evolving calling conventions for DllImport and similar scenarios?

Yep. This is handling the case where someone wants to consume a native function. We have it already for C# function pointers so we have NaitveLibrary users covered. The only other existing ways are traditional COM interop and P/Invoke.

The CallConv* types specified determine what the correct convention is?

Yes. Now we can enable all existing interop mechanisms to consume the CallConv* approach.

What is the behavior if the attribute doesn't exist or contains invalid types?

My preference would be a hard fail. The attribute is the contract and the CallingConvention is stating - look here.

How far up the type hierarchy will the runtime look?

Great question. I would say on the closest enclosing type. From the runtime data structure view it would be what is on the associated MethodTable. More than willing to drop the class support but I think it might be useful longer term with source generators involving C++ interop. COM interop was simply to use an existing concrete example.

Have any alternative names to UnmanagedCallee been suggested (such as just UnmanagedCallConv)?

You're the first. Will add.

[jkotas]

UnmanagedCallee = 7

A alternative design would be allow this attribute with the default calling convention, ie:

[UnmanagedCallee(CallConvs = new[] { typeof(CallConvVectorCall) })]
[DllImport("nativelib")]
static extern int Func(IntPtr a, nint val);

It looks better, but it is more expensive at runtime because we have to lookup an extra attribute most of the time. I am not sure which one I like better.

FWIW, we have introduced an extra attribute lookup for SuppressGCTransitionAttribute recently and it has not showed up on the radar, so adding one more would be probably fine.

More than willing to drop the class support but I think it might be useful longer term with source generators involving C++ interop

I think we should keep this simple and only support this on DllImport methods. We can always extend it later if needed. None of the existing CallConv constructs (function pointers, UnmanagedCallersOnlyAttribute) have a type default like this.

[AaronRobinsonMSFT]

A alternative design would be allow this attribute with the default calling convention

Agreed. I had considered this approach because of the potential tooling impact of adding a new enum value. Would you then get rid of the new enum entirely?

we have introduced an extra attribute lookup for SuppressGCTransitionAttribute recently and it has not showed up on the radar, so adding one more would be probably fine.

That is true, but this one will likely have data content as opposed to SuppressGCTransitionAttribute which acts as an indicator with no data content. Either way, I think it warrants a quick perf investigation to see exactly what we could expect.

I think we should keep this simple and only support this on DllImport methods. We can always extend it later if needed.

That is reasonable.

[jkotas]

Would you then get rid of the new enum entirely?

Yep.

[AaronRobinsonMSFT]

Another aspect of the default proposal is that regardless of the user setting CallingConvention = CallingConvention.Winapi or not, we would need to check for the attribute since DllImport is a pseudo attribute and embedded in metadata - I don't think there is a way to disambiguate between explicitly setting CallingConvention.Winapi or the user not setting anything.

I'm a little leery of that approach because I would prefer this sort of thing to be explicit and not a magic contract. However, a rebuttal to that is it follows our desire to make the default declaration the canonical approach and move away from mechanisms that expose .NET's legacy x86 focus – see C#'s function pointer design.

I am also moving this to .NET 6 since I would like to fold support here into the DllImport source generator and be able to avoid the CallingConvention on any new attribute.

[jkoritzinsky]

In IL there's a way to disambiguate between Winapi and no setting, but Roslyn makes the default Winapi, so we'd have to check for the attribute whenever Winapi is set.

[AaronRobinsonMSFT]

In IL there's a way to disambiguate between Winapi and no setting, but Roslyn makes the default Winapi, so we'd have to check for the attribute whenever Winapi is set.

No, I don't think there is - #51156 (comment).

Really? How do we do that in IL?