COM interface methods are nicer to use if they return the last `out` parameter instead of void

[jnm2]

It's nicer to be able to dot off returned values than to come up with names for out var parameters for void method calls.

It enables code that looks like this:

FileName = dialog.GetResult().GetDisplayName(SIGDN.SIGDN_DESKTOPABSOLUTEPARSING);
fileTypeIndex = (int)dialog.GetFileTypeIndex();

Instead of:

dialog.GetResult(out var shellItem);
shellItem.GetDisplayName(SIGDN.SIGDN_DESKTOPABSOLUTEPARSING, out var displayName);
FileName = new string(displayName);
// TODO: free displayName (my declaration returned `string` from GetDisplayName instead of PWSTR)
dialog.GetFileTypeIndex(out var uintFileTypeIndex);
fileTypeIndex = (int)uintFileTypeIndex;

My own declaration:

/// <summary>
/// https://docs.microsoft.com/en-us/windows/win32/api/shobjidl_core/nn-shobjidl_core-ifileopendialog
/// </summary>
[ComImport, Guid("D57C7288-D4AD-4768-BE02-9D969532D960")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
internal interface IFileOpenDialog
{
    // ...
    uint GetFileTypeIndex();
    // ...
    IShellItem GetResult();
    // ...
}

[AArnott]

We already do approximately what you're asking for (#120).
But we're not going to automatically move whatever the last out parameter is to be a return type. It's based on the metadata's use of [RetVal] on that last parameter. Those tend to come from the IDL or header files. If you search the metadata for uses of RetValAttribute you'll see a great many hits. So if your feedback is targeting specific APIs that lack it that you feel should have it, please file an issue for those APIs against the win32metadata repo.

[jnm2]

This is going to be a lot of API requests, since I'm not going to come across a void-returning method with a final out parameter that I'm going to appreciate.

[AArnott]

TBH I'm not sure we should deliver on those requests in the metadata repo either. Our goal is to produce the same methods that Windows defines, with friendly overloads that handle some type conversions and memory management in a .NET-friendly way. The methods should still closely resemble the native ones so that code conversion from C++ to C# is straightforward and documentation still applies.
Our goal is not to make the APIs more like a BCL. Revising a method signature just to make it more convenient (rather than for memory management or type compatibility reasons) seems to at least approach the scope line.

But you're welcome to file an issue on win32metadata of course and more folks can participate in the discussion there.

[jnm2]

The reason I bring it up is not to make APIs more like a BCL but to provide parity with handwritten interop definitions. Every time using CsWin32 results in worse code than handwritten definitions, I do like to have the discussion.

[AArnott]

Another concern is I've learned as part of COM interop is that very subtle changes can surprisingly impact the ABI, which on some CPU architectures can break the runtime. I wonder whether an out parameter vs. a return value might break things. That's something else we'll need to verify before we generate something different.

[AArnott]

@weltkante @AaronRobinsonMSFT Do you have any concerns with rewriting COM interface methods with a signature like this:

HRESULT Some(out int foo);

which without preservesig would turn into this in C#:

void Some(out int foo);

into something even more refined like this:

int Some();

?

Any ABI concerns to be aware of?

As mentioned above, I know the IDL allows for [retval] annotations, which CsWin32 already honors to do the above transformation. But the suggestion in this issue is to perform this transformation even without this annotation.

[weltkante]

• you get exceptions instead of HRESULTs which can be a debugging/performance issue, especially when failures are common in a given API usage (I occasionally encounter that an API asks you to "try" something and will return a failure HRESULT if its not available, and then it needs to be done in a loop, ...)
• you cannot distinguish success results (S_OK vs. S_FALSE etc.)
• some out parameters are optional, but the out keyword is already the wrong projection for those anyways, so moving an out parameter into return parameter probably doesn't break those cases more than they are already broken now

In summary, the common case will probably profit from this transformation, but doing it in general breaks a minority of APIs and/or makes them much more painful to use.

[AaronRobinsonMSFT]

@weltkante's list is great. One thing that might impact the ABI in profound ways is for out value types that are moved to retval.

From a COM purest standpoint, if the metadata doesn't have the retval it shouldn't be converted to be one. There are lots that can likely be done to manipulate the API to conform to C# patterns, but this one violates the API writer's intent. That may not be a good enough reason not to do it, but I could imagine niche cases, especially for value types, where this is not a good idea.

[jnm2]

I noticed that everything on @weltkante's list applies just as much to void Some(out int foo); as to int Some();, given that the preservesig definition would be HRESULT Some(out int foo);, so they would not be reasons to prefer void Some(out int foo); over int Some();.

@AaronRobinsonMSFT Do you think we should know more about the niche cases to consider whether (or how much) they affect CsWin32?

[AaronRobinsonMSFT]

so they would not be reasons to prefer void Some(out int foo); over int Some();.

Agree for int and perhaps other primitive types. However, the distinction does break with the API author's definition and could impact ABI for value types. I put a lot of value on how the API is defined, if it doesn't have retval it is changing the intent.

Do you think we should know more about the niche cases to consider whether (or how much) they affect CsWin32?

Unfortunately, yes. I would be less concerned if CsWin32 could use .NET 6 APIs and wasn't limited to targeting .NET Standard. The reason for this is in .NET 6 we've added many interop niceties that help interop library developers tell the runtime additional context that makes the dispatch better (for example, CallConvMemberFunction).

[AArnott]

Yes, the inefficiencies concerns are all based on [PreserveSig] which is a separate (and decided) question. That question was answered with "yes, preservesig by default but allow consumers to turn it off for an interface or individual method. Also include a list of APIs known to return S_FALSE and turn PreserveSig off by default for those."

And yes, I agree treating the API as if [retval] were present when it isn't violates the author's intent (assuming it was thought through).

But the key question I am looking for @AaronRobinsonMSFT and @weltkante to answer is around ABI compatibility. In what cases can we expect ABI incompatibilities when moving the last out parameter to be a return type on an otherwise void-returning method, while PreserveSig is not present? @AaronRobinsonMSFT mentioned value types might break. Which value types? What about pointers to arbitrary memory or pointers to COM objects?

If we're not sure the set, but are pretty sure it could present problems, I would proceed with the Won't Fix resolution of this suggestion. If we could get a defined set of problems and could avoid them, then we can consider this issue vs. staying true to the author's intent.

[AaronRobinsonMSFT]

Which value types?

Value types in general can be an issue but specifically those that are <= 8 bytes are the biggest concern in my mind.

However, I just realized this shouldn't be a technical issue if we are talking about properly formed COM interfaces. Properly formed means that they all return an HRESULT. If this is the case, then the method will always pass the value type as a pointer and thus should always be okay to convert to a retval – barring API author's intent. For interfaces that don't strictly follow COM rules (for example, DirectX), the problem is much more difficult because the IDL/header may indicate the return type is a struct and not an HRESULT – this issue is the biggest concern. There are ways to mitigate this, but they are complicated, difficult to get right, and annoying to maintain.

What about pointers to arbitrary memory or pointers to COM objects?

Pointers are primitive types and should "just work" like any other primitive that can reside in a register.

This is why value types that are marked as out can be converted easily. It took me looking at output in godbolt to recall that fact.

Another consideration should be out-of-proc COM and the generated proxy/stubs. For in-proc, lots can be done to make it work, but when we get into out-of-proc we need to be careful that retval doesn't indicate something in the generated proxy/stubs. That isn't something I know off the top of my head. We should reach out to the COM team and verify that.