SocketsHttpHandler: Timeout from ConnectTimeout should use timeout exception pattern

Description

Inconsistent behavior of Socket class on different platforms doesn't allow to use a new timeout handling mechanism in .NET 5.0 as described in this blog post.

Here is the repro code:

static async Task Main()
        {
            using var client = new HttpClient(new SocketsHttpHandler { ConnectTimeout = TimeSpan.FromMilliseconds(100) }, true)
            {
                BaseAddress = new Uri("http://localhost:3262")
            };

            using var cts = new CancellationTokenSource();
            cts.CancelAfter(200);
            var response = await client.SendAsync(new HttpRequestMessage { Method = HttpMethod.Get }, cts.Token);
            Console.WriteLine(response);
        }

Assume that nothing is hosted on http://localhost:3262 address. On Windows, I have the following exception:

System.Threading.Tasks.TaskCanceledException
  HResult=0x8013153B
  Message=The operation was canceled.
  Source=System.Net.Http
  StackTrace:
   at System.Net.Http.ConnectHelper.<ConnectAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<ConnectAsync>d__82.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<CreateHttp11ConnectionAsync>d__86.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<GetHttpConnectionAsync>d__67.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<SendWithRetryAsync>d__72.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<SendAsyncCore>d__85.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ConsoleApp2.Program.<Main>d__0.MoveNext() in Program.cs:line 20
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ConsoleApp2.Program.<Main>()

  This exception was originally thrown at this call stack:
    System.Net.Http.ConnectHelper.ConnectAsync(System.Func<System.Net.Http.SocketsHttpConnectionContext, System.Threading.CancellationToken, System.Threading.Tasks.ValueTask<System.IO.Stream>>, System.Net.DnsEndPoint, System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
    System.Threading.Tasks.ValueTask<TResult>.Result.get()
    System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable<TResult>.ConfiguredValueTaskAwaiter.GetResult()
    System.Net.Http.HttpConnectionPool.ConnectAsync(System.Net.Http.HttpRequestMessage, bool, System.Threading.CancellationToken)
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)

Exception type is TaskCanceledException with InnerException equals to null instead of TimeoutException as described in the blog post.

Now let's try to launch this code on Linux:

System.Net.Http.HttpRequestException: Connection refused (localhost:3262)
System.Net.Sockets.SocketException : Connection refused
    Stack Trace:
       at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at ConsoleApp2.Program.<Main>d__0.MoveNext() in Program.cs:line 20
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at ConsoleApp2.Program.<Main>()
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)

Exception type is HttpRequestException with InnerException equals to SocketException.

Configuration

Checked on .NET 5.0.102
Linux Ubuntu 20.04.1 LTS x86_64
Windows 10 Enterprise x86_64

Regression?

N/A

Other information

Related: #34474

UPD: The same behavior observed when real address is used instead of localhost.

From the trace it seems like there is nothing listening on the port. Socket error would be propagated up if it happens before timeout. You can use Wireshark and look at what is happening at the network.

there is nothing listening on the port.

Correct, but why the exception is different on various platforms?

From my point of view, such behavior increases complexity of code for handling errors produced by HttpClient:

try
{
  await client.SendAsync(...);
}
catch (HttpRequestException e)
{
  // handle HTTP-related exceptions such as unexpected error code
  // OR
  // handle refused connection on Linux
}
catch (OperationCanceledException e) when (!token.IsCancellationRequested)
{
  // handle refused connection on Windows
}
catch (OperationCanceledException e) when (e.InnerException is TimeoutException)
{
  // handle timeout on all platforms
}

It depends on system you are connecting to. I get something very similar exception even on Windows if you connect to Linux host with closed port:

C:\Users\test\source\repos\proxy-test\proxy-test>dotnet run http://10.211.55.5:3030/
System.Net.Http.HttpRequestException: No connection could be made because the target machine actively refused it. (10.211.55.5:3030)
 ---> System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

The difference is that in some cases you get RST (or ICMP unreachable) and in some cases it just times out of there is no response. So the inconsistence is on the server, not on the client IMHO.

In both cases I tried to connect to the same Linux host in my home network. I think it's valid to expect the same behavior on various platforms. But the exception type depends on client platform, not server.

it just times out of there is no response

In this case it would be great to see TimeoutException as inner exception.

If you make the ConnectionTimeout smaller, you'll get System.Threading.Tasks.TaskCanceledException: The operation was canceled. on Linux as well (for me it was 10ms for localhost). And I'd guess that if you make it higher you'll get System.Net.Http.HttpRequestException: Connection refused on Windows.
This probably boil down to different timing of things on Linux/Windows OSes.

BTW, the TimeoutException mentioned in the blog post doesn't apply here at all since that's throw as reaction to HttpClient.Timeout which is by default 100 seconds, much higher than your connection timeout.

@ManickaP

@ManickaP , I tried to omit custom cancellation token but result is the same. Increasing connection timeout doesn't change the picture, exception still different on various platforms. Also, I checked Timeout property of the client and if it's about equal to ConnectTimeout then sometimes I can get TaskCanceledException with TimeoutException, sometimes not. ConnectTimeout should be less than or equal to Timeout value, otherwise it makes no sense. If they are equal and you're lucky then you can get TimeoutException. In some situations, connection timeout must be less than request timeout (in my case it's an implementation of Raft consensus protocol).

I'm trying to say that the pattern

catch (OperationCanceledException e) when (!token.IsCancellationRequested)

cannot be fully replaced with

catch (OperationCanceledException e) when (e.InnerException is TimeoutException)

Moreover, this replacement is unreliable and dangerous.

Another main concern that exception type varies depends on the platform.

@stephentoub

Triage: The Linux behavior is by design -- Socket fails on Linux, reporting inability early.

On Windows (and also on Linux) we should have InnerException with TimeoutException if we hit ConnectTimeout. We should address that part.
We should check if there are more timeouts (likely not per quick search by @stephentoub).

It should be fairly straightforward to do - use same pattern we have for the implemented HttpClient.Timeout.

If you make the ConnectionTimeout smaller, you'll get System.Threading.Tasks.TaskCanceledException: The operation was canceled. on Linux as well (for me it was 10ms for localhost). And I'd guess that if you make it higher you'll get System.Net.Http.HttpRequestException: Connection refused on Windows.
This probably boil down to different timing of things on Linux/Windows OSes.

BTW, the TimeoutException mentioned in the blog post doesn't apply here at all since that's throw as reaction to HttpClient.Timeout which is by default 100 seconds, much higher than your connection timeout.

I met this problem too, and I've done this before I saw this issue. My tests failed on GitHub(ubuntu) env but pass in Windows with HttpRequestException in .net 5.0.

ref #21965, ref #24392

And, yes, as it is a frequency component that it should be the same pattern (exception) in the different platforms or it makes ppl confused.

Navigation Menu

Repository navigation

SocketsHttpHandler: Timeout from ConnectTimeout should use timeout exception pattern #47484

Description

Configuration

Regression?

Other information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SocketsHttpHandler: Timeout from ConnectTimeout should use timeout exception pattern #47484

Description

Description

Configuration

Regression?

Other information

Activity

sakno commented on Jan 26, 2021

wfurt commented on Jan 26, 2021

sakno commented on Jan 26, 2021

sakno commented on Jan 26, 2021

wfurt commented on Jan 26, 2021

sakno commented on Jan 27, 2021

ManickaP commented on Jan 27, 2021

sakno commented on Jan 27, 2021

karelz commented on Jan 28, 2021

nov30th commented on Apr 13, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions