Thread count increases dramatically regardless of maxConcurrentRequests

[will-beta]

Here is my simple http triggered function app.
Function1.cs

    public class Function1
    {
        [FunctionName(nameof(Function1))]
        public static async Task RunAsync(
            [HttpTrigger("POST")]string body,
            [Queue("myqueue")]ICollector<string> queue,
            [Blob("mycontainer/{rand-guid}")]CloudBlockBlob cbc,
            ILogger logger
            )
        {
            try
            {
                logger.LogWarning(new EventId(-1, "FunctionStart"), $"{body?.Length}");

                if (body != null)
                {
                    try
                    {
                        await cbc.UploadTextAsync(body);
                    }
                    finally
                    {
                        queue.Add(cbc.Name);
                    }
                }

                logger.LogWarning(new EventId(-1, "FunctionEnd"), $"{body?.Length}");
            }
            catch (Exception gex)
            {
                logger.LogError(new EventId(-1, "FunctionError"), gex, body);
            }
        }
    }

host.json

{
  "version": "2.0",
  "functionTimeout": "00:05:00",
  "healthMonitor": {
    "enabled": true,
    "healthCheckInterval": "00:00:10",
    "healthCheckWindow": "00:02:00",
    "healthCheckThreshold": 6,
    "counterThreshold": 0.80
  },
  "extensions": {
    "http": {
      // the maximum number of outstanding requests that will be held at any given time. The default is unbounded (-1).
      "maxOutstandingRequests": -1,
      // the maximum number of http functions that will be executed in parallel. The default is unbounded (-1).
      "maxConcurrentRequests": 5,
      // The default is false.
      "dynamicThrottlesEnabled": true
    }
  }
}

In my opinion, the thread count should depend on the maxConcurrentRequests .
But it would go up to more than 300 even I had stopped creating new requests from JMeter. Then it took quite some time to decrease to 30 or so.
My VS2017 gave me the following information when I attached it to the process,

Making the function Sync solves the problem on my local computer, but it still happens when I deploy the app onto the Azure.
Function1.cs

    public class Function1
    {
        [FunctionName(nameof(Function1))]
        public static void Run(
            [HttpTrigger("POST")]string body,
            [Queue("myqueue")]ICollector<string> queue,
            [Blob("mycontainer/{rand-guid}")]CloudBlockBlob cbc,
            ILogger logger
            )
        {
            try
            {
                logger.LogWarning(new EventId(-1, "FunctionStart"), $"{body?.Length}");

                if (body != null)
                {
                    try
                    {
                        cbc.UploadTextAsync(body).Wait();
                    }
                    finally
                    {
                        queue.Add(cbc.Name);
                    }
                }

                logger.LogWarning(new EventId(-1, "FunctionEnd"), $"{body?.Length}");
            }
            catch (Exception gex)
            {
                logger.LogError(new EventId(-1, "FunctionError"), gex, body);
            }
        }
    }

The same thing recurs after I replace the await cbc.UploadTextAsync(body) with await Task.Delay(1000).
So, my question is : maxConcurrentRequests is not capable of limiting the thread count? How does it work exactly?

[kashimiz]

@fabiocav @brettsam

[jeffhollan]

Does your host.json actually have the // comments in it? May be throwing another error as that isn't valid JSON and only used for our docs.

[will-beta]

@jeffhollan Yes, my host.json actually have the // comments in it.
But I didn't find any exceptions on debugging , neither in my log files.

[brettsam]

The comments look like they're ignored by the JSON.NET parsing we do -- so that's not the issue.

Would you mind sharing your JMeter test run details with us so we can try to repro this?

/cc: @mathewc, the HTTP throttle expert

[will-beta]

Replacing the ICollector with IAsyncCollector solves the problem!

[will-beta]

@brettsam All I got from JMeter test run was a bunch of Timeout.

[brettsam]

That's an interesting finding; I'm glad you're moving in the right direction.

@mathewc -- although we'd almost always recommend using an IAsyncCollector, I don't quite understand why using an ICollector would seem to lock up here.

I did a quick JMeter run against this same set up last week and things seemed to run okay for me, but I may have been running with drastically different settings that @will-beta was. @will-beta -- would you mind sharing your JMeter test setup with us? We should see if we can reproduce this.

[mathewc]

Not sure. @mhoeger made fixes Azure/azure-webjobs-sdk#1658 for an issue similar to this. I wonder if this scenario is hitting codepaths not addressed by those fixes?

[will-beta]

@brettsam Here is my JMeter test setup file:
azure functionapp.zip

The function app behaves differently on the Azure vs on my local computer.
So to reproduce this, it's better to deploy the app onto the Azure.