Azure IOT Hub Service Client SDK Issue --> Invoke Device Method fails after sixteen method calls

I am having issues with C# Microsoft.Azure.Devices Service SDK (current version 1.38.0) and experienced the same issue with former version1.77.1

I have a .NET 6 Core Web API project that is using the SDK to invoke device methods on the IOT Device (IOT Device is a seperate .NET 6 Worker Project)

Issue:

After invoking a device method exactly sixteen times in a row, the SDK returns an invalid error code 504101 within the try/catch block I'm using to handle the exception from the SDK. I have repeated the tests a number of times and perpelxed why it fails on the 17th time I invoke the method. The only way to resolve this is re-starting the project, then repeat the test and I then have the same issue.

Below is the Repository in my Web Api project that I use for calling the IOT Hub with the service client:

public class CloudToDeviceRepo : ICloudToDeviceRepo
{
    private readonly IConfiguration _config;
    private readonly ILogExtension _logExtension;
    private static ServiceClient? _serviceClient;
    private readonly string _primaryConnectionString;

    private static string? LogData { get; set; }

    public CloudToDeviceRepo(
    IConfiguration config,
    ILogExtension logExtension)
    {
        _config = config;
        _logExtension = logExtension;
        _primaryConnectionString = _config.GetSection("AzureIotHub").GetValue<string>("PrimaryConnectionString");
        _serviceClient = ServiceClient.CreateFromConnectionString(_primaryConnectionString);
    }

    /// <summary>
    /// Example methodName = "RestartIotDevice"
    /// </summary>
    /// <param name="deviceId"></param>
    /// <param name="methodName"></param>
    /// <param name="methodPayload"></param>
    /// <param name="responseTimeoutSeconds"></param>
    /// <returns></returns>
    public async Task<CloudToDeviceMethodResult> InvokeDeviceMethod(string deviceId, string methodName, string methodPayload, int responseTimeoutSeconds)
    {
        var methodInvocation = new CloudToDeviceMethod(methodName)
        {
            ResponseTimeout = TimeSpan.FromSeconds(responseTimeoutSeconds),
        };
        methodInvocation.SetPayloadJson(methodPayload);

        try
        {
             var response = await _serviceClient!.InvokeDeviceMethodAsync(deviceId, methodInvocation);
             return response;
         }
         catch (Exception ex)
         {
             LogData = "IotHubService-API encountered an exception when invoking a cloud to device method with method name [" + methodName + "] for an iot device with id [" + deviceId + "]. Azure IOT Hub returned an error with message: [" + ex.Message + "]";
             await _logExtension.WriteLogEvent<CloudToDeviceRepo>("Error", "IOT Devices", "Invoke Device Method", deviceId, LogData, null!);
            throw;
        }
    }
}

In my program startup, I run the repository as a singleton service (and had the same issue using transient or scoped)

// Service for calling Azure IOT Hub
builder.Services.AddSingleton<ICloudToDeviceRepo, CloudToDeviceRepo>();

Below is a sample method handler in my IOT Device Project. I am getting the same issue with the service client no matter what device method I call on the IOT Device from the Hub, using the Service Client SDK.

private async Task<MethodResponse> TestMethod(MethodRequest methodRequest, object userContext)
{
    // Acknowlege the direct method call with a 200 success message
    string result = $"{<!-- -->{\"result\":\"Executed direct method: {methodRequest.Name}\"}}";
    //return Task.FromResult(new MethodResponse(Encoding.UTF8.GetBytes(result), 200));
    return new MethodResponse(Encoding.UTF8.GetBytes(result), 200);
}

Below is the details of the exception I receive in my Web API after invoking the device method after sixteen times:

When inpecting the exception, it seems to be that the issue resides from something within the SDK, related to the HttpClientHelper:

Exception Message:

{"{"Message":"{\"errorCode\":504101,\"message\":\"Timed out waiting for the response from device.\",\"trackingId\":\"EFEA228A0E5F4E96932DB46DCB36D9B8-G2:-TimeStamp:2022-08-07T10:32:51.149877054+00:00\",\"timestampUtc\":\"2022-08-07T10:32:51.149877054+00:00\",\"info\":{\"timeout\":\"00:00:30\"}}","ExceptionMessage":""}"}

ex.StackTrace

at Microsoft.Azure.Devices.HttpClientHelper.<ExecuteAsync>d__36.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.HttpClientHelper.<ExecuteWithCustomOperationTimeoutAsync>d__33.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Devices.HttpClientHelper.<PostAsync>d__262.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at IotHubService_API.Repositories.AzureIotHub.CloudToDeviceRepo.<InvokeDeviceMethod>d__9.MoveNext() in G:\My Drive\MyProject\Repositories\AzureIotHub\CloudToDeviceRepo.cs:line 48

ex.SerializationStackTraceString

at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.ExecuteWithCustomOperationTimeoutAsync(HttpMethod httpMethod, Uri requestUri, TimeSpan operationTimeout, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsyncT,T2 at IotHubService_API.Repositories.AzureIotHub.CloudToDeviceRepo.InvokeDeviceMethod(String deviceId, String methodName, String methodPayload, Int32 responseTimeoutSeconds) in G:\My Drive\MyProject\Repositories\AzureIotHub\CloudToDeviceRepo.cs:line 48

I really dont know to how to troubleshoot this further, I had checked the diagnositics health reports in the Azure portal for the IOT Hub account, but no issues were reported. I do beleive the issue lies somewhere at my end i.e. calling the Azure Hub using the Service Client SDK from my Web Api project. The fact that I can suscessfully invoke a device method the first sixteen times without issue would have me beleive there is nothing wrong with my own code implemtation, but is very stange how the SDK returns the exception on the 17th run each time before I restart the project. I'm wandering whether the SDK is suffering from some form of socket exhaustion and somehow not handling the HttpClient connection correctly.

Update 1 --> In order to elimiante anything on the client side in my Web Api project when calling the Service Hub client, I used a sample template from the IOT Hubs samples.

I am still getting the same issue after invoking an IOT device method after calling it more than sixteen times in a row, leaving a two second gap in between each call.

Console Application code sample below:

using System;
using System.Threading.Tasks;
using Microsoft.Azure.Devices;

namespace IotHubServiceClientTest
{
    /// <summary>
    /// This sample illustrates the very basics of a service app invoking a method on a device.
    /// </summary>
    internal class Program
    {
        private static ServiceClient s_serviceClient;

        private static async Task Main(string[] args)
        {
            Console.WriteLine("IoT Hub Quickstarts - InvokeDeviceMethod application.");

            // Create a ServiceClient to communicate with service-facing endpoint on your hub.
            s_serviceClient = ServiceClient.CreateFromConnectionString("HostName=myservicehubconnectionstring");

            for (int i = 1; i < 18; i++)
            {
                Console.WriteLine(i);
                await InvokeMethodAsync("F4-7B-09-A1-13-64");
                await Task.Delay(2000); // Wait 2 secs before next call.
            }

            s_serviceClient.Dispose();

            //Console.WriteLine("Press Enter to exit.");
            Console.ReadLine();
        }

        // Invoke the direct method on the device, passing the payload.
        private static async Task InvokeMethodAsync(string deviceId)
        {
            var methodInvocation = new CloudToDeviceMethod("SetTelemetryInterval")
            {
                ResponseTimeout = TimeSpan.FromSeconds(30),
            };
            methodInvocation.SetPayloadJson("10");

            Console.WriteLine($"Invoking direct method for device: {deviceId}");

            try
            {
                // Invoke the direct method asynchronously and get the response from the simulated device.
                var response = await s_serviceClient.InvokeDeviceMethodAsync(deviceId, methodInvocation);

                Console.WriteLine("");
                Console.WriteLine($"Response status: {response.Status}, payload:\n\t{response.GetPayloadAsJson()}");
            }
            catch (Exception ex)
            {
                Console.WriteLine("");
                Console.WriteLine($"Exception message: {ex.Message}");
                Console.WriteLine("");
                Console.WriteLine($"Exception inner exception: {ex.InnerException}");
                Console.WriteLine("");
                Console.WriteLine($"Exception stack trace: {ex.StackTrace}");
            }
        }
    }
}

Below is a screenshot showing the output error given from the service client:

I will keep troublshooting, I guess my next step will be to use a sample app code for the IOT device end in order to receive the request invoking the device method. Weird thing is that no errors or exceptions are noted in my IOT evice app, all that happens is on the 17th time I invoke a device method, the device method handler simply dosen't run at all which Is noteably why I get a timeout exception returned from the service client sdk.

Update 2 --> I have now implemented a test IOT Device client, again using an IOT Hubs code sample. I am still getting the issue so can now confirm that using the most basic example of invoking an IOT Device method fails after the first sixteen iterations.

Below is the code sample for the IOT Device Client:

namespace IotDeviceClientTest
{
    internal class Program
    {
        private static DeviceClient s_deviceClient;
        private static readonly TransportType s_transportType = TransportType.Mqtt;

        private static async Task Main(string[] args)
        {
            Console.WriteLine("IoT Hub Quickstarts #1 - Simulated device.");

            // Connect to the IoT hub using the MQTT protocol
            s_deviceClient = DeviceClient.CreateFromConnectionString("HostName=myIotHubDeviceClientConnectionString", s_transportType);

            // Create a handler for the direct method call
            await s_deviceClient.SetMethodHandlerAsync("SetTelemetryInterval", SetTelemetryInterval, null);

            Console.ReadLine();
        }

        private static Task<MethodResponse> SetTelemetryInterval(MethodRequest methodRequest, object userContext)
        {
            var data = Encoding.UTF8.GetString(methodRequest.Data);

            // Acknowlege the direct method call with a 200 success message.
            string result = $"{<!-- -->{\"result\":\"Executed direct method: {methodRequest.Name}\"}}";

            Console.WriteLine("");
            Console.WriteLine(result);
            
            return Task.FromResult(new MethodResponse(Encoding.UTF8.GetBytes(result), 200));
        }
    }
}

Update 3 --> My next test was to invoke the device method directly from the Azure IOT Hub portal, again I get the same issue so we can now rule out the IOT Hub Service Client SDK. As above, invoking the device meothod fails after 16qty times.

Screenshot shows the portal is waiting once it's sent the 17th method invoke request:

Screenshot shows the error message on the 17th method invoke:

Other troubleshooting steps I've tried:

Tried registering a new IOT Device in the Azure portal, and tested invoking methods directly from the hub for that device. Changed the Device Id to a simple name "TestDevice1" in case the SDK didnt like the fact I was using a MAC address as the id. Tried experiementing with using various different Reponse Timeout periods I fired the first 16qty device method invokes, then waited a few mins before firing off the 17th method invoke, the 17th call still failed.

My overall conclusion so far as follows:

• There is either an issue with Azure IOT Hubs for my particular account
• Or there is an issue with the Azure IOT Hubs C# Device Client SDK (Microsoft.Azure.Devices.Client current ver.1.41.1 but I have tried previous versons of the SDK as well, going down as far as 1.30.0)
• Or there is some limitation I'm not aware of in being able to invoke device client messages more than 16qty times in a row, in a fairly short duration of 2 seconds apart from each call (seems unlikely going by the MS throttling quotas which are well within the tolerences of my basic testing)