Direct Methods 'Not Found' under load in IoT Hub

@Iain White Can you provide us more information to the below questions to helps us better understand the behavior you notice

• When you say ~10 out of 10000 fail, did you notice the 10 calls fail consecutively
• Is it always 404 348 the response you get for all the 10 failed cases. 404 is the status code being returned here and 348 is part of response payload. I suspect it could be timeout in seconds.
• What is the interval between each call you are make to the direct method
• Did you set any explicit responseTimeoutInSeconds as you make the call
• Can you share the snippet of the code on how you are invoking the direct method
• Can you call the ping direct method on the Iot Edge module
• Capture additional debug logs by using support-bundle and see if the logs capture any additional information

If the calls to the direct method resume after a slight delay, it sounds like a transient error on IoT Edge end. Try to increase the timeout on the direct method call by increasing the responseTimeoutInSeconds parameter. Please refer the example for the following document on how to make this call. Incorporate a retry mechanism and try to make a call after a slight delay when error occurs.

@Iain White Did you get a chance to see Leela Suggestions?
Please let us know if you are still blocked.

Thanks for the assistance @LeelaRajeshSayana-MSFT .

When failures happen they there tend to be a number of failures consecutively. Seems to me like there's some kind of blip in the handling of the Direct Method requests at the Azure side. Maybe while a refresh of config or something happens.

My current theory is that this is a issue at the IoT Hub because I can't see any failures in my edge module. I see all the successful calls. Judging by the 404 'Not Found' response, the IoT Hub seems to think the module doesn't exist and is 'Not Found' for a short period. This response is coming from the IoT Hub. It isn't coming from my module because it isn't reaching my module.

It is always 404 348, but I think after investigation this is the number of bytes in the response and nothing more insightful.

To rule out connectivity issues between our Edge Device and the IoT Hub, I've also tested on a Azure hosted VM running as an edge device. We see exactly the same sporadic 'Not Found' failures in Direct Methods calls when the load test are pointed towards it. We send around 5 Direct Method calls a second.
I didn't set specific a timeout - but since I am getting a 404 I don't expect this to be related to timeouts. If it is then I should be getting a timeout response code, not a 404, right?

Sorry - I don't understand why you're asking if I am able to call the ping Direct Method. I am able to call my actual Direct Method on the vast majority of occasions. But if it does help, yes I can call the 'ping' direct method on the edgeAgent.

We do have a robust retry mechanism implemented, but I want to find out why this is happening at relatively low load. Our system needs to be able to potentially handle hundreds of Direct Methods a second. I need to be sure it can be handled by our module and also by the IoT Hub.

Here's the class I'm using to invoke the Direct Methods. As you can see it uses the azure-iot-hub python module.

A bit more info @LeelaRajeshSayana-MSFT after I've added more logging.

• When we get these 'Not Found' responses, the requests are actually reaching the edge module and our edge module is sending a success response, within the same timeframe as all of the other responses (less than a second).
• When the issue occurs, it seems we get a few 200 responses but with empty payloads and a response.status of '0'- in the most recent example there were 3 of these. Then we got a 19 404's with a response.status of 'Not Found'. This happened within a 2 second window and then it was fine again.

We've been running these load tests a lot now and the issue only seems to happen every hour or so.

Today it happened at 09.17 and 10.27.

I am still trying to see if this is an issue on our side, but it would be helpful to know the extent of the load testing that has been done already at the Azure IoT Hub side by Microsoft.

Why I think this is an issue at the Azure side -

• My module logs show no errors and for every failure I can see in the logs the particular unique command ID being received and responded to.
• The Azure side Direct Method logging reports '"The operation failed because the requested device isn't online. To learn more, see https://aka.ms/iothub404103\"' but the device clearly is online as I can see it handling the requests in the module logs.