Azure IoT Hub - Total number of messages used doubled unexpectedly - What is the reason?

Question

Azure IoT Hub - Total number of messages used doubled unexpectedly - What is the reason?

Nikolay Valchanov 5

Recently we encountered a problem with IoT Hub related to hitting the daily quota for total number of messages used while the amount of devices and telemetry messages remained the same.

Screenshot 2025-04-11 at 16.05.08 Purple: Total number of messages used (Max) | Blue: Telemetry messages sent (Sum)

The size of the telemetry messages is stable as well. Around that period (but later than the peak has happened) new message route to storage container was created, however according to MS Docs routing should not count towards the daily message quota.

There weren't any additional C2D messages as well. Would you have any clue what could have happened?

Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-11T16:12:33.2033333+00:00
Hi Nikolay Valchanov

There are several potential reasons why the total number of messages used in Azure IoT Hub might have doubled unexpectedly: Below are areas we can check in diagnostics setting for clue.

Device-to-Cloud File Uploads: One common cause is an increase in device-to-cloud file uploads. Each file upload is treated as a message, and large files may be split into multiple messages, each counted towards the message quota

Device-to-Cloud Messages:

Twin Updates: IoT Hub operations such as twin reads and updates are also charged as messages. A sudden increase in these operations can lead to a higher message count

Cloud-to-Device Messages: Messages sent from the cloud to devices, including property updates, can contribute to the total message count. Even if telemetry messages are not being sent, these other types of messages can increase the usage.

Module Identity Operations: Operations related to module identities, such as updates from $edgeAgent and $edgeHub, can also contribute to the message count

Job Operation

To diagnose the issue, you can enable Diagnostic setting with all metric params and Check Diagnostic Logs with Kusto query from logs tab of your IOT hub/log analytic workspace

Here is sample Kusto query to find d2c, c2d and twin operations, feel free to modify to analyze

AzureDiagnostics | where ResourceType == "IOTHUBS" | where Category == "IoTHubD2COperations" or Category == "IoTHubC2DOperations" or Category == "IoTHubTwinOperations" | summarize Count = count() by Category, bin(TimeGenerated, 1h) | order by TimeGenerated asc

Hope it helps.

Reference thread

Thank you.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-14T07:18:30.21+00:00

Hi Nikolay Valchanov

We are checking if the pointer shared helped debug the issue. Please let us know if you still need assistance on it.

Thank you.
Nikolay Valchanov 5 Reputation points

2025-04-14T10:49:43.7333333+00:00

Thank you for the answer @Sander van de Velde | MVP

Messages used is counted in chunks so it seems the message size/content has changed.

I had the same thought, however after investigating the Cosmos DB where the data is routed, it seems like there is no change in size, even the messages that were sent earlier were sometimes larger.

Size of the body of D2C message 30-03-2025 (before the increase of messages used)

Size of the body of D2C message 11-04-2025 (after increase of messages used)

C2D communication has is almost not used in our case. Exception is the twin file reads / writes, however that was stable as well.
Nikolay Valchanov 5 Reputation points

2025-04-14T12:07:42.4833333+00:00

Hi Manas Mohanty,

Thank you for the response, I will enable collection of diagnostic logs and will investigate the results. After I have them I will include them in my further response.
Sander van de Velde | MVP 36,131 Reputation points MVP

2025-04-14T16:10:49.0633333+00:00

Hello @Nikolay Valchanov ,

This is probably the body of the message being sent.

Keep in mind the (user) properties are also taken into account.

Please consult your developer if there are any changes regarding the properties.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-15T13:16:15+00:00

Hi Nikolay Valchanov

Please let us know if you have been able to find out the cause of this spike from diagnostic setting.

Thank you.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-16T08:10:56.87+00:00

Hi Nikolay Valchanov

We have not heard from you. Hope the pointers shared were useful to you.

Thank you.
Nikolay Valchanov 5 Reputation points

2025-04-17T09:12:55.5933333+00:00

@Manas Mohanty I have enabled the diagnostic settings with all metrics, however executing suggested query didn't had any results.The following diagnostic settings were enabled

Result of running suggested query

All diagnostic logs

Would you have any suggestions?
Nikolay Valchanov 5 Reputation points

2025-04-17T09:25:45.52+00:00

@Sander van de Velde | MVP I have discussed that topic with embedded software team and there were no changes in the data that was sent from the devices to the cloud.

I investigated the full size of the messages and they are similar.

Size of the full payload in Cosmos DB Before daily usage increase

Size of the full payload in Cosmos DB After daily usage increase
Nikolay Valchanov 5 Reputation points

2025-04-17T09:28:44.4733333+00:00

Hi @Manas Mohanty

Unfortunately inspecting the diagnostic logs didn't gave expected result. Please see my answer in the topic where they were suggested for more details.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-17T09:35:26.83+00:00
Hi Nikolay Valchanov

As per your screenshot, I see a lot of "connections and D2Ctwinoperations"

Please check with Category != " " and select "custom time range" between April 1 and 5 and Between "March 23 and April 1" to find counts on respective data ranges and operations adding up total message limit

AzureDiagnostics | where ResourceType == "IOTHUBS" | where Category != "" | summarize Count = count() by Category, bin(TimeGenerated, 1h) | order by TimeGenerated asc

If you find any spike in any operation in April, you should check in that direction.

As an additional step, if you see Bin counts for operation did not change after April 1 but charts are different, you can unregister and register below registration providers to get accurate results.

Navigate to the subscription on the Azure portal, find Resource Providers under the Settings section.

Filter Microsoft.Devices and hit Re-register

Filter microsoft.insights and hit Re-register

Wait for 24 hours for this registration to propagate and try querying the metrics

Thank you.
Nikolay Valchanov 5 Reputation points

2025-04-17T09:47:54.7833333+00:00

Hi @Manas Mohanty

Unfortunately collection of diagnostic metrics was enabled after your suggestion on 14.04.2025 therefore there isn't any data before that date.
Sander van de Velde | MVP 36,131 Reputation points MVP

2025-04-17T15:17:08.6133333+00:00

Hi Nikolay Valchanov,

If the telemetry message size did not change, It's either that some devices have become more 'chatty' or cloud-to-device messages has increased.

Please consider applying an Azure support ticket. Then, the support team can take a detailed look at the backend of your services.

1 answer

Your answer

Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-11T16:12:33.2033333+00:00

Hi Nikolay Valchanov

There are several potential reasons why the total number of messages used in Azure IoT Hub might have doubled unexpectedly: Below are areas we can check in diagnostics setting for clue.

Device-to-Cloud File Uploads: One common cause is an increase in device-to-cloud file uploads. Each file upload is treated as a message, and large files may be split into multiple messages, each counted towards the message quota

Device-to-Cloud Messages:

Twin Updates: IoT Hub operations such as twin reads and updates are also charged as messages. A sudden increase in these operations can lead to a higher message count

Cloud-to-Device Messages: Messages sent from the cloud to devices, including property updates, can contribute to the total message count. Even if telemetry messages are not being sent, these other types of messages can increase the usage.

Module Identity Operations: Operations related to module identities, such as updates from $edgeAgent and $edgeHub, can also contribute to the message count

Job Operation

To diagnose the issue, you can enable Diagnostic setting with all metric params and Check Diagnostic Logs with Kusto query from logs tab of your IOT hub/log analytic workspace

Here is sample Kusto query to find d2c, c2d and twin operations, feel free to modify to analyze

AzureDiagnostics | where ResourceType == "IOTHUBS" | where Category == "IoTHubD2COperations" or Category == "IoTHubC2DOperations" or Category == "IoTHubTwinOperations" | summarize Count = count() by Category, bin(TimeGenerated, 1h) | order by TimeGenerated asc

Hope it helps.

Reference thread

Thank you.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-14T07:18:30.21+00:00

Hi Nikolay Valchanov

We are checking if the pointer shared helped debug the issue. Please let us know if you still need assistance on it.

Thank you.
Nikolay Valchanov 5 Reputation points

2025-04-14T10:49:43.7333333+00:00

Thank you for the answer @Sander van de Velde | MVP

Messages used is counted in chunks so it seems the message size/content has changed.

I had the same thought, however after investigating the Cosmos DB where the data is routed, it seems like there is no change in size, even the messages that were sent earlier were sometimes larger.

Size of the body of D2C message 30-03-2025 (before the increase of messages used)

Size of the body of D2C message 11-04-2025 (after increase of messages used)

C2D communication has is almost not used in our case. Exception is the twin file reads / writes, however that was stable as well.
Nikolay Valchanov 5 Reputation points

2025-04-14T12:07:42.4833333+00:00

Hi Manas Mohanty,

Thank you for the response, I will enable collection of diagnostic logs and will investigate the results. After I have them I will include them in my further response.
Sander van de Velde | MVP 36,131 Reputation points MVP

2025-04-14T16:10:49.0633333+00:00

Hello @Nikolay Valchanov ,

This is probably the body of the message being sent.

Keep in mind the (user) properties are also taken into account.

Please consult your developer if there are any changes regarding the properties.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-15T13:16:15+00:00

Hi Nikolay Valchanov

Please let us know if you have been able to find out the cause of this spike from diagnostic setting.

Thank you.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-16T08:10:56.87+00:00

Hi Nikolay Valchanov

We have not heard from you. Hope the pointers shared were useful to you.

Thank you.
Nikolay Valchanov 5 Reputation points

2025-04-17T09:12:55.5933333+00:00

@Manas Mohanty I have enabled the diagnostic settings with all metrics, however executing suggested query didn't had any results.The following diagnostic settings were enabled

Result of running suggested query

All diagnostic logs

Would you have any suggestions?
Nikolay Valchanov 5 Reputation points

2025-04-17T09:25:45.52+00:00

@Sander van de Velde | MVP I have discussed that topic with embedded software team and there were no changes in the data that was sent from the devices to the cloud.

I investigated the full size of the messages and they are similar.

Size of the full payload in Cosmos DB Before daily usage increase

Size of the full payload in Cosmos DB After daily usage increase
Nikolay Valchanov 5 Reputation points

2025-04-17T09:28:44.4733333+00:00

Hi @Manas Mohanty

Unfortunately inspecting the diagnostic logs didn't gave expected result. Please see my answer in the topic where they were suggested for more details.
Manas Mohanty 3,125 Reputation points Microsoft External Staff

2025-04-17T09:35:26.83+00:00

Hi Nikolay Valchanov

As per your screenshot, I see a lot of "connections and D2Ctwinoperations"

Please check with Category != " " and select "custom time range" between April 1 and 5 and Between "March 23 and April 1" to find counts on respective data ranges and operations adding up total message limit

AzureDiagnostics | where ResourceType == "IOTHUBS" | where Category != "" | summarize Count = count() by Category, bin(TimeGenerated, 1h) | order by TimeGenerated asc

If you find any spike in any operation in April, you should check in that direction.

As an additional step, if you see Bin counts for operation did not change after April 1 but charts are different, you can unregister and register below registration providers to get accurate results.

Navigate to the subscription on the Azure portal, find Resource Providers under the Settings section.

Filter Microsoft.Devices and hit Re-register

Filter microsoft.insights and hit Re-register

Wait for 24 hours for this registration to propagate and try querying the metrics

Thank you.
Nikolay Valchanov 5 Reputation points

2025-04-17T09:47:54.7833333+00:00

Hi @Manas Mohanty

Unfortunately collection of diagnostic metrics was enabled after your suggestion on 14.04.2025 therefore there isn't any data before that date.
Sander van de Velde | MVP 36,131 Reputation points MVP

2025-04-17T15:17:08.6133333+00:00

Hi Nikolay Valchanov,

If the telemetry message size did not change, It's either that some devices have become more 'chatty' or cloud-to-device messages has increased.

Please consider applying an Azure support ticket. Then, the support team can take a detailed look at the backend of your services.

Answer 1

Hello @Nikolay Valchanov,

welcome to this moderated Azure community forum.

The stream of incoming messages seems constant but the 'message used' has more than doubled.

Messages used is counted in chunks so it seems the message size/content has changed.

An alternative is that the Cloud-to-Device communication has increased but that is less obvious.

Is this does not help, consider applying an Azure support ticket.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Share via

Azure IoT Hub - Total number of messages used doubled unexpectedly - What is the reason?

1 answer

Your answer