CMG - Azure Virtual Machine Scale Set Cloud Management Gateway instance is Unhealthy in Azure

Hi Flint,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

It sounds like your customer is experiencing issues with their Cloud Management Gateway (CMG) in Azure. Here are some steps and considerations to help troubleshoot and resolve the issue:

• Make sure that the CMG connection point is correctly configured and connected to the primary site. Check that there is no network security groups or firewalls blocking the connection.
• Examine the logs for both the CMG connection point and the CMG itself. Look for any errors or warnings that might indicate the cause of the unhealthy state.
• Verify the health of the Azure VM Scale Set. Ensure all instances are running correctly and there are no issues with the underlying Azure infrastructure.
• Confirm that the certificates used by the CMG are valid and correctly configured. Expired or misconfigured certificates can lead to connectivity issues.
• Since the Test-NetConnection for ports 10124 and 10125 is successful, it indicates that the primary server can reach the CMG. Ensure all required ports and protocols are open and not blocked by any network security groups or firewalls.
• If the issue persists, consider deleting and recreating the CMG. Follow all steps correctly during the setup process to ensure proper configuration.

For more detailed guidance, please refer to the documentation on setting up and troubleshooting CMG:

• Set up CMG - Configuration Manager
• Plan for CMG - Configuration Manager
• CMG performance and scale

If you have any further queries, do let us know. If the comment is helpful, please click "Upvote".

Hi Sai,

1. Its previously working fine without any issue.
2. I have checked the log files for certification expiry from Client Machine( C:\Windows\CCM\Logs\CcmMessaging.log), but it's looks good.
3. We have configured the 2 VM Instances in CMG. After deleting the unhealthy VM instance in Azure VM Scale Set, it will create again the Unhealthy VM instance.
4. Analyze the Cloudmgr.log log file, but not able to find out any error.
5. We analyze the smsAdminUI.log, get the below error in Port 443, but firewall is disabled on the SCCM Primary Server where CMG configured. Check with Network team they also confirmed no blockage in 443 port.

smsAdminUI.log

[13, PID:28384][10/01/2024 18:33:01] :System.Net.WebException\r\nUnable to connect to the remote server\r\n at System.Net.HttpWebRequest.GetResponse()

at Microsoft.ConfigurationManagement.AdminConsole.AzureServices.CMGAnalyzer.backgroundWorker_DoWork(Object sender, DoWorkEventArgs e)\r\nSystem.Net.Sockets.SocketException\r\nA connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xxx.xxx.xxx:443\r\n at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)\r\n

Hi Flint,

Thank you for the additional information. Since the CMG was previously functioning without issues and you have confirmed that the certificates are valid and there are no port 443 blockages, here are further steps to troubleshoot:

Verify the Health of the Azure VM Scale Set (VMSS):
Although you have deleted the unhealthy VM instance, it seems the new instance is still being marked as unhealthy. Please check the health of the entire VM Scale Set in Azure.

Go to the Azure Portal > Virtual Machine Scale Sets > Your CMG instance > Instances. Ensure all VM instances are running properly and there are no errors related to the infrastructure.

Azure Virtual Machine Scale Sets - Troubleshooting

Network Security and Firewall:
Although the firewall on your SCCM Primary Server is disabled, verify that there are no issues with the Network Security Groups (NSGs) in Azure. Ensure that NSGs allow port 443 for inbound and outbound traffic between the SCCM server and the CMG in Azure.

Network security groups (NSGs) in Azure
Azure firewall logs and troubleshooting

Azure Load Balancer Configuration:
If your CMG uses an Azure Load Balancer, ensure that the health probes are correctly configured to check the VM instances' status. Misconfigured health probes may cause the VMSS instances to report as unhealthy.

Azure Load Balancer Troubleshooting

CMG Analyzer Tool:
I recommend using the CMG Analyzer tool in the SCCM console to check the connectivity between the CMG, SCCM, and Azure. This will help identify any underlying connection issues. The error you mentioned in smsAdminUI.log may indicate a transient network or infrastructure issue.

Check DNS and Resource Limits:
Ensure that DNS resolution between the SCCM server and the CMG is working correctly. Also, check your Azure subscription quotas and limits to confirm that resource constraints are not causing the issue.

For additional information, please refer to the following documentation: Monitor the Cloud Management Gateway - Configuration Manager

If you have any further queries, do let us know. If the comment is helpful, please click "Upvote".