Issues with SBE patching and updates with our system

Hello Joe Comeau, it looks like you're having an issue with the AzStackHciSBEHealth status showing as failed under the Azure Update Manager, this kind of problem is often related to update failures, particularly with the Solution Builder Extension (SBE) updates.

Verify the current status of the SBE by running the following PowerShell command to see if your SBE is installed correctly:

$Update = Get-SolutionUpdateEnvironment
$Update | ft SbeFamily, HardwareModel, CurrentSbeVersion, State

This will show you the state of the SBE. If it shows "UpdateFailed", you'll need to follow specific troubleshooting steps.

• If outbound connections to Azure Update Manager endpoints are restricted, update discovery and validation can fail. https://docs.azure.cn/en-us/azure-local/concepts/firewall-requirements
• If its single-node deployments, SBE patching can sometimes misreport status after updates complete https://github.com/Azure/AzureLocal-Supportability/blob/main/TSG/SolutionExtension/HealthCheck-known-issues/SBE-HealthCheck-content-integrity-failure.md
• If the SBE package itself is not applied correctly, the health check will remain in a failed state.

On the Azure Stack HCI cluster nodes:

• Open Event Viewer > Applications and Services Logs > Microsoft > Windows > HCI.
• Look under UpdateManager and ClusterUpdate channels for errors tied to SBE patching. C:\Windows\Logs\ClusterUpdate
• PowerShell diagnostic commands: Get-WinEvent -LogName "Microsoft-Windows-HCI/UpdateManager".

In the Azure Portal:

• Azure Update Manager > Update Readiness: The readiness check results often include error codes or failure reasons.
• Activity Log (subscription level: Go to Monitor > Activity Log and filter for your cluster resource. Failed operations related to updates or extensions will be listed here.

Check the relevant documentation for more guidance on troubleshooting SBE update issues and how to ensure your system is properly set up for updates.

SBE Update Failure

Discover Solution Builder Extension Updates

If you've tried all of this and it's still not resolved, could you help us with the below details:

• What specific error messages or codes are you seeing in the Azure Update Manager?
• Have you verified that all cluster nodes are operational?
• Did you perform any manual updates or sideload updates recently which could have impacted the SBE update process?

Thanks.

What specific error messages or codes are you seeing in the Azure Update Manager?

• Have you verified that all cluster nodes are operational?
• Did you perform any manual updates or sideload updates recently which could have impacted the SBE update process?

  yes, Manually downloaded as instructed on the azure update manager

  https://www.dell.com/support/kbdoc/en-ca/000224407/dell-for-microsoft-azure-stack-hci-ax-hardware-updates-release-notes

Hello Joe Comeau, Thanks for the update. Azure Update Manager runs pre-update checks for Azure Stack HCI clusters. The Environment Validator executes tests like: Test-AzStackHciConnectivity to verifies WinRM connectivity to cluster nodes and Test-AzStackHciSBEHealth to Checks Storage Bus Engine health. As per the error your shared, the failure indicates, the WinRM sessions cannot be established. SBE prerequisites (disk, network, cluster health) may not meet requirements.

1. Verify WinRM service status on each node: This ensures the core service responsible for remote management is operational on every node.

• Log into each cluster node.
• Run  Get-Service WinRM
• Ensure the service is Running and StartupType is set to Automatic.
• If not, start it with:

Start-Service WinRM
Set-Service WinRM -StartupType Automatic

2. Run the connectivity validator manually: This step validates the success of your WinRM fixes and checks for other critical environment dependencies.

• On each node, execute: Test-AzStackHciConnectivity
• This will attempt to validate WinRM sessions between nodes and outbound connectivity to Azure endpoints.
• Review the output for any failed connections or exceptions.

3. Check firewall rules for WinRM ports : WinRM relies on specific TCP ports. These rules must be open for inter-node communication.

• WinRM uses TCP 5985 (HTTP) and TCP 5986 (HTTPS).
• Run: Get-NetFirewallRule | Where-Object { $_.DisplayName -like "WinRM" }
• Confirm inbound rules exist for these ports.
• If missing, add them:

New-NetFirewallRule -Name "WinRM-HTTP" -DisplayName "WinRM over HTTP" -Protocol TCP -LocalPort 5985 -Action Allow -Direction Inbound
New-NetFirewallRule -Name "WinRM-HTTPS" -DisplayName "WinRM over HTTPS" -Protocol TCP -LocalPort 5986 -Action Allow -Direction Inbound

4. Validate outbound connectivity: If the SBE update fails late in the process, it might be an issue with reaching Azure services (like Update Manager).

• Ensure nodes can reach required Azure endpoints (Update Manager, Arc services).
• Test with: Test-NetConnection <endpoint> -Port 443
• Replace <endpoint> with the Azure service URLs documented for Azure Stack HCI. For more details: Firewall requirements for Azure Stack HCI | Azure Docs

5. Re-run the environment validator

• After fixing connectivity and firewall rules, run: Test-AzStackHciConnectivity
• Confirm all nodes pass validation before retrying updates.

Reference document: Firewall requirements for Azure Stack HCI | Azure Docs

Hope this helps, If you still encounter any issue, please do let us know. thanks

1. Verify WinRM service status on each node: This ensures the core service responsible for remote management is operational on every node 2. Run the connectivity validator manually: This step validates the success of your WinRM fixes and checks for other critical environment dependencies
   3. Check firewall rules for WinRM ports : WinRM relies on specific TCP ports. These rules must be open for inter-node communication. & 4.Validate outbound connectivity: If the SBE update fails late in the process, it might be an issue with reaching Azure services (like Update Manager). 5.Re-run the environment validator
   4. we re run the validator and everything is success as connections go.
      1. retried updating but still failing

           ![Still failed status](/api/attachments/fdf291b1-8333-4290-829a-244f3003e10e?platform=QnA)
         

         P.S thank you for a hasty response.
         1. Would love to have a remote session if possible.
         2. My systems admin will be working on my behalf and will use my account to continue communicating with you as I will be on medical leave to get this issue fixed, His name is Andy Chua

Best regards,

Joe

Hello Joe Comeau, Thanks for the update, we have reached out to you in Private messages for additional details, could you please look into it and share us the details? Thanks