Service Fabric: Cluster Management Operations Failing After Certificate Rotation — Stuck Node Types

Question

Service Fabric: Cluster Management Operations Failing After Certificate Rotation — Stuck Node Types

David-1839 0

Problem description

I am experiencing issues with my Service Fabric Managed Cluster after performing a cluster certificate rotation. Specifically, node-type management operations such as scale, resize, create, and delete are failing, although the cluster itself remains healthy and continues to serve traffic.

Environment

Service Fabric Managed Cluster, multiple node types, region not specified in case information.

What I've already tried

I analyzed the issue as a certificate-rotation and configuration mismatch rather than a runtime outage. I reviewed the rotation date, the configured thumbprint list, and the node certificate coverage, noting that the stale thumbprint was still present on some nodes. I also reviewed the public Service Fabric certificate-management guidance to understand rotation behavior and thumbprint handling. No additional diagnostic outputs or support findings have been recorded yet.

Current status

The cluster remains operational and serving traffic, but node-type provisioning is blocked by a certificate mismatch. I am seeking guidance on how to reconcile the cluster certificate thumbprints and resolve the management operation failures.

0 comments

1 answer

Your answer

Answer 1

Hello David-1839,

Thank you for reaching out to us on Microsoft Q&A forum.

Based on the symptoms, the issue appears to be associated with the certificate state following the rotation. However, the current information is not sufficient to confirm that the obsolete thumbprint itself is the root cause.

A Service Fabric cluster can remain healthy and continue serving workloads while node-type provisioning operations, such as scale, resize, create, or delete, fail in the management plane. Certificate-based authentication also requires the certificate’s private key to be present and accessible, not only for the certificate thumbprint to match the cluster configuration. X.509 certificate-based authentication in Service Fabric

The following validations are recommended:

Review the failed node-type operation in Resource group > Deployments and the Azure Activity log. Record the complete error, UTC timestamp, status code, and activity or correlation ID.
Confirm that the newly provisioned certificate is valid, not expired or revoked, and contains a usable private key.
If the failure includes WindowsCryptographicException:`` ``Keyset`` ``does`` ``not`` ``exist, investigate whether the certificate’s private key is missing or inaccessible to the provisioning process. This exception alone does not confirm that removing an old thumbprint is the correct remediation.
Verify that the certificate rotation completed successfully and that the intended certificate version is available for use by Service Fabric.
Avoid manually PATCHing the managed-cluster resource or directly removing values from clusterCertificateThumbprints. The public certificate add/remove procedures are primarily documented for classic Service Fabric cluster resources, and removing a certificate that is still selected or required could affect cluster authentication. Manage certificates in a Service Fabric cluster

Microsoft documents that Service Fabric may select the valid certificate with the furthest future expiration date. Therefore, the active certificate should be established before removing or changing any certificate reference. Manually roll over a Service Fabric cluster certificate

A Microsoft support ticket is already open for this issue, and the assigned support engineer is actively investigating the failed node-type provisioning operations. Please continue sharing the relevant UTC timestamps and activity or correlation IDs through the existing support case. Any validated remediation or required managed-cluster reconciliation will be communicated through that ticket.