Cluster communication audit and recovery

Chang Min 40 Reputation points
2026-09-04T07:35:25.3066667+00:00

Hi

I have a question about restoring cluster communication after a firewall configuration issue.

Recently, a perimeter firewall rule change inadvertently blocked UDP port 3343 between cluster nodes. This caused node eviction and triggered split-brain mitigation protocols.

I would like to know how to audit cluster heartbeats and re-establish inter-node cluster communication. Specifically, I’m looking for guidance on verifying NLB and cluster network health, and ensuring that the heartbeat path is trusted and reachable.

We are using Windows Server Failover Clustering in a production environment.

Windows for business | Windows 365 Business
0 comments

3 answers

Sort by: Oldest
Most helpful Newest Oldest

  1. AI answer

    2026-09-04T07:35:59.8466667+00:00

    Use this recovery and audit path for Windows Server Failover Clustering heartbeat issues on UDP 3343:

    1. Verify the required cluster ports are open again
      • Confirm UDP 3343 is allowed between all cluster nodes. This is the cluster heartbeat port.
      • Also verify related cluster traffic is not blocked: TCP 3343 during node join, RPC 135, SMB 445, and ICMP4/ICMP6 for validation.
      • If firewalls separate nodes, also review the RPC-related ranges noted for cluster communication.
    2. Audit the Windows Firewall rule on each node
      • Check the built-in cluster heartbeat rule: 
             netsh advfirewall firewall show rule name="Failover Clusters (UDP-In)" verbose
      • On affected Windows Server 2008 R2 systems, a known issue can block inbound UDP after network interruption and profile reload. In that case, inspect the rule and, if required, disable the Failover Clusters (UDP-In) rule so the Cluster service can re-enable node communication at startup.
      • GUI path: wf.msc → Inbound Rules → Failover Clusters (UDP-In).
      Warning: changing or disabling this rule can stop the Cluster service. If possible, stop the Cluster service before making the change, then restart it after the change.
    3. Test heartbeat reachability between nodes
      • Run Test-NetConnection for key ports, especially 3343.
      • Verify basic node-to-node connectivity with ICMP/ping.
      • If other nodes show failed connection attempts on port 3343, treat that as a network communication issue affecting quarantine or eviction.
    4. Review cluster and system evidence
      • In Event Viewer, check System and Failover Clustering logs for heartbeat failures and node removal events such as Event ID 1135.
      • For quarantined nodes, also review events such as 1641, 1647, and 1649.
      • Collect cluster logs with Get-ClusterLog and compare timestamps with the firewall change window.
    5. Validate cluster network health
      • Run Validate Configuration in Failover Cluster Manager and review the Network results for errors or warnings.
      • This helps confirm whether adapters, bindings, and cluster communication paths are healthy after the firewall correction.
    6. Check for common causes of continued heartbeat failure
      • Network hardware faults or packet loss between nodes.
      • Network profile changes between Domain and Public, which can block network I/O during transition. Review Event Viewer → Applications and Services Logs\Microsoft\Windows\NetworkProfile\Operational.
      • IPv6 enabled while these firewall rules are disabled:
        • Core Networking - Neighbor Discovery Advertisement
        • Core Networking - Neighbor Discovery Solicitation
      • Antivirus or security software blocking cluster traffic.
      • MTU mismatches across cluster adapters.
      • Outdated network adapter drivers or firmware.
      • Misconfigured RDMA, if used.
    7. Re-establish stable cluster communication
      • After restoring firewall access, bring the affected node online again.
      • Confirm the cluster can communicate over UDP 3343 and that the node no longer shows as Unreachable in Failover Cluster Manager.
      • If the node entered quarantine because of heartbeat loss, verify that required ports are open, packet loss is resolved, and firewall/security software is not blocking traffic before returning workloads.
    8. For Azure VM-based WSFC environments
      • If transient latency is contributing to heartbeat instability, use more relaxed heartbeat thresholds.
      • For Windows Server 2012-2019: 
             (get-cluster).SameSubnetThreshold = 40
     (get-cluster).CrossSubnetThreshold = 40
      • Verify: 
             get-cluster | fl *subnet*

    About NLB: the available material covers WSFC heartbeat and cluster networking, but not a supported audit procedure for NLB health in this scenario.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

  2. Gipfel Ehrlich 60 Reputation points
    2026-09-04T08:39:54.0766667+00:00

    interested too

    Was this answer helpful?

    0 comments
  3. Domic Vo 32,140 Reputation points Independent Advisor
    2026-09-04T08:43:14.5133333+00:00

    Hello,

    when UDP 3343 was blocked, the cluster lost its heartbeat channel and evicted nodes. To restore communication, first re‑enable bidirectional UDP 3343 between all nodes. Then audit with Get-ClusterNetwork to confirm networks are marked as “Cluster” or “ClusterAndClient,” and run Test-Cluster to validate firewall and network health. Use Get-ClusterLog -UseLocalTime to check NETFT entries and verify heartbeat packets are flowing again.

    Ensure Windows Firewall allows UDP/TCP 3343 and RPC for the Cluster service, and dedicate at least one trusted network for heartbeat traffic. Once rules are corrected, nodes will automatically rejoin and communication resumes.

    I hope you've found something useful here. If it helps you get more insight into the issue, it's appreciated to accept the answer. Should you have more questions, feel free to leave a message. Have a nice day!

    Domic Vo.

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.