Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Wednesday, November 6, 2019 9:28 PM
Evening all,
We've recently ran through an in-place upgrade of our Cluster servers from 2016 to 2019 Datacenter edition. It all went very smooth and everything seemed to be working.
However - a week later and suddenly the cluster is failing! A validation report shows that the three nodes cannot talk to each other on any cluster network via UDP Port 3343.
I've opened the necessary ports on the firewall, in addition to what HyperV probably adds - but it's still happening. I've restarted the machines - still happening. I've used Telnet on the servers to test they are open - and they are.
This was working fine as 2016 - so something has gone wrong somewhere.
I'd be grateful if anyone could help or offer advice on what to look for.
Thanks
Gareth
All replies (9)
Wednesday, November 6, 2019 9:42 PM
Here is the ipconfig /all from one of the servers (they are identical):
Windows IP Configuration
Host Name . . . . . . . . . . . . : ygg-hyperv1
Primary Dns Suffix . . . . . . . : school.education.swansea.sch.uk
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : school.education.swansea.sch.uk
education.swansea.sch.uk
Tunnel adapter Local Area Connection* 2:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter
Physical Address. . . . . . . . . : 02-C2-1D-33-44-E6
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::c36:92a3:8fd7:6258%21(Preferred)
IPv4 Address. . . . . . . . . . . : 169.254.1.170(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . :
DHCPv6 IAID . . . . . . . . . . . : 318903973
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-25-44-A1-24-6C-3B-E5-A6-96-CA
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter Live Migration Team:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft Network Adapter Multiplexor Driver #2
Physical Address. . . . . . . . . : E4-11-5B-97-29-59
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.0.0.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.224
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter CSV-Cluster:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : HPE Ethernet 1Gb 4-port 366i Adapter #2
Physical Address. . . . . . . . . : 6C-3B-E5-A6-96-CA
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 192.168.0.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter iSCSI-1:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : HPE Ethernet 1Gb 4-port 366i Adapter #3
Physical Address. . . . . . . . . : 6C-3B-E5-A6-96-C8
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.10.10.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.224
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter iSCSI-2:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : HPE Ethernet 1Gb 2-port 361T Adapter #6
Physical Address. . . . . . . . . : E4-11-5B-97-29-58
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 30.30.30.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.224
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter Management:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : HPE Ethernet 1Gb 2-port 361T Adapter
Physical Address. . . . . . . . . : 48-DF-37-34-98-4F
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.154.26.19(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
IPv4 Address. . . . . . . . . . . : 10.154.26.25(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.154.26.254
DNS Servers . . . . . . . . . . . : 10.163.192.105
10.163.192.106
NetBIOS over Tcpip. . . . . . . . : Enabled
The VMTraffic network cards are not showing up for some reason here. They are teamed together.
Also I cannot ping the CSV/Cluster networks between servers. I am getting 'TTL Expired in Transit' errors - for example when I ping 192.168.0.2 from 192.168.0.1
Gar
Thursday, November 7, 2019 7:01 AM
Hi Gar,
Please first update the NIC driver and patch your system with fully windows updates.
If the issue still happens, please run the following command to do a check:
Netsh int tcp set global chimney=disabled
Netsh int tcp set global netdma=disabled
netsh int tcp set global rss=disabled
And disable checksum offloads and Rss on all adapters:
Disable-NetAdapterChecksumOffload -Name "XXXX"
Disable-NetAdapterRss -name "XXXX"
Best Regards,
Candy
Please remember to mark the replies as an answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]
Thursday, November 7, 2019 11:47 AM
Hi Candy,
Drivers Updated : Check
All Windows Updates : Check
Nothing worked.
So... I ran the commands. The first command didn't work so I had to it through each individual network card setting. Took a look on Google and found this: http://lifeofageekadmin.com/how-to-disabling-tcp-chimney-offload-rss-and-netdma-in-windows-2008-r2/ Hopefully I've done that right.
The other command were all run on the three servers and they have been restarted in order. 1, 2 and then 3.
Still getting the same issue (as shown in this screenshot)
:-(
Thursday, November 7, 2019 12:38 PM
Attached are the Netstat outputs from all three servers. From these I think I can see that port 3343 is open
Server 1
Server 2
Server 3
What am I missing?
Gareth
Thursday, November 7, 2019 1:05 PM
Next test has been to turn the Windows Defender Firewall off...
This is the error I now get....
Thursday, November 7, 2019 1:30 PM
Some more info regarding the CSV/Cluster network:
Server 1 (192.168.0.1) can ping Server 3 (192.168.0.3) - but cannot ping server 2 (TTL Expired in Transit)
Server 2 (192.168.0.2) cannot ping Server 1 or 3 (TTL Expired in Transit)
Server 3 (192.168.0.3) can ping Server 1 and Server 3 but not server 2 (TTL Expired in Transit)
Thursday, November 7, 2019 4:08 PM
I've now removed Server 2 from the cluster and it's all validating correctly. Server 1 and Server 3 can ping each other back and forth on all networks.
Weird thing with Server 2 it was only the 192.168.0.2 address that could not ping the others. The other cluster networks could ping fine. Could it be an issue with the network port on that network card? It's a 4 port 1Gbit card.
In other news - I can now live migrate from Server 1 to Server 3, but for some reason, not the other way around - but that's an issue for tomorrow.
Not sure what to do with Server 2.
Gareth
Friday, November 8, 2019 2:54 AM
Hi Gareth,
>>*Weird thing with Server 2 it was only the 192.168.0.2 address that could not ping the others. *
Did you install any third-party software on your server 2? If yes, please temperately disable third-party software to do a check.
Check if you could ping default gateway. We might need to view the Routing Tables.
Open a Command Prompt window and then enter the ROUTE PRINT command.
Best Regards,
Candy
Please remember to mark the replies as an answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]
Monday, November 11, 2019 3:29 AM
Hi ,
Just want to confirm the current situations.
Please feel free to let us know if you need further assistance.
Best Regards,
Candy
Please remember to mark the replies as an answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]