Windows 2012 + iSCSi slow IO performance

Question

_{Thursday, December 24, 2015 2:36 AM}

Hi Everyone

I have an issue and I ran out of ideas so I need some help
SQl restore of a 1 GB file takes around 30 minutes and after a while I noticed Read/Write to the storage seems to be slow
Here my environment:
Windows 2012 Standard R2 x64 (virtual server running in esx)
Storage connected with iSCSi going thru the production network (I know is not recomended :S)

Test using DiskSpd.exe -c20G -d120 -r -w40 -t8 -o32 -b8K -h -L G:\temp\sqlIOTest.dat
Read test using diskspd.exe

Write IO , AvgLat = 323.391
Read IO , AvgLat = 323.003

Versus another server (virtual server running in the same esx) in the same vlan, also same OS version, same configuration in regards of virtual hardware and Storage connected with iSCSi going thru the production network

Test using DiskSpd.exe -c20G -d120 -r -w40 -t8 -o32 -b8K -h -L G:\temp\sqlIOTest.dat
Write IO , AvgLat = 20.900
Read IO , AvgLat = 21.266

What could be causing this?
I know the iSCSi should be using an exclusive network but there is another cluster with the same conf and no issues
I already disabled TCP values (Chimeny, RSS, Netdma)
No performance issues, CPU and Memory are barely used at the time the restore is running

Any help would be highly appreciated

Best regards

Joaquin Camarero Muñoz

All replies (11)

_{Tuesday, December 29, 2015 4:28 AM ✅Answered}

Hi Everyone

Just FYI.. the culprit is the windows firewall
I already had iSCSi service allowed in the firewall but Read/Write IO was really slow, I had to completely disable the firewall and then all was fine.

Weird thing is that I have another cluster 2012 OS where the iSCSi Service rule is not even configured to be allowed and also the windows firewall enabled and it is running with no issues.

I am trying to find out what specific port/application I have to add to the windows firewall to be allowed.

Thanks, regards

Joaquin Camarero Muñoz

_{Thursday, December 24, 2015 8:19 AM}

Hi Joaquin,

Are you using the different iSCSI Virtual Disks to test? Or using a CSV?

If CSV, ensure the node is the owner node of the CSV when testing.

If not, as far as I know, the configuration on Windows Server OS may not cause the issue. And it is also difficult to reproduce the issue to analyze.

I suggest you check the configuration on esx server.

Best Regards,

Leo

Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected].

_{Thursday, December 24, 2015 11:54 AM | 2 votes}

Hi Everyone

I have an issue and I ran out of ideas so I need some help
SQl restore of a 1 GB file takes around 30 minutes and after a while I noticed Read/Write to the storage seems to be slow
Here my environment:
Windows 2012 Standard R2 x64 (virtual server running in esx)
Storage connected with iSCSi going thru the production network (I know is not recomended :S)

Test using DiskSpd.exe -c20G -d120 -r -w40 -t8 -o32 -b8K -h -L G:\temp\sqlIOTest.dat
Read test using diskspd.exe

Write IO , AvgLat = 323.391
Read IO , AvgLat = 323.003

Versus another server (virtual server running in the same esx) in the same vlan, also same OS version, same configuration in regards of virtual hardware and Storage connected with iSCSi going thru the production network

Test using DiskSpd.exe -c20G -d120 -r -w40 -t8 -o32 -b8K -h -L G:\temp\sqlIOTest.dat
Write IO , AvgLat = 20.900
Read IO , AvgLat = 21.266

What could be causing this?
I know the iSCSi should be using an exclusive network but there is another cluster with the same conf and no issues
I already disabled TCP values (Chimeny, RSS, Netdma)
No performance issues, CPU and Memory are barely used at the time the restore is running

Any help would be highly appreciated

1) You *never* configure guest iSCSI from your VMs. What you do you leave iSCSI job to *hypervisor*, make hypervisor allocate you shared LUN and for guest VM cluster you put shared VHDX on top of it. That's the only reliable way to go. Trying to troubleshoot performance hiccups from iSCSI initiator running inside guest VM and routed thru all burdens of vSwitch etc is a way to nowhere.

Deploy a Guest Cluster Using a Shared Virtual Hard Disk

https://technet.microsoft.com/en-us/library/dn265980.aspx

2) This is NOT true. iSCSI does not need separate physical network rather you use VLANs and QoS to properly chunk your physical LANs into VLANs to give iSCSI maximum bandwidth and more of physical copper/fiber pairs for redundancy. Similar discussion with a good step-by-step here:

Isolate iSCSI on VLANs

http://serverfault.com/questions/354325/isolate-iscsi-traffic-on-a-separate-vlan

3) You don't do that! That's for PHYSYCAL hardware with issues. You have VIRTUAL hardware and another set of issues (ballooning, vSwitch, VM priorities, paging, context switches etc). VMware config guide should be handy:

Configuring and troubleshooting basic software iSCSI setup (1008083)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008083

Good luck :)

Cheers,

Anton Kolomyeytsev [MVP]

StarWind Software Chief Architect

Profile:   Blog:   Twitter:   LinkedIn:  

Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

_{Thursday, December 24, 2015 12:12 PM | 1 vote}

Mostly I just want to follow this thread, but also I`d like to mention that the issue may rely on the iSCSI target settings. How do you present the iSCSI storage to server?  

_{Monday, December 28, 2015 1:46 PM}

Hello Leo

I am not using CSV, I was running the test on the iSCSi disks.
In ESX I have another cluster with 2012 that is running with no issues, a restore of the same file takes ~ 9 minutes and the cluster with issues is running in the same esx host, using the same vswitch, same data store for C/D drive.

Thanks, regards

Joaquin Camarero Muñoz

_{Monday, December 28, 2015 2:02 PM}

Hello VR38DETT

Thanks for the information provided, It gives me an idea of the path I should follow
The first article looks like is for Windows Hypervisor so I am looking for it but for ESX guests

Thanks again, regards

Joaquin Camarero Muñoz

_{Monday, December 28, 2015 2:20 PM}

Hello AnatolySV

From the iSCSi iniciator - Targets tab - in targed field I put the storage name and then quick connect.
Not sure if this was what you were asking.

All the rest of the settings are the default.
I also would like to mention that I have another cluster with the exact same settings, connected to the same storage, same disk configurations that is running with no issues

Thanks, regards

Joaquin Camarero Muñoz

_{Tuesday, December 29, 2015 6:09 PM}

Hi Everyone

Just FYI.. the culprit is the windows firewall
I already had iSCSi service allowed in the firewall but Read/Write IO was really slow, I had to completely disable the firewall and then all was fine.

Weird thing is that I have another cluster 2012 OS where the iSCSi Service rule is not even configured to be allowed and also the windows firewall enabled and it is running with no issues.

I am trying to find out what specific port/application I have to add to the windows firewall to be allowed.

Thanks, regards

Joaquin Camarero Muñoz

3260 is default iSCSI port

Cheers,

Anton Kolomyeytsev [MVP]

StarWind Software Chief Architect

Profile:   Blog:   Twitter:   LinkedIn:  

Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

_{Saturday, January 2, 2016 1:57 AM}

Hello VR38DETT

Thanks for your reply, I added that port to the windows firewall and other ports during my troubleshoot but they did not work.
I decided to turn off the firewall profiles but it did not work either, the only thing that works is to disable the firewall service itself but I don´t want that.

So looks like nothing is being blocked at the firewall level, instead looks like the filtering the firewall is performing is affecting the IO performance.

Do you have any advice on where to look at or what else to do?

Happy New Year!!

Thanks, regards

Joaquin Camarero Muñoz

_{Monday, January 4, 2016 1:05 PM}

Hello VR38DETT

Thanks for your reply, I added that port to the windows firewall and other ports during my troubleshoot but they did not work.
I decided to turn off the firewall profiles but it did not work either, the only thing that works is to disable the firewall service itself but I don´t want that.

So looks like nothing is being blocked at the firewall level, instead looks like the filtering the firewall is performing is affecting the IO performance.

Do you have any advice on where to look at or what else to do?

Happy New Year!!

Thanks, regards

Joaquin Camarero Muñoz

OK, do you have the antivirus? I saw multiple times how that thing affects iSCSI

Also, about iSCSI initiator - Quick Connect option may misconfigure the connection, that is why it is recommended to specify exact addressed while configuring connections. You may refer to this pages 15-25 of the document below as an example:

https://www.starwindsoftware.com/technical_papers/Hyperconverged-scenario-Basic-2-Node-Setup-manual.pdf 

_{Monday, January 4, 2016 1:09 PM}

Hi,

Did you disable VMQ on the NIC? May be you can try to check here:- http://www.ms4u.info/2013/03/slow-performance-when-using-nic-teaming.html

Lai (My blog:- http://www.ms4u.info)