netft.sys is the cause for the bugchk blue screen on the server Windows 2008 R2 Datacenter

Article
2013-03-08

Question

_{Friday, March 8, 2013 7:59 PM}

we have the server geting rebooted by a bugchk error for netft.sys Please let me know if we have any fix for this issue. i am not sure wht is causing the issue on the server

the server is windows 2008 R2 Datacenter and it is on the HyperV cluster

Thanks in advance

All replies (71)

_{Friday, March 8, 2013 9:28 PM ✅Answered}

I assume you are referring to a stop 0x9e for netft.sys, which is an intentional bugcheck caused by the cluster service due to a deadlock condition identified. I'd recommend reviewing the following for further tips to troubleshoot this issue:

http://blogs.technet.com/b/askcore/archive/2009/06/12/why-is-my-2008-failover-clustering-node-blue-screening-with-a-stop-0x0000009e.aspx

Visit my blog about multi-site clustering

_{Saturday, March 9, 2013 3:02 AM ✅Answered | 1 vote}

Clustering has health detection between the user mode service and the kernel mode NetFT driver. If user mode goes unresponsive, then clustering bugchecks the box in an effort to force a failover. A STOP 0x9e is expected cluster behavior. You should troubleshoot the condition as a user mode hang...

Thanks!
Elden

_{Friday, March 8, 2013 10:29 PM}

Thanks John,

thi was the same error i was refering to. I did go through the MS site and i found some Hotfix for it, Not very sure if that Hot fix is for the same issue.

http://support.microsoft.com/kb/2135160/en-us error “FIX: "0x0000009E" Stop error when you host Hyper-V virtual machines in a Windows Server 2008 R2-based failover cluster

do u have any clue on it ?

_{Friday, March 8, 2013 11:00 PM}

I think the most important line in that KB article:

Not all "0x0000009E" Stop errors are caused by this problem.

This blue screen just indicates that something caused the cluster to believe that the node was hung. If you are running Hyper-V, this might be a good place to start with this hotfix. Otherwise, you might consider opening a ticket with PSS to troubleshoot this further.

Visit my blog about multi-site clustering

_{Sunday, March 10, 2013 7:16 PM}

Techei, I am a developer on the clustering team. if the problem persists then you can share dump file with me (c:\windows\memory.dmp). I will take a look what exactly caused the bugcheck. if you do not want to share it with the world then give me your email, I'll reply with my email and you can share this dump with me. If you prefer work through customer support then they also should be able to look at the dump and tell you what netft is not happy about.

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, March 13, 2013 3:02 PM}

We're having a very similar problem.. which i'd love for someone to actually investigate the memory dump to give me a real clue as to the problem. We're in the midst of multiple cases with PSS but a solution couldn't come quick enough. How can I share this dump with you?

_{Friday, October 11, 2013 8:20 PM}

Hi Vladimir, I have the same issue with the bugcheck 0x0000009e

How can I send you the memory.dmp? Thaanks a lot

_{Saturday, October 12, 2013 3:28 AM}

If you can put it to some location I can download it from that would work.

Thanks,

Vladimir.

Regards, Vladimir Petter, Microsoft Corporation

_{Tuesday, October 15, 2013 3:55 PM}

I used my skydrive. You can dowload it from there

https://skydrive.live.com/redir?resid=95BBBD60D3F2B190!117

thanks!

_{Wednesday, October 16, 2013 5:05 AM}

look at the properties of the files c:\windows\system32\drivers\EmcpXcr.sys, EmcpBase.sys and EmcpGpx.sys to see what company these drivers came from and contact that company for support, and share with them your dump.

Cluster bugchecked the machine because cluster tried to terminate resource host monitor - the host process where the plug-ins that control applications, disks etc are running in, and this process did not go away in 20 minutes. Termination is stuck because some threads are stuck in the kernel. They all end up waiting for the thread below. That thread has been stuck for 25 minutes

THREAD fffffa800793f660 Cid 0004.0050 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
    fffffa8008467db8 SynchronizationEvent
    fffffa8008467dd0 NotificationEvent
IRP List:
    fffffa801452f4b0: (0006,0310) Flags: 00060070 Mdl: 00000000
    fffffa8013d9ba90: (0006,0310) Flags: 00060070 Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a000008500
Owning Process            fffffa80078c49e0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      262955         Ticks: 96163 (0:00:25:00.152)
Context Switch Count      10707          IdealProcessor: 12
UserTime                  00:00:00.000
KernelTime                00:00:12.105
Win32 Start Address nt!ExpWorkerThread (0xfffff800024e0150)
Stack Init fffff88002b5cdb0 Current fffff88002b5be20
Base fffff88002b5d000 Limit fffff88002b57000 Call 0
Priority 13 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`02b5be60 fffff800`024cc5f2 nt!KiSwapContext+0x7a
fffff880`02b5bfa0 fffff800`024d90ea nt!KiCommitThreadWait+0x1d2
fffff880`02b5c030 fffff880`0146ee3b nt!KeWaitForMultipleObjects+0x272
fffff880`02b5c2f0 fffff880`01603601 EmcpBase!PowerSleep+0x83
fffff880`02b5c350 fffff880`016038d9 EmcpXcr!XcryptUpdateAssocCallout+0xc45
fffff880`02b5c3b0 fffff880`014f8c10 EmcpXcr!XcryptUpdateAssocCallout+0xf1d
fffff880`02b5c3f0 fffff880`01467adf EmcpGpx!GpxDestroySplitPirp+0x78
fffff880`02b5c420 fffff880`014f8c30 EmcpBase!PowerDispatchX+0x243
fffff880`02b5c470 fffff880`01467adf EmcpGpx!GpxDestroySplitPirp+0x98
fffff880`02b5c4a0 fffff880`01468b41 EmcpBase!PowerDispatchX+0x243
fffff880`02b5c4f0 fffff880`01475681 EmcpBase!PowerSyncIoTopDispatch+0x89
fffff880`02b5c520 fffff880`01478686 EmcpBase!PowerWinIsIrpSync+0x965
fffff880`02b5c570 fffff880`0180d362 EmcpBase!PowerWinIsPseudoBusPDO+0x219a
fffff880`02b5c5d0 fffff880`0180971f disk!DiskGetPortGeometry+0x92
fffff880`02b5c630 fffff880`018015ec disk!DiskUpdateGeometry+0x4ef
fffff880`02b5c670 fffff880`0180164c disk!DiskReadDriveCapacity+0x1c
fffff880`02b5c6a0 fffff880`0159ce7d disk!DiskDeviceControl+0x2e3
fffff880`02b5c700 fffff880`012f1244 CLASSPNP!ClassDeviceControlDispatch+0x2d
fffff880`02b5c730 fffff800`027f745d partmgr!PmFilterDeviceControl+0xd4
fffff880`02b5c790 fffff800`027f72b4 nt!FstubGetDiskGeometry+0x12d
fffff880`02b5c810 fffff800`027f76e2 nt!FstubAllocateDiskInformation+0x44
fffff880`02b5c840 fffff880`012f3512 nt!IoReadPartitionTableEx+0x1a
fffff880`02b5c870 fffff880`012f1398 partmgr!PmGetDriveLayoutEx+0x5d2
fffff880`02b5c970 fffff880`0162d48f partmgr!PmFilterDeviceControl+0x228
fffff880`02b5c9d0 fffff880`0162ae3d ClusDisk!ClusDskpSendIoctl+0x8f
fffff880`02b5ca70 fffff800`02742eb0 ClusDisk!ClusDskDeviceChangeNotification+0xc9
fffff880`02b5cae0 fffff800`02741787 nt!PnpNotifyDriverCallback+0x5c
fffff880`02b5cb70 fffff800`02742ffc nt!PnpNotifyTargetDeviceChange+0x16b
fffff880`02b5cc20 fffff800`027420ca nt!PnpProcessCustomDeviceEvent+0x2c
fffff880`02b5cc50 fffff800`024e0261 nt!PnpDeviceEventWorker+0x142
fffff880`02b5ccb0 fffff800`027732ea nt!ExpWorkerThread+0x111
fffff880`02b5cd40 fffff800`024c78e6 nt!PspSystemThreadStartup+0x5a
fffff880`02b5cd80 00000000`00000000 nt!KxStartSystemThread+0x16

Storage class device fffffa800f2ef060 with extension at fffffa800f2ef1b0

Classpnp Internal Information at fffffa801371c530

Transfer Packet Engine:

Packet Status DL Irp Opcode Sector/ListId UL Irp

Pending Idle Requests: 0x0

Failed Requests:

           Srb    Scsi
    Opcode Status Status Sense Code Sector/ListId   Time Stamp

      28     04     02    02 04 03      00000001    12:28:45.684
      28     04     02    02 04 03      003fffff    12:28:45.684
      28     04     02    02 04 03      00000001    12:28:45.684
      28     04     02    02 04 03      003fffff    12:28:45.684
      28     04     02    02 04 03      00000001    12:28:45.684
      28     04     02    02 04 03      003fffff    12:28:45.684
      28     04     02    02 04 03      00000001    12:28:45.699
      28     04     02    02 04 03      003fffff    12:28:45.699
      28     04     02    02 04 03      00000001    12:28:45.699
      28     04     02    02 04 03      003fffff    12:28:45.699
      28     04     02    02 04 03      00000001    12:28:45.699
      28     04     02    02 04 03      003fffff    12:28:45.715
      28     04     02    02 04 03      00000001    12:28:45.824
      28     04     02    02 04 03      003fffff    12:28:45.824
      28     04     02    02 04 03      00000001    12:28:45.840
      28     04     02    02 04 03      003fffff    12:28:45.840

-- dt classpnp!_CLASS_PRIVATE_FDO_DATA fffffa801371c530 --

Classpnp External Information at fffffa800f2ef1b0

DGC RAID 5 0429 CKM00100500355

    Minidriver information at fffffa800f2ef670
    Attached device object at fffffa800f2ed060
    Physical device object at fffffa800f2ed060

Media Geometry:

        Bytes in a Sector = 512
        Sectors per Track = 63
        Tracks / Cylinder = 255
        Media Length      = 2147483648 bytes = ~2 GB

-- dt classpnp!_FUNCTIONAL_DEVICE_EXTENSION fffffa800f2ef1b0 --

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, October 16, 2013 4:26 PM}

Thanks a lot! I ve already talked with EMC and they recomended to us do an updated of drivers and multipath.

I have another question but using the debug tool. I download it from http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx but always it give me error from symbols

Loading Dump File [C:\Temp\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: srv*C:\websymbols*http://msdl.microsoft.com/download/symbols;srv\*
Executable search path is: srv*
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntkrnlmp.exe -
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Built by: 7601.18247.amd64fre.win7sp1_gdr.130828-1532
Machine Name:
Kernel base = 0xfffff800`02461000 PsLoadedModuleList = 0xfffff800`026a46d0
Debug session time: Fri Oct 11 16:53:45.992 2013 (UTC - 3:00)
System Uptime: 0 days 1:33:22.285
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntkrnlmp.exe -
Loading Kernel Symbols
...............................................................

Loading User Symbols

Loading unloaded module list
...........
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 9E, {fffffa8008a5eab0, 4b0, 0, 0}

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*************************************************************************
*** ***
*** ***
*** Either you specified an unqualified symbol, or your debugger ***
*** doesn't have full symbol information. Unqualified symbol ***
*** resolution is turned off by default. Please either specify a ***
*** fully qualified symbol module!symbolname, or enable resolution ***
*** of unqualified symbols by typing ".symopt- 100". Note that ***
*** enabling unqualified symbol resolution with network symbol ***
*** server shares in the symbol path may cause the debugger to ***
*** appear to hang for long periods of time when an incorrect ***
*** symbol name is typed or the network symbol server is down. ***
*** ***
*** For some commands to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: nt!_KPRCB ***
*** ***
*************************************************************************

*************************************************************************
Probably caused by : netft.sys ( netft!NetftWatchdogTimerDpc+b9 )

Followup: MachineOwner

8: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Debugging Details:

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

ADDITIONAL_DEBUG_TEXT:
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

MODULE_NAME: netft

FAULTING_MODULE: fffff80002461000 nt

DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bc48a

PROCESS_OBJECT: fffffa8008a5eab0

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0x9E

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff88000e626a5 to fffff800024d6bc0

STACK_TEXT:
fffff880`0253d518 fffff880`00e626a5 : 00000000`0000009e fffffa80`08a5eab0 00000000`000004b0 00000000`00000000 : nt!KeBugCheckEx
fffff880`0253d520 fffff800`024e185c : fffff880`0253d618 00000000`00000001 00000000`400e0088 00000000`00000001 : netft!NetftWatchdogTimerDpc+0xb9
fffff880`0253d570 fffff800`024e16f6 : fffff880`00e6f100 00000000`00057ace 00000000`00000000 00000000`00000000 : nt!KeReleaseMutant+0xb2c
fffff880`0253d5e0 fffff800`024e15de : 0000000d`0b38db3c fffff880`0253dc58 00000000`00057ace fffff880`02518f48 : nt!KeReleaseMutant+0x9c6
fffff880`0253dc30 fffff800`024e13c7 : 00000003`0ebda1c2 00000003`00057ace 00000003`0ebda10c 00000000`000000ce : nt!KeReleaseMutant+0x8ae
fffff880`0253dcd0 fffff800`024ce8ca : fffff880`02515180 fffff880`025202c0 00000000`00000000 fffff880`01a95a48 : nt!KeReleaseMutant+0x697
fffff880`0253dd80 00000000`00000000 : fffff880`0253e000 fffff880`02538000 fffff880`0253dd40 00000000`00000000 : nt!KiCpuId+0x6fa

STACK_COMMAND: kb

FOLLOWUP_IP:
netft!NetftWatchdogTimerDpc+b9
fffff880`00e626a5 cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: netft!NetftWatchdogTimerDpc+b9

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: netft.sys

BUCKET_ID: WRONG_SYMBOLS

Followup: MachineOwner

8: kd> !process fffffa8008a5eab0 3
NT symbols are incorrect, please fix symbols

_{Wednesday, October 16, 2013 5:10 PM}

Once open the dump first run ".symfix" then ".reload". After that always start from "!analyze -v". If you see
USER_MODE_HEALTH_MONITOR (9e) then it complains about a process not doing something in time (in the case above not terminating in time). The process address is in Arg1. run "!process <Arg1> 1f", and go from there.

Regards, Vladimir Petter, Microsoft Corporation

_{Tuesday, August 5, 2014 7:35 AM}

Hi all

Thanks for this information.
3. days ago we had a blusceen as well on one of our Exchange 2010 MBX server.
All drives are connected via iSCSI except C. These drives are not pysical. We are using NetApp and on top VmWare.

As I could see in the dump file while using WinDBG is:

Probably caused by : netft.sys ( netft+26a5 )What do you suggest to solve this issue?Kind regards Matthias

_{Tuesday, August 5, 2014 4:09 PM}

Hello Matthias,

If you can share the dump with me I'll be happy to take a look.

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, August 6, 2014 2:37 PM}

Hi Vladimir

Thanks for your reply. I rely appreciate this.
Here is the link to have a look into the dump: https://drive.google.com/file/d/0BxDBxOjFJ8GLRXpVZE9JN19BXzA/edit?usp=sharing.

I am waiting for your feedback.

Kind regards

Matthias

_{Wednesday, August 6, 2014 6:38 PM}

Looks like your VMWare disks are timing out IO at the moment, and also I see some netapp LUNs were failing. Please check your storage.

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 04 22 00 00 00 01f73420 16:17:20.276 Retried

28 04 22 00 00 00 018807e0 16:17:20.276 Retried

2a 04 22 00 00 00 000599c8 16:17:20.276 Retried

2a 04 22 00 00 00 01aa0bb0 17:23:49.276 Retried

2a 04 22 00 00 00 02e790f0 17:23:49.276 Retried

2a 04 22 00 00 00 013bfc50 17:23:49.276 Retried

2a 04 22 00 00 00 02692250 17:23:49.276 Retried

2a 04 22 00 00 00 03553f80 17:23:49.276 Retried

2a 04 22 00 00 00 0311d0a5 17:23:49.276 Retried

2a 04 22 00 00 00 00608b90 17:23:49.276 Retried

2a 04 22 00 00 00 02b2a210 17:23:49.276 Retried

2a 04 22 00 00 00 006327f8 17:23:49.276 Retried

2a 04 22 00 00 00 0007a050 17:23:49.276 Retried

2a 04 22 00 00 00 0333eb68 17:23:49.276 Retried

2a 04 22 00 00 00 0004e538 17:23:49.276 Retried

28 04 22 00 00 00 009fa980 17:23:49.276 Retried

VMware Virtual disk 1.0 6000c292caa09c5f634b3b75741142ea

Opcode Status Status Sense Code Sector/ListId Time Stamp

28 04 22 00 00 00 006bbac0 16:11:00.276 Retried

28 04 22 00 00 00 008e1d30 16:12:26.276 Retried

28 04 22 00 00 00 0099b8c8 16:12:26.276 Retried

28 04 22 00 00 00 0084d9c8 16:12:26.276 Retried

28 04 22 00 00 00 006cdc08 16:12:26.276 Retried

28 04 22 00 00 00 006cdb88 16:12:26.276 Retried

28 04 22 00 00 00 008e0020 16:14:50.276 Retried

28 04 22 00 00 00 0099b828 16:14:50.276 Retried

28 04 22 00 00 00 006bbbb8 16:14:50.276 Retried

28 04 22 00 00 00 0079ee58 16:14:50.276 Retried

28 04 22 00 00 00 006ed580 16:14:50.276 Retried

28 04 22 00 00 00 008fd188 16:14:50.276 Retried

28 04 22 00 00 00 00701e60 16:17:22.276 Retried

28 04 22 00 00 00 00757f68 16:17:22.276 Retried

28 04 22 00 00 00 00a87f28 16:17:22.276 Retried

28 04 22 00 00 00 00783c60 16:17:22.276 Retried

VMware Virtual disk 1.0 6000c29be6a5c7d86a9a12fabe644f5d

Opcode Status Status Sense Code Sector/ListId Time Stamp

28 04 02 06 3f 0e 0fbac380 13:33:27.572 Retried

2a 0e 00 00 00 00 137a3c00 16:09:42.041 Retried

2a 0e 00 00 00 00 137a3e00 16:09:42.041 Retried

2a 0e 02 06 29 00 137a3c00 16:09:42.119 Retried

2a 0e 00 00 00 00 00614818 17:24:09.432 Retried

NETAPP LUN 811a 7SRjT+BTl8Uf

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 00 00 00 00 0e910e60 16:09:42.729 Retried

2a 0e 02 06 29 00 006407f8 16:09:42.807 Retried

2a 0e 00 00 00 00 005fd608 16:11:00.291 Retried

2a 0e 00 00 00 00 00632330 16:11:00.291 Retried

2a 0e 00 00 00 00 0063d620 16:11:00.291 Retried

2a 0e 00 00 00 00 0bfc32e0 16:11:00.291 Retried

2a 0e 02 06 29 00 005fd608 16:11:00.322 Retried

2a 0e 00 00 00 00 0c35dfe0 16:12:29.994 Retried

2a 0e 02 06 29 00 0c35dfe0 16:12:29.994 Retried

2a 0e 00 00 00 00 0060b1b8 16:12:30.010 Retried

2a 0e 00 00 00 00 0060b1c8 16:14:50.479 Retried

2a 0e 00 00 00 00 000d0070 16:14:50.479 Retried

2a 0e 00 00 00 00 0e9e6b60 16:14:50.479 Retried

2a 0e 02 06 29 00 0060b1c8 16:14:50.510 Retried

2a 0e 00 00 00 00 0e80ece0 16:17:21.354 Retried

2a 0e 02 06 29 00 0e80ece0 16:17:21.510 Retried

NETAPP LUN 811a 7SRjT+BTl8Uh

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 00 00 00 00 000f09d8 16:12:30.088 Retried

2a 0e 00 00 00 00 002a6be8 16:12:30.088 Retried

28 0e 02 06 29 00 0e223ce8 16:12:30.088 Retried

28 0e 02 06 29 00 0e240ce8 16:13:43.276 Retried

NETAPP LUN 811a 7SRjT+BTl8Uj

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 00 00 00 00 00635770 16:11:02.369 Retried

2a 0e 02 06 29 00 005fd5f0 16:11:02.369 Retried

2a 0e 00 00 00 00 0011c320 16:13:42.979 Retried

2a 0e 02 06 29 00 0011c320 16:13:43.010 Retried

NETAPP LUN 811a 7SRjT+BTl8Ul

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 02 06 29 00 005fd608 16:13:43.260 Retried

2a 0e 02 06 29 00 00640a20 16:17:30.916 Retried

NETAPP LUN 811a 7SRjT+BTl8Un

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 00 00 00 00 0351aef8 16:11:02.463 Retried

2a 0e 00 00 00 00 0060fb78 16:11:02.463 Retried

28 0e 02 06 29 00 0b1ad938 16:11:02.463 Retried

2a 0e 00 00 00 00 06557578 16:12:30.057 Retried

28 0e 02 06 29 00 0b218f38 16:12:30.213 Retried

28 0e 02 06 29 00 0b236338 16:13:43.369 Retried

28 04 02 06 3f 0e 0cec5338 17:08:29.760 Retried

28 04 02 06 3f 0e 0d187b38 17:21:28.932 Retried

NETAPP LUN 811a 7SRjT+BTl8Up

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 00 00 00 00 1379fca0 16:09:43.197 Retried

2a 0e 02 06 29 00 002c8a60 16:09:43.229 Retried

2a 0e 02 06 29 00 005fd600 16:11:02.307 Retried

2a 0e 00 00 00 00 1379fca0 16:13:43.026 Retried

2a 0e 00 00 00 00 00628278 16:13:43.026 Retried

2a 0e 02 06 29 00 1379fca0 16:13:43.041 Retried

2a 0e 00 00 00 00 0060d340 16:17:30.713 Retried

2a 0e 00 00 00 00 002c8a60 16:17:30.713 Retried

2a 0e 02 06 29 00 0060d340 16:17:30.729 Retried

NETAPP LUN 811a 7SRjT+BTl8Ut

Opcode Status Status Sense Code Sector/ListId Time Stamp

2a 0e 00 00 00 00 02f88e50 16:17:30.572 Retried

2a 0e 00 00 00 00 012aa038 16:17:30.588 Retried

2a 0e 00 00 00 00 0063d768 16:17:30.588 Retried

2a 0e 00 00 00 00 0286d1a0 16:17:30.588 Retried

2a 0e 00 00 00 00 006199c0 16:17:30.588 Retried

2a 0e 00 00 00 00 01175778 16:17:30.588 Retried

2a 0e 00 00 00 00 0004cae0 16:17:30.588 Retried

2a 0e 02 06 29 00 0323e8c8 16:17:30.791 Retried

2a 04 02 06 3f 0e 025d736f 16:54:35.307 Retried

NETAPP LUN 811a 7SRjT+BTl8Uv

Regards, Vladimir Petter, Microsoft Corporation

_{Thursday, August 21, 2014 1:00 PM}

Hi Vladimir,

I've used WinDBG on many occassions, mainly for clr debugging. however, i'm amazed at the information you keep getting out of the dump files.

What are the most useful commands to help with this? It's a science in itself. Any guidance will be hugely appreciated

Thanks

Andreas

_{Thursday, August 21, 2014 5:06 PM}

Hi Vladamir,

Can you please help me with debugging the minidump file.
https://drive.google.com/folderview?id=0B0T9i-iE-IVhWDYwY1Z1Nm1yRFk&usp=sharing

Regards

Techwiz

_{Friday, August 22, 2014 7:42 AM}

Hello Techwiz,

Unfortunately I cannot tell much from the kernel minidump. Please change your OS settings to collect kernel full dump on bugcheck, and share that dump on the next repro.

From the minidump it looks like a thread of RHS.exe process has been stuck in the kernel for 20 minutes preventing process termination from making forward progresss. After that a watchdog has recycled machine by bugchecking it. Minidump does not contain thread's stack so I cannot tell where is it stuck.

Regards, Vladimir Petter, Microsoft Corporation

_{Monday, December 15, 2014 7:36 PM}

Vladimir,

Can you take a look at my DMP file?

https://drive.google.com/file/d/0BzIR_AB_PyatQmowSkJZX2gyQTg/view?usp=sharing

I've had two crashes in the past two weeks with Netft.sys listed as the culprit.

I would appreciate it if you noticed anything else. From a host on a Hyper-V Cluster.

Thanks,

Jon

_{Monday, December 15, 2014 8:09 PM}

Please open a case with Microsoft support and they will be able to debug the dump and identify root cause.

Also see this blog:
http://blogs.msdn.com/b/clustering/archive/2013/11/13/10467483.aspx

Thanks!
Elden

_{Tuesday, December 16, 2014 3:58 AM | 1 vote}

Hello Jon,

The problem is that you have such high DPC rate coming from network on CPU0 and CPU14 that it is stalling threads, and thread scheduling on CPU0.

On CPU 0 DPCs were running back-to-back for almost 27 seconds. CPU0 currently handling a DPC, and in the queue there are 4 to go. By the time when these 4 are processed probably new DPCs will be enqued.

CPU#0

Current DPC: NDIS!ndisInterruptDpc (Normal DPC)
Debugger Saved IRQL: 0
Cumulative DPC Time Limit: 120.000 seconds
Current Cumulative DPC Time: 26.906 seconds
Single DPC Time Limit: 20.000 seconds
Current Single DPC Time: 0.000 seconds

Pending DPCs:

CPU Type KDPC Function
0: Normal : 0xffffe00152bafb20 0xfffff800608b6ae0 vmbkmclr!InpProcessingDpcRoutine
0: Normal : 0xffffe00149e628e8 0xfffff80060014c60 NDIS!ndisInterruptDpc
0: Normal : 0xffffe00149d1f8e8 0xfffff80060014c60 NDIS!ndisInterruptDpc
0: Normal : 0xffffe00149ddf8e8 0xfffff80060014c60 NDIS!ndisInterruptDpc

On CPU14 they were running for almost 1.8 seconds

CPU#14

Current DPC: netft!NetftWatchdogTimerDpc (Normal DPC)

Debugger Saved IRQL: 2

Cumulative DPC Time Limit: 120.000 seconds

Current Cumulative DPC Time: 1.828 seconds

Single DPC Time Limit: 20.000 seconds

Current Single DPC Time: 0.000 seconds

The thread that is currently (RUNING) not making forward progress because CPU0 is busy processing DPCs from the network

THREAD ffffe00147e1f040 Cid 0004.01a4 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0

Not impersonating

DeviceMap ffffc000c360dca0

Owning Process ffffe001463c3900 Image: System

Attached Process N/A Image: N/A

Wait Start TickCount 52759634 Ticks: 52 (0:00:00:00.812)

Context Switch Count 11411046 IdealProcessor: 0

UserTime 00:00:00.000

KernelTime 00:09:27.515

Win32 Start Address vmbusr!AwWorkerThread (0xfffff80060a1d9f0)

Stack Init ffffd001e478add0 Current ffffd001e478a7b0

Base ffffd001e478b000 Limit ffffd001e4785000 Call 0

Priority 15 BasePriority 8 UnusualBoost 7 ForegroundBoost 0 IoPriority 2 PagePriority 5

# Child-SP RetAddr Call Site

00 fffff803`7df681b0 fffff800`608b9007 vmbusr!BusChSendInterrupt+0x7

01 (Inline Function) ` vmbkmclr!KmclSendSignal+0xf

02 (Inline Function) ` vmbkmclr!OutpProcessRingResult+0x29

03 (Inline Function) ` vmbkmclr!OutpTrySendControlPacket+0x227

04 (Inline Function) ` vmbkmclr!OutSendPacket+0x2ac

05 fffff803`7df681e0 fffff800`6091f89a vmbkmclr!VmbPacketSendWithTransferPageRanges+0x307

06 fffff803`7df68290 fffff800`6091f2ee vmswitch!VmsVmNicPvtRndisHostMessageSend+0xfa

07 fffff803`7df68350 fffff800`6091ec7f vmswitch!RndisDevHostDeviceIndicatePackets+0x61e

08 (Inline Function) ` vmswitch!RndisDevDeviceIndicatePackets+0x25

09 fffff803`7df68540 fffff800`6092ed39 vmswitch!VmsVmNicPvtPacketForward+0x17f

0a fffff803`7df68750 fffff800`6092bb0a vmswitch!VmsRouterDeliverNetBufferLists+0x5c9

0b fffff803`7df68830 fffff800`60012a53 vmswitch!VmsExtPtReceiveNetBufferLists+0x13a

0c fffff803`7df68890 fffff800`60012f19 NDIS!ndisMIndicateNetBufferListsToOpen+0x123

0d (Inline Function) ` NDIS!ndisMDispatchReceiveNetBufferListsInternal+0x27e

0e fffff803`7df68950 fffff800`600136b2 NDIS!ndisMTopReceiveNetBufferLists+0x2c9

0f (Inline Function) ` NDIS!ndisIterativeDPInvokeHandlerOnTracker+0x2d3

10 (Inline Function) ` NDIS!ndisInvokeNextReceiveHandler+0x64d

11 (Inline Function) ` NDIS!ndisMIndicateReceiveNetBufferListsInternal+0x6a2

12 fffff803`7df689e0 fffff800`6091963e NDIS!NdisMIndicateReceiveNetBufferLists+0x732

13 fffff803`7df68bd0 fffff800`60918aea vmswitch!VmsExtMpIndicatePackets+0x96

14 fffff803`7df68c10 fffff800`60017f81 vmswitch!VmsExtMpSendNetBufferLists+0x47a

15 (Inline Function) ` NDIS!ndisMSendNBLToMiniportInternal+0xca

16 (Inline Function) ` NDIS!ndisMSendNBLToMiniport+0xca

17 (Inline Function) ` NDIS!ndisCallSendHandler+0x24d

18 (Inline Function) ` NDIS!ndisIterativeDPInvokeHandlerOnTracker+0x27c

19 (Inline Function) ` NDIS!ndisInvokeNextSendHandler+0x417

1a (Inline Function) ` NDIS!ndisSendNBLToFilter+0x497

1b (Inline Function) ` NDIS!ndisMTopSendNetBufferLists+0x4a5

1c fffff803`7df68da0 fffff800`6091e347 NDIS!NdisSendNetBufferLists+0x551

1d fffff803`7df68f90 fffff800`6091de14 vmswitch!VmsExtPtRouteNetBufferLists+0x377

1e fffff803`7df69060 fffff800`60012a53 vmswitch!VmsPtNicReceiveNetBufferLists+0x3c4

1f fffff803`7df691c0 fffff800`60012f19 NDIS!ndisMIndicateNetBufferListsToOpen+0x123

20 (Inline Function) ` NDIS!ndisMDispatchReceiveNetBufferListsInternal+0x27e

21 fffff803`7df69280 fffff800`600136b2 NDIS!ndisMTopReceiveNetBufferLists+0x2c9

22 (Inline Function) ` NDIS!ndisIterativeDPInvokeHandlerOnTracker+0x2d3

23 (Inline Function) ` NDIS!ndisInvokeNextReceiveHandler+0x64d

24 (Inline Function) ` NDIS!ndisMIndicateReceiveNetBufferListsInternal+0x6a2

25 fffff803`7df69310 fffff800`61a0f814 NDIS!NdisMIndicateReceiveNetBufferLists+0x732

26 fffff803`7df69500 fffff800`61a0f23e NdisImPlatform!implatTryToIndicateReceiveNBLs+0x1e8

27 fffff803`7df69570 fffff800`60012a53 NdisImPlatform!implatReceiveNetBufferLists+0x1a2

28 fffff803`7df695f0 fffff800`60012f19 NDIS!ndisMIndicateNetBufferListsToOpen+0x123

29 (Inline Function) ` NDIS!ndisMDispatchReceiveNetBufferListsInternal+0x27e

2a fffff803`7df696b0 fffff800`60013094 NDIS!ndisMTopReceiveNetBufferLists+0x2c9

2b (Inline Function) ` NDIS!ndisInvokeNextReceiveHandler+0x2f

2c (Inline Function) ` NDIS!ndisMIndicateReceiveNetBufferListsInternal+0x84

2d fffff803`7df69740 fffff800`606251c4 NDIS!NdisMIndicateReceiveNetBufferLists+0x114

2e fffff803`7df69930 fffff800`60625a9d e1i63x64!RECEIVE::RxIndicateNBLs+0xd4

2f fffff803`7df69970 fffff800`60618150 e1i63x64!RECEIVE::RxProcessInterrupts+0x19d

30 fffff803`7df699f0 fffff800`6061857e e1i63x64!INTERRUPT::MsgIntDpcTxRxProcessing+0x1c0

31 fffff803`7df69a60 fffff800`60617b78 e1i63x64!INTERRUPT::MsgIntMessageInterruptDPC+0x13e

32 fffff803`7df69ac0 fffff800`60014e02 e1i63x64!INTERRUPT::MiniportMessageInterruptDPC+0x28

33 (Inline Function) ` NDIS!ndisMiniportDpc+0x110

34 fffff803`7df69b00 fffff803`7c342cd0 NDIS!ndisInterruptDpc+0x1a3

35 fffff803`7df69be0 fffff803`7c341f87 nt!KiExecuteAllDpcs+0x1b0

36 fffff803`7df69d30 fffff803`7c3cbad5 nt!KiRetireDpcList+0xd7

37 fffff803`7df69fb0 fffff803`7c3cb8d9 nt!KyRetireDpcList+0x5

38 ffffd001`e478aa10 fffff803`7c3cd9fa nt!KiDispatchInterruptContinue

39 ffffd001`e478aa40 fffff803`7c343cd3 nt!KiDpcInterrupt+0xca

3a (Inline Function) ` nt!KzLowerIrql+0x9

3b ffffd001`e478abd0 fffff800`608b7d4b nt!KeInsertQueueDpc+0x1e3

3c (Inline Function) ` vmbkmclr!InpReleaseLockAndPerformWork+0xe8

3d ffffd001`e478ac50 fffff800`608b6a90 vmbkmclr!InpTransitionRunningQueue+0x17b

3e ffffd001`e478ac90 fffff800`60a1daaf vmbkmclr!InpProcessingWorkerRoutine+0xf0

3f (Inline Function) ` vmbusr!AwRunWorkItem+0x29

40 ffffd001`e478ace0 fffff803`7c379c70 vmbusr!AwWorkerThread+0xbf

41 ffffd001`e478ad40 fffff803`7c3cefc6 nt!PspSystemThreadStartup+0x58

42 ffffd001`e478ada0 00000000`00000000 nt!KxStartSystemThread+0x16

There are several threads on the CPU0 ready to run

Processor 0: Ready Threads at priority 15

THREAD ffffe8006e2d2080 Cid 0c08.2894 Teb: 00007ff7ac976000 Win32Thread: 0000000000000000 READY on processor 0

THREAD ffffe0014e53f880 Cid 0c08.0d58 Teb: 00007ff7ac978000 Win32Thread: 0000000000000000 READY on processor 0

THREAD ffffe8006df75880 Cid 0c08.247c Teb: 00007ff7ac972000 Win32Thread: fffff901407ac010 READY on processor 0

THREAD ffffe8006bd24880 Cid 0c08.177c Teb: 00007ff7ac96a000 Win32Thread: 0000000000000000 READY on processor 0

THREAD ffffe00148889880 Cid 0004.0dc8 Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

THREAD ffffe8006a321880 Cid 0820.1e64 Teb: 00007ff671f88000 Win32Thread: fffff901406c4b60 READY on processor 0

THREAD ffffe00150278040 Cid 0004.045c Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

THREAD ffffe0014d98e300 Cid 0004.2160 Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

THREAD ffffe8006e65b880 Cid 0004.1b00 Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

I see a clussvc thread has been sitting in the READY state on CPU0 for almost 58 seconds. It is possible that this thread eventually was supposed to send the heartbeat.

THREAD ffffe0014d98e300 Cid 0004.2160 Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0

Not impersonating

DeviceMap ffffc000c360dca0

Owning Process ffffe001463c3900 Image: System

Attached Process N/A Image: N/A

Wait Start TickCount 52755922 Ticks: 3764 (0:00:00:58.812)

Context Switch Count 56741 IdealProcessor: 0

UserTime 00:00:00.000

KernelTime 00:00:01.109

Win32 Start Address nt!ExpWorkerThread (0xfffff8037c314100)

Stack Init ffffd001eb77efd0 Current ffffd001eb77ebc0

Base ffffd001eb77f000 Limit ffffd001eb779000 Call 0

Priority 15 BasePriority 7 UnusualBoost 8 ForegroundBoost 0 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

ffffd001`eb77ec00 fffff803`7c29fe9b nt!KiSwapContext+0x76

(Inline Function) ` nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff803`7c29fe9b)

ffffd001`eb77ed40 fffff803`7c29fb7b nt!KiCheckForThreadDispatch+0x153

ffffd001`eb77ed80 fffff803`7c29f98d nt!KeSetSystemGroupAffinityThread+0xfb

ffffd001`eb77edd0 fffff803`7c2f5a4b nt!KeGenericProcessorCallback+0xdd

ffffd001`eb77ef40 fffff803`7c3cc2f7 nt!KeGenericCallDpc+0x27

ffffd001`eb77ef80 fffff803`7c3cc2bd nt!KySwitchKernelStackCallout+0x27 (TrapFrame @ ffffd001`eb77ee40)

ffffd001`eb8f97f0 fffff803`7c2b6a7d nt!KiSwitchKernelStackContinue

ffffd001`eb8f9810 fffff803`7c3024ab nt!KeExpandKernelStackAndCalloutInternal+0x2fd

ffffd001`eb8f9900 fffff803`7c32634b nt!MiSwapStackPage+0x2d7

ffffd001`eb8f99d0 fffff803`7c304cee nt!MiClaimPhysicalRun+0x44f

ffffd001`eb8f9a50 fffff803`7c3043cd nt!MiFindContiguousPages+0x282

ffffd001`eb8f9bb0 fffff803`7c723a8c nt!MiRebuildLargePage+0x99

ffffd001`eb8f9c40 fffff803`7c31438c nt!MiRebuildLargePages+0x88

ffffd001`eb8f9c90 fffff803`7c379c70 nt!ExpWorkerThread+0x28c

ffffd001`eb8f9d40 fffff803`7c3cefc6 nt!PspSystemThreadStartup+0x58

ffffd001`eb8f9da0 00000000`00000000 nt!KxStartSystemThread+0x16

Because of DPC storm on CPU0 that clussvc did not send to netft heartbeat for 1 minute and netft finally bugchecked the machine.

I would be curios to learn what scenario are you running that leads to that.

As a remediation I see several options

Increase ClussvcHangTimeout (cluster public property) to a value above “Cumulative DPC Time Limit”. You can set it to 135 (2 minutes 15 seconds). In that case either machine survives or it will be bugchecked by DPC watchdog, in any case it might be better than 9e from netft,sys.
Look at you NIC settings. Looks like you are using 4 “Intel(R) Gigabit ET Quad Port Server Adapter” hooked to VMswitch and 3 “Broadcom BCM5709C NetXtreme II GigE” available for the host. Perhaps you can tune up how traffic from these NICs is load balanced across CPUs using RSS (Receive Side Scaling) and VMQ. Please also check that you have latest and greatest drivers for these NICs.
I see you are using Windows 2012 R2. Please make sure you have latest fixes: https://support.microsoft.com/kb/2920151?wa=wsignin1.0.

I would second Elden’s advise to contact Microsoft support for this case. You should share with them your dump, and you are welcome to share analysis above.

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, August 26, 2015 8:25 AM}

Hello Vladimir,

it seems that we are having a very similar problem with a new installation at customer site. Maybe you can help us locating what the Problem is?

You can find the latest dump file here: https://www.dropbox.com/s/i6m0ic49epbxk6g/MEMORY.zip?dl=0

There is a 2012 R2 Cluster running on a DataCore SANsymphonyV virtual SAN. Once we start to initialize new virtual disks the cluster crashes down.

Would be great if you could help us, because the System will not go productive until the problem is fixed...

Thanks a lot,

Martin

_{Wednesday, August 26, 2015 7:20 PM}

Hello Martin,

Netft bugchecked machine because RHS.exe was not able to complete termination in 20 minutes. One of the threads of that process is stuck in kernel waiting for an IO to complete by the storage.

I would suggest to share that dump with support of folks who implemented DcsPoll.sys. Or see if they have any updates.

    Loaded symbol image file: DcsPoll.sys
    Image path: \SystemRoot\System32\drivers\DcsPoll.sys
    Image name: DcsPoll.sys
    Timestamp:        Mon Aug 10 11:26:39 2015 (55C8ECDF)
    CheckSum:         00013D53
    ImageSize:        00016000
    File version:     15.0.300.5312
    Product version: 15.0.300.5312
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      DataCore Software Corporation
    ProductName:      DcsPoll.sys
    InternalName:     DcsPoll.sys
    OriginalFilename: DcsPoll.sys
    ProductVersion:   15.0.300.5312
    FileVersion:      15.0.300.5312
    FileDescription: DcsPoll.sys
    LegalCopyright:   Copyright © 1998-2015 DataCore Software Corporation. All Rights Reserved.
    Comments:         All Rights Reserved.

System Uptime: 4 days 2:58:57.605, and this thread is doing some busy loop.

THREAD ffffe000e1174880 Cid 0004.0248 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2e
Not impersonating
DeviceMap                 ffffc0015500c0b0
Owning Process            ffffe000e10b05c0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22805606       Ticks: 0
Context Switch Count      1196804        IdealProcessor: 44
UserTime                  00:00:00.000
KernelTime                4 Days 02:06:17.359
*** ERROR: Symbol file could not be found. Defaulted to export symbols for DcsPoll.sys -
Win32 Start Address DcsPoll (0xfffff8013fa85504)
Stack Init ffffd00023569c90 Current ffffd00023569810
Base ffffd0002356a000 Limit ffffd00023564000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
ffffd000`23569b20 fffff801`3fa856a0 DcsPoll+0x5b2d
ffffd000`23569bc0 fffff803`c931536c DcsPoll+0x66a0
ffffd000`23569c00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58@ 5906]
ffffd000`23569c60 00000000`00000000 nt!KxStartSystemThread+0x16

I see they are also busy spinning on the 3 other CPUs

THREAD ffffe000e3dd1380 Cid 0004.0644 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 4
Not impersonating
DeviceMap                 ffffc0015500c0b0
Owning Process            ffffe000e10b05c0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22805606       Ticks: 0
Context Switch Count      37419169       IdealProcessor: 4
UserTime                  00:00:00.000
KernelTime                00:23:29.843
Win32 Start Address DcsPool (0xfffff80140a71d44)
Stack Init ffffd0002118fc90 Current ffffd0002118f890
Base ffffd00021190000 Limit ffffd0002118a000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
ffffd000`2118f920 fffff801`40a90462 DcsPool+0xe40a
ffffd000`2118fb20 fffff801`40a903ed DcsPool+0x23462
ffffd000`2118fb60 fffff801`40a9cd9e DcsPool+0x233ed
ffffd000`2118fb90 fffff801`40a71dde DcsPool+0x2fd9e
ffffd000`2118fbc0 fffff803`c931536c DcsPool+0x4dde
ffffd000`2118fc00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58
ffffd000`2118fc60 00000000`00000000 nt!KxStartSystemThread+0x16

THREAD ffffe8008628a040 Cid 0004.26f0 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 10
Not impersonating
DeviceMap                 ffffc0015500c0b0
Owning Process            ffffe000e10b05c0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22805606       Ticks: 0
Context Switch Count      2805765        IdealProcessor: 16
UserTime                  00:00:00.000
KernelTime                00:19:47.625
Win32 Start Address DcsFcEng (0xfffff8013f609cf4)
Stack Init ffffd0002b121c90 Current ffffd0002b121810
Base ffffd0002b122000 Limit ffffd0002b11c000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
ffffd000`2b121968 fffff801`3fa6f6a4 DcsSup+0x13eb
ffffd000`2b121970 fffff801`3f60a3ce DcsSup!DcsSup::memset+0x48
ffffd000`2b1219a0 fffff801`3f60a208 DcsFcEng+0xa3ce
ffffd000`2b121a40 fffff801`3f607a6c DcsFcEng+0xa208
ffffd000`2b121ad0 fffff801`3f607ead DcsFcEng+0x7a6c
ffffd000`2b121b00 fffff801`3f609c11 DcsFcEng+0x7ead
ffffd000`2b121b40 fffff801`3f609cfd DcsFcEng+0x9c11
ffffd000`2b121bd0 fffff803`c931536c DcsFcEng+0x9cfd
ffffd000`2b121c00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58
ffffd000`2b121c60 00000000`00000000 nt!KxStartSystemThread+0x16

THREAD ffffe000e1174040 Cid 0004.0244 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2c
Not impersonating
DeviceMap                 ffffc0015500c0b0
Owning Process            ffffe000e10b05c0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      22805606       Ticks: 0
Context Switch Count      767536         IdealProcessor: 16
UserTime                  00:00:00.000
KernelTime                4 Days 02:41:57.718
Win32 Start Address DcsPoll (0xfffff8013fa85504)
Stack Init ffffd000269e2c90 Current ffffd000269e26f0
Base ffffd000269e3000 Limit ffffd000269dd000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
ffffd000`269e2970 fffff801`40db3264 DcsIs+0x1d4ba
ffffd000`269e29a0 fffff801`414d4c33 DcsIs+0x1d264
ffffd000`269e29d0 fffff801`3fb815aa DcsiMgr+0x20c33
ffffd000`269e2a10 fffff801`3f621150 DcsShim+0xa5aa
ffffd000`269e2a80 fffff801`3f61d82a DcsFcEng+0x21150
ffffd000`269e2ac0 fffff801`3f60ed11 DcsFcEng+0x1d82a
ffffd000`269e2af0 fffff801`3fa84b71 DcsFcEng+0xed11
ffffd000`269e2b20 fffff801`3fa856a0 DcsPoll+0x5b71
ffffd000`269e2bc0 fffff803`c931536c DcsPoll+0x66a0
ffffd000`269e2c00 fffff803`c936c2c6 nt!PspSystemThreadStartup+0x58
ffffd000`269e2c60 00000000`00000000 nt!KxStartSystemThread+0x16

In this dump I see 3 IOs outstanding to disks

DO ffffe80085c2a4a0   Ext ffffe80085c2a5f0   Adapter ffffe000e48b31a0   Working
   Vendor: DataCore   Product: Mirror Disk        SCSI ID: (0, 3, 1)
   Claimed Enumerated
   SlowLock Free   RemLock 2   PageCount 0
   QueueTagList: ffffe80085c2a6b0      Outstanding: Head ffffe000e4ff0050 Tail ffffe000e4ff0050 Timeout 50
   DeviceQueue ffffe80085c2a6e0   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0
   IO Gateway: Busy Count 0   Pause Count 0
   Requests: Outstanding 1   Device 0   ByPass 0

[Device-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Bypass-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Outstanding Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ffffe80084c69350 [SCSI] ffffe800850bd220 ffffe000e4ff0020 SCSI/UNMAP ffffe80084222d00 0000000000000000 50

[Completed Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

   DO ffffe800851d1480   Ext ffffe800851d15d0   Adapter ffffe000e48b31a0   Working
   Vendor: DataCore   Product: Virtual Disk       SCSI ID: (0, 0, 1)
   Claimed Enumerated
   SlowLock Free   RemLock 2   PageCount 0
   QueueTagList: ffffe800851d1690      Outstanding: Head ffffe000e53f1c10 Tail ffffe000e53f1c10 Timeout 50
   DeviceQueue ffffe800851d16c0   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0
   IO Gateway: Busy Count 0   Pause Count 0
   Requests: Outstanding 1   Device 0   ByPass 0

[Device-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Bypass-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Outstanding Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ffffe80085c142f0 [SCSI] ffffe80083f7dec0 ffffe000e53f1be0 **SCSI/UNMAP ** ffffe80085d7b750 0000000000000000 50

[Completed Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

   DO ffffe80086beb060   Ext ffffe80086beb1b0   Adapter ffffe000e48b31a0   Working
   Vendor: DataCore   Product: Mirror Disk        SCSI ID: (0, 3, 2)
   Claimed Enumerated
   SlowLock Free   RemLock 2   PageCount 0
   QueueTagList: ffffe80086beb270      Outstanding: Head ffffe000e4ff7050 Tail ffffe000e4ff7050 Timeout 50
   DeviceQueue ffffe80086beb2a0   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0
   IO Gateway: Busy Count 0   Pause Count 0
   Requests: Outstanding 1   Device 0   ByPass 0

[Device-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Bypass-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Outstanding Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ffffe000e1392ee0 [SCSI] ffffe80084579890 ffffe000e4ff7020 **SCSI/UNMAP ** ffffe000edf20360 0000000000000000 50

[Completed Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

   DO ffffe80089d6c7f0   Ext ffffe80089d6c940   Adapter ffffe000e48b31a0   Working
   Vendor: DataCore   Product: Virtual Disk       SCSI ID: (0, 1, 0)
   Claimed Enumerated
   SlowLock Free   RemLock 2   PageCount 0
   QueueTagList: ffffe80089d6ca00      Outstanding: Head ffffe000e53f8c10 Tail ffffe000e53f8c10 Timeout 50
   DeviceQueue ffffe80089d6ca30   Depth: 250   Status: Not Frozen   PauseCount: 0   BusyCount: 0
   IO Gateway: Busy Count 0   Pause Count 0
   Requests: Outstanding 1   Device 0   ByPass 0

[Device-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Bypass-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Outstanding Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ffffe800864be010 [SCSI] ffffe800856d2df0 ffffe000e53f8be0 **SCSI/UNMAP ** ffffe80084b7fa10 0000000000000000 50

[Completed Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

Since this is scsi/unmap my guess this is some solution that provides SSD/NVME based cache.

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, September 9, 2015 3:41 PM}

Hello Vladimir,

Could you please help in reviewing memory dump uploaded at below link?

https://drive.google.com/file/d/0B1Z6Q5Mfd7nid2lDWGJKLVZDWWs/view?usp=sharing

I have 2 node windows 2012 R2 active/passive cluster setup on Hyper-V. cluster nodes generate crash dump with bug check 09e very frequently.

Thanks & Regards,

Suchit Patil

_{Wednesday, September 9, 2015 6:56 PM}

Hello Suchit,

Cluster bugchecked machine because Resource Host Monitor has not completed termination in 20 minutes. One of the RHS threads is stuck in the kernel for about 20 minutes. Looks like things are getting stuck in TmXPFlt.sys.

As a remediation you might want to uninstall this product until issue is resolved. I would also suggest to talk to support of the company that provided that solution to see if they have a fix and to make sure they are aware of that issue.

I see lots of threads in the system are stuck with a similar call stack.

Loaded symbol image file: TmXPFlt.sys

Image path: \?\C:\Program Files (x86)\Trend Micro\OfficeScan Client\TmXPFlt.sys

Image name: TmXPFlt.sys

Timestamp: Sat Aug 30 06:11:38 2014 (5401CD8A)

CheckSum: 0005DDB6

ImageSize: 0006C000

Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

THREAD fffffa806e2d6080 Cid 0d94.0f6c Teb: 000007f7a0c6e000 Win32Thread: fffff90102e3ab80 WAIT: (Executive) KernelMode Non-Alertable

fffff88007394440 SynchronizationEvent

IRP List:

fffffa8033b24010: (0006,03e8) Flags: 00000884 Mdl: 00000000

Not impersonating

DeviceMap fffff8a00000c310

Owning Process fffffa8032dd9980 Image: rhs.exe

Attached Process N/A Image: N/A

Wait Start TickCount 750939 Ticks: 76757 (0:00:19:59.328)

Context Switch Count 378 IdealProcessor: 5

UserTime 00:00:00.015

KernelTime 00:00:00.015

Win32 Start Address 0x000007f7a13cbc24

Stack Init fffff88007395c90 Current fffff88007394190

Base fffff88007396000 Limit fffff88007390000 Call 0

Priority 14 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

fffff880`073941d0 fffff800`342aff79 nt!KiSwapContext+0x76

(Inline Function) ` nt!KiSwapThread+0xfa (Inline Function @ fffff800`342aff79)

fffff880`07394310 fffff800`342ac21f nt!KiCommitThreadWait+0x229

fffff880`07394380 fffff880`05050457 nt!KeWaitForSingleObject+0x1cf

fffff880`07394410 fffff880`050460df TmXPFlt+0xe457

fffff880`07394470 fffff880`04384df5 TmXPFlt+0x40df

fffff880`07394590 fffff880`016ae844 TmPreFlt!TmpQueryFullName+0xd61

fffff880`07394660 fffff880`016afa6c fltmgr!FltpPerformPreCallbacks+0x324

fffff880`07394770 fffff880`016da349 fltmgr!FltpPassThroughInternal+0x8c

fffff880`073947a0 fffff800`34655228 fltmgr!FltpCreate+0x339

(Inline Function) ` nt!IoCallDriverWithTracing+0xc3 (Inline Function @ fffff800`34655228)

fffff880`07394850 fffff800`34668470 nt!IopParseDevice+0x173c

fffff880`07394a30 fffff800`34656978 nt!ObpLookupObjectName+0x644

fffff880`07394b40 fffff800`3466930e nt!ObOpenObjectByName+0x258

fffff880`07394c10 fffff800`3463f96c nt!IopCreateFile+0x37c

fffff880`07394cb0 fffff800`34284d53 nt!NtOpenFile+0x58

fffff880`07394d40 fffff800`34289f30 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07394db0)

fffff880`07394f48 fffff800`34626a68 nt!KiServiceLinkage

fffff880`07394f50 fffff800`34284d53 nt!NtCreateUserProcess+0x400

fffff880`07395a90 000007fb`572a371b nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07395b00)

000000a4`76ced028 00000000`00000000 0x000007fb`572a371b

I’ve also noticed several threads where TmXPFlt is trying to open a file over SMB. Perhaps all other activity is stuck behind these activities, but it is hard to tell without symbols.

THREAD fffffa806e078080 Cid 0004.0c9c Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable

fffffa803444e190 SynchronizationEvent

IRP List:

fffffa806f440010: (0006,01f0) Flags: 00000884 Mdl: 00000000

Impersonation token: fffff8a00dc72270 (Level Impersonation)

DeviceMap fffff8a00e5514b0

Owning Process fffffa8030bc9980 Image: System

Attached Process N/A Image: N/A

Wait Start TickCount 728673 Ticks: 99023 (0:00:25:47.234)

Context Switch Count 42000 IdealProcessor: 7

UserTime 00:00:00.000

KernelTime 00:00:40.156

Win32 Start Address TmXPFlt (0xfffff8800504dddc)

Stack Init fffff88009395fd0 Current fffff88009395b80

Base fffff88009396000 Limit fffff88009390000 Call 0

Priority 12 BasePriority 8 UnusualBoost 3 ForegroundBoost 0 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

fffff880`09395bc0 fffff800`342aff79 nt!KiSwapContext+0x76

(Inline Function) ` nt!KiSwapThread+0xfa (Inline Function @ fffff800`342aff79)

fffff880`09395d00 fffff800`342ac21f nt!KiCommitThreadWait+0x229

fffff880`09395d70 fffff880`056483bb nt!KeWaitForSingleObject+0x1cf

fffff880`09395e00 fffff880`0563ffde mrxsmb10!SmbCeInitiateExchange+0x30f

fffff880`09395e70 fffff880`043a40db mrxsmb10!MRxSmbCreate+0x8d6

fffff880`09395f50 fffff800`342804a7 mrxsmb!SmbpShellCreateWithNewStack+0x1b

fffff880`09395f80 fffff800`3428046d nt!KySwitchKernelStackCallout+0x27 (TrapFrame @ fffff880`09395e40)

fffff880`0664b880 fffff800`342c786e nt!KiSwitchKernelStackContinue

fffff880`0664b8a0 fffff800`34243fc5 nt!KeExpandKernelStackAndCalloutInternal+0x20e

fffff880`0664b9a0 fffff880`043a40aa nt!KeExpandKernelStackAndCallout+0x15

fffff880`0664b9e0 fffff880`01ba8620 mrxsmb!SmbShellCreate+0x4a

fffff880`0664ba10 fffff880`01ba547d rdbss!RxCollapseOrCreateSrvOpen+0x210

fffff880`0664baa0 fffff880`01ba69ab rdbss!RxCreateFromNetRoot+0x63d

fffff880`0664bbd0 fffff880`01b6e652 rdbss!RxCommonCreate+0x15b

fffff880`0664bc70 fffff880`01ba059b rdbss!RxFsdCommonDispatch+0x522

fffff880`0664bdd0 fffff880`043d209c rdbss!RxFsdDispatch+0xcb

fffff880`0664be30 fffff880`01f37161 mrxsmb!MRxSmbFsdDispatch+0x8c

fffff880`0664be70 fffff880`01f34215 mup!MupiCallUncProvider+0x1b1

fffff880`0664bee0 fffff880`01f32475 mup!MupStateMachine+0xb6

fffff880`0664bf10 fffff880`016b04ee mup!MupCreate+0x165

fffff880`0664bf80 fffff880`016da35d fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x25e

fffff880`0664c020 fffff800`34655228 fltmgr!FltpCreate+0x34d

(Inline Function) ` nt!IoCallDriverWithTracing+0xc3 (Inline Function @ fffff800`34655228)

fffff880`0664c0d0 fffff800`34668470 nt!IopParseDevice+0x173c

fffff880`0664c2b0 fffff800`34656978 nt!ObpLookupObjectName+0x644

fffff880`0664c3c0 fffff800`3466930e nt!ObOpenObjectByName+0x258

fffff880`0664c490 fffff800`34669a59 nt!IopCreateFile+0x37c

fffff880`0664c530 fffff800`34284d53 nt!NtCreateFile+0x79 fffff880`0664c5c0 fffff800`34289f30 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0664c630) fffff880`0664c7c8 fffff880`04fb1651 nt!KiServiceLinkage

fffff880`0664c7d0 fffff880`04fb243a VSApiNt!VSSwapShortTable+0x721

fffff880`0664c840 fffff880`05049d16 VSApiNt!VSKDZwCreateFile+0x5a

fffff880`0664c8b0 fffff880`0504c1b9 TmXPFlt+0x7d16

fffff880`0664c980 fffff880`0504ce6e TmXPFlt+0xa1b9

fffff880`0664c9e0 fffff880`0504da7f TmXPFlt+0xae6e

fffff880`0664cb20 fffff880`0504def1 TmXPFlt+0xba7f

fffff880`0664cbe0 fffff800`3422f2c5 TmXPFlt+0xbef1

fffff880`0664cc10 fffff800`3426c656 nt!PspSystemThreadStartup+0x59 [d:\win8_ldr\minkernel\ntos\ps\psexec.c @ 5691]

fffff880`0664cc60 00000000`00000000 nt!KxStartSystemThread+0x16 [d:\win8_ldr\minkernel\ntos\ke\amd64\threadbg.asm @ 75]

Loaded symbol image file: VSApiNt.sys

Image path: \?\C:\Program Files (x86)\Trend Micro\OfficeScan Client\VSApiNt.sys

Image name: VSApiNt.sys

Timestamp: Sat Aug 30 06:03:46 2014 (5401CBB2)

CheckSum: 0024476C

ImageSize: 00238000

Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Here is a list of all opens that are stack over SMB.

RxContext RDR [ Maj, Min] Irp Thread FCB

fffffa8032dd4bb0 0 [ 0x 0, 0x 0] fffffa806e7df010 fffffa806eea9440 0000000000000000

16:08.471 CREATE '\HMEL-BTH-DC03.hmel.int\IPC$'

fffffa8034749950 0 [ 0x 0, 0x 0] fffffa806f624d90 fffffa80342c4b00 fffff88001b998c0

13:29.967 CREATE '<<empty>>'

fffffa8033ce44b0 0 [ 0x 0, 0x 0] fffffa806f130d10 fffffa80338bbb00 fffff8a00e5bf010

25:47.248 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E4\F0E43290E239950FABB7730FEA0B4421.DVS'

fffffa8033ee15e0 0 [ 0x 0, 0x 0] fffffa8033f27400 fffffa806f4a0900 fffff8a00ee432a0

25:47.248 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-07\6\125\6125F1071BA45DE8BA67A9D1E7004ED1~90~9F3EAD6D~00~1.DVSSP'

fffffa8034459200 0 [ 0x 0, 0x 0] fffffa806ef81be0 fffffa803413cb00 fffff8a00d5d4670

25:47.247 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E4\F0E4B8A43ECF2FB81443FFE354A7A931.DVS'

fffffa8033ac7950 2 [ 0x e, 0x 0] fffffa8032de64f0 fffffa8033896080 fffff8a00f667610

76:14.999 IOCTL '\2'

fffffa8031f94010 2 [ 0x e, 0x 0] fffffa80335c2010 fffffa803216b600 fffff8a00f667610

44:46.553 IOCTL '\2'

fffffa80335e05a0 0 [ 0x 0, 0x 0] fffffa803453ec20 fffffa8033ff5080 fffff8a00daac010

25:47.250 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-16\F\051\F0516C6C8D4D24CED66C01341EBC0F71.DVS'

fffffa8033b59610 0 [ 0x 0, 0x 0] fffffa80342b9600 fffffa806f8af5c0 fffff8a0118532a0

25:47.248 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D856FB14A3D5416A61EF0D7116911~0F~C97B4131~00~1.DVSSP'

fffffa8033c93240 0 [ 0x 0, 0x 0] fffffa8033b2c580 fffffa8034381b00 fffff8a00da1d600

25:47.248 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2014\07-17\6\0A5\60A5BB94AFB6B48F2B15E53045337701~35~1FD28490~00~1.DVSSP'

fffffa806ef187f0 0 [ 0x 0, 0x 0] fffffa80340a4630 fffffa806e0c0080 fffff8a010c947e0

25:46.524 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-17\D\06B\D06B43ABD3EEB44CA7CB2FE2CAB27721~39~6A552458~00~1.DVSSP'

fffffa80336269a0 0 [ 0x 0, 0x 0] fffffa8033528780 fffffa806e0fa080 fffff8a012ccf010

25:46.499 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\03-05\3\03B\303B66B5ACE86D5696529DF90977A8F1~6D~5FD6F55D~00~1.DVSSP'

fffffa8034749cb0 0 [ 0x 0, 0x 0] fffffa806f030750 fffffa806f53e080 fffff88001b998c0

13:29.967 CREATE '<<empty>>'

fffffa806f35fc20 0 [ 0x 0, 0x 0] fffffa80343ced10 fffffa803417fb00 fffff8a00d22e350

25:47.247 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D8ED60F35534A9D9860B16528A501~C4~46589CF8~00~1.DVSSP'

fffffa803444e010 0 [ 0x 0, 0x 0] fffffa806f440010 fffffa806e078080 fffff8a00dd3e2f0

25:47.241 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D8CD7E12525CA7EE50328316C7AF1~85~6174E8BD~00~1.DVSSP'

fffffa8034151010 0 [ 0x 0, 0x 0] fffffa8034366690 fffffa806e0fab00 fffff8a00d89d8e0

25:47.234 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E4\F0E4B2D443D4F8D070887D83728C6411.DVS'

fffffa8033fe7cb0 0 [ 0x 0, 0x 0] fffffa8033bf2d80 fffffa806e031b00 fffff8a00d66b500

25:47.234 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D83CE4F080D152AB519841E0F9551.DVS'

fffffa806e9acb30 0 [ 0x 0, 0x 0] fffffa8033468940 fffffa806e0bfb00 fffff8a00e6bf010

25:47.234 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\08-18\6\11D\611D8585CAAFD30258F5F9204E8B8F21.DVS'

fffffa806f214cb0 0 [ 0x 0, 0x 0] fffffa8033e719a0 fffffa806f56e080 fffff8a012de3010

25:46.500 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\06-15\5\048\50488155ACE08770781A75BBD1F269C1~29~B78492F0~00~1.DVSSP'

fffffa8034747cb0 0 [ 0x 0, 0x 0] fffffa806f567380 fffffa806ebda480 fffff88001b998c0

13:29.967 CREATE '<<empty>>'

fffffa80346e2010 0 [ 0x 0, 0x 0] fffffa8033c48730 fffffa803424ca80 fffff88001b998c0

13:29.967 CREATE '<<empty>>'

fffffa80349fd310 0 [ 0x 0, 0x 0] fffffa8033d55b00 fffffa8033dbd100 fffff88001b998c0

3:29.963 CREATE '<<empty>>'

fffffa803493e010 0 [ 0x 0, 0x 0] fffffa8033c3c010 fffffa80341a1b00 fffff88001b998c0

3:29.962 CREATE '<<empty>>'

fffffa806f6fe010 0 [ 0x 0, 0x 0] fffffa806f32b680 fffffa806e0bdb00 fffff8a010d06730

25:47.234 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E9\F0E9001E013394D9C07EF02330BFB911.DVS'

fffffa806ec38010 0 [ 0x 0, 0x 0] fffffa806f07e700 fffffa806f366b00 fffff8a012ca2a80

25:47.225 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-18\F\0E7\F0E756057B9D6D2ABECB8E13E3415CC1.DVS'

fffffa806ee48230 0 [ 0x 0, 0x 0] fffffa806f132010 fffffa806e0be6c0 fffff8a01242a010

25:47.014 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2014\12-20\D\0C2\D0C292F3FB7A73F70208526C10EAF491.DVS'

fffffa806f78e9a0 0 [ 0x 0, 0x 0] fffffa806e8e12c0 fffffa8034337b00 fffff8a004aea010

25:46.501 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-17\C\0E1\C0E1AAE496323E417F8B4BEFFAAE1FB1~D5~E770E121~00~1.DVSSP'

fffffa806eb5a450 0 [ 0x 0, 0x 0] fffffa806ef3b6b0 fffffa806ee98b00 fffff88001b998c0

13:29.967 CREATE '<<empty>>'

fffffa806f5f87f0 0 [ 0x 0, 0x 0] fffffa806f677010 fffffa806e0bf080 fffff8a00ffa4a60

25:47.190 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP02\2015\05-29\5\0BC\50BCEB2E36C8C4FABB47EC3453CB13B1~F3~00A34DF4~00~1.DVSSP'

fffffa806f1d9770 0 [ 0x 0, 0x 0] fffffa806f46d390 fffffa806f048080 fffff8a01282a010

25:47.249 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\01-16\1\124\1124F16CCD8694D455954CAABDF10111~97~5E195B14~00~1.DVSSP'

fffffa806f71ecb0 0 [ 0x 0, 0x 0] fffffa806f543370 fffffa806f5cb080 fffff8a012705a80

25:47.248 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-15\1\00F\100F697A6B99B08748C65EBD6A56BC21.DVS'

fffffa806f703900 0 [ 0x 0, 0x 0] fffffa806f742010 fffffa806e0c06c0 fffff8a012b51380

25:47.221 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-13\F\011\F0116842118398CEE7462D0AE65D7601.DVS'

fffffa806eeee780 0 [ 0x 0, 0x 0] fffffa80345d6a10 fffffa806f38b300 fffff88001b998c0

3:29.962 CREATE '<<empty>>'

fffffa806f529190 0 [ 0x 0, 0x 0] fffffa806f7f8320 fffffa806e0bab00 fffff8a012b145b0

25:47.249 CREATE '\Hmelbthdd\bth-evmjavsp01'

fffffa806eb4b910 0 [ 0x 0, 0x 0] fffffa8034263e10 fffffa803438e080 fffff8a010b89a80

25:47.027 CREATE '\Hmelbthdd\bth-evmjavsp01\EVMJAVSP01\2015\08-16\F\054\F054CC18A0FC36B2B94C5D416455D4B1.DVS'

fffffa803415c010 0 [ 0x 0, 0x 0] fffffa806f800510 fffffa806f267680 fffff88001b998c0

13:29.967 CREATE '<<empty>>'

fffffa806ebee830 0 [ 0x 0, 0x 0] fffffa806f6fd010 fffffa8033ee4080 fffff88001b998c0

13:29.966 CREATE '<<empty>>'

SMB client has several Irps stuck in the networking stack for a long time

Time Pending IRP

25:47.251 fffffa806f775010

16:08.472 fffffa8033fb3b90

6: kd> !irp fffffa806f775010

Irp is active with 2 stacks 1 is current (= 0xfffffa806f7750e0)

No Mdl: No System Buffer: Thread 00000000: Irp stack trace.

cmd flg cl Device File Completion-Context

>[IRP_MJ_INTERNAL_DEVICE_CONTROL(f), N/A(10)]

0 e1 fffffa80315c6c10 00000000 fffff8800439d8b0-fffffa8033a6fd10 Success Error Cancel pending

\Driver\AFD mrxsmb!SmbWskGetAddressInfoComplete

Args: fffffa803209e410 fffff880096e5ae0 fffffa803392d5c0 00000000

[N/A(0), N/A(0)]

0 0 00000000 00000000 00000000-00000000

Args: fffff88001b97a00 fffff88001b97a00 fffffa806f775010 7184f95f

6: kd> !irp fffffa8033fb3b90

Irp is active with 2 stacks 1 is current (= 0xfffffa8033fb3c60)

No Mdl: No System Buffer: Thread 00000000: Irp stack trace.

cmd flg cl Device File Completion-Context

>[IRP_MJ_INTERNAL_DEVICE_CONTROL(f), N/A(10)]

0 e1 fffffa80315c6c10 00000000 fffff8800439d8b0-fffffa806f77a240 Success Error Cancel pending

\Driver\AFD mrxsmb!SmbWskGetAddressInfoComplete

Args: fffffa803209e410 fffff8800bf71c50 fffffa8033c71ec0 00000000

[N/A(0), N/A(0)]

0 0 00000000 00000000 00000000-00000000

Args: fffff88001b98b00 fffff88001b98b00 fffffa8033fb3b90 718dce3a

Looks like these Irps are calls from SMB client to DNS client to resolve names. NDIS should send up-call back to the DNS client service in user mode to resolve them. DNS client service is hosted in one of the svchost.exe processes. I see many threads from svchost processes are stuck in TmXPFlt so it is possible that this is what is causing the deadlock.

Regards, Vladimir Petter, Microsoft Corporation

_{Thursday, September 10, 2015 7:12 AM}

Thank you very much Vladimir for your immediate review of memory dump, we have taken required action on Trend Micro Antivirus and cluster nodes are under observation.

_{Friday, September 11, 2015 12:25 PM}

Hello Vladimir,

Could you please help me to understand what command did you use to get all thread, specifically THREAD fffffa806e078080 , also how did you see the list of all opens threads that are stack over SMB, and how did you check the stuck IRP's in SMB.

Thanks & Regards,

Suchit Patil

_{Friday, September 11, 2015 12:26 PM}

Hello Vladimir.

I know this is an old post, but I would appreciate if you can analyse another minidump file about 0x0000009E message.

It´s an Exchange Mailbox Server.

https://dl.dropboxusercontent.com/u/12198734/091015-233437-01.dmp

Tks a lot

Dan Pinheiro

Rio de Janeiro - Brasil

Dan Pinheiro RJ - Brasil

_{Friday, September 11, 2015 6:51 PM}

Hello Suchit,

To dump ongoing SMB requests I am using a debugger extension that unfortunately is not publically released. As a workaround you can always find synchronies SMB requests using "!stacks 2 mrx" or "!stacks 2 rdbss"

Regards, Vladimir Petter, Microsoft Corporation

_{Friday, September 11, 2015 6:56 PM}

Hello Dan, what you've shared with me is a kernel minidump, which would not have thread stacks, and unfortunately would not be useful. Please configure full memory dump

http://www.bing.com/search?q=how+to+configure+windows+to+collect+full+kernel+dump&qs=n&form=QBLH&pq=how+to+configure+windows+to+collect+full+kernel+dump&sc=0-36&sp=-1&sk=&cvid=186d14b3517640b8896347f68916c892

And once you get one feel free to share it with me.

Regards, Vladimir Petter, Microsoft Corporation

_{Monday, October 19, 2015 4:07 PM}

Vladimir,

I've also run into this issue and would appreciate you taking a look at a dump file if you are still willing and available?

_{Tuesday, October 20, 2015 2:00 AM}

Sure, I can take a look. Please share a kernel dump with me.

Regards, Vladimir Petter, Microsoft Corporation

_{Tuesday, October 20, 2015 12:25 PM}

Thank you it's much appreciated. Should be able to download the dump from the link below.

_{Wednesday, October 21, 2015 6:48 AM}

I think you are running into a known issue. My guess this is happening when backup is reading data from snapshot. In this case it is reading 700GB VHD file, which builds a big section. 32GB of RAM on this machine allows that file to fit into the file cache. Eventually memory manager tries to trim active pages from the file cache and move them to stand by list. While doing that it is holding a lock on the thread fffffa803f730080. It has been doing that for about 1 minute on CPU 15

THREAD fffffa803f730080 Cid 0e60.1944 Teb: 000007f7d3a06000 Win32Thread: 0000000000000000 RUNNING on processor 15

IRP List:

fffffa803e217010: (0006,01f0) Flags: 00060070 Mdl: 00000000

Not impersonating

DeviceMap fffff8a00b793f00

Owning Process fffffa8020179980 Image: vmwp.exe

Attached Process N/A Image: N/A

Wait Start TickCount 9213470 Ticks: 283 (0:00:00:04.414)

Context Switch Count 99 IdealProcessor: 12

UserTime 00:00:00.000

KernelTime 00:01:05.286

Win32 Start Address 0x000007ff24521290

Stack Init fffff88007075c90 Current fffff88007074a80

Base fffff88007076000 Limit fffff88007070000 Call 0

Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

fffff880`07074e00 fffff803`494593f4 nt!MiTrimSection+0x18e

fffff880`07074f60 fffff803`49457bda nt!MiTrimSharedPage+0x188

fffff880`07075000 fffff803`49459059 nt!MiClaimPhysicalRun+0x38a

fffff880`07075080 fffff803`495f6170 nt!MiFindContiguousPages+0x39d

fffff880`070751f0 fffff803`49545efd nt!MiAllocatePagesForMdl+0xb1f64

fffff880`070752f0 fffff880`065d8454 nt!MmAllocateNodePagesForMdlEx+0x75

fffff880`07075340 fffff880`065d89ca Vid!VsmmpAllocatePagesFromNumaNode+0x58

fffff880`07075390 fffff880`065bd947 Vid!VsmmAllocatePagesStrictQoS+0x10a

fffff880`07075440 fffff880`065da6e8 Vid!VsmmReserveBucketReservePages+0xcb

fffff880`070754b0 fffff880`065b7f50 Vid!VsmmReservePages+0x160

fffff880`07075560 fffff880`065b761f Vid!VidIoControlPartition+0x7d0

fffff880`07075630 fffff880`065b7708 Vid!VidIoControlDispatch+0x143

fffff880`070756e0 fffff880`01078706 Vid!VidIoControlPreProcess+0x5c

(Inline Function) ` Wdf01000!PreprocessIrp+0x3f (Inline Function @ fffff880`01078706)

(Inline Function) ` Wdf01000!DispatchWorker+0xe7 (Inline Function @ fffff880`01078706)

fffff880`07075730 fffff880`01078866 Wdf01000!FxDevice::Dispatch+0x106

fffff880`07075770 fffff803`498822e8 Wdf01000!FxDevice::DispatchWithLock+0xa6

(Inline Function) ` nt!IoCallDriverWithTracing+0x1b (Inline Function @ fffff803`498822e8)

(Inline Function) ` nt!IopCallDriverReference+0x9b (Inline Function @ fffff803`498822e8)

fffff880`070757b0 fffff803`49841d23 nt!IopSynchronousServiceTail+0x158

fffff880`07075880 fffff803`4985b406 nt!IopXxxControlFile+0xde0

fffff880`07075a20 fffff803`4946b453 nt!NtDeviceIoControlFile+0x56

fffff880`07075a90 000007ff`244e2bda nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07075b00)

00000011`b777f828 00000000`00000000 0x000007ff`244e2bda

Meanwhile cluster service is trying to send a heartbeat to the kernel, and the thread that is supposed to do that got blocked on the same lock while trying to lock a page.

THREAD fffffa802750e080 Cid 09cc.1048 Teb: 000007f73e43c000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable

fffff880089b6360 SynchronizationEvent

IRP List:

fffffa80198536c0: (0006,0118) Flags: 00060000 Mdl: 00000000

Not impersonating

DeviceMap fffff8a00000c370

Owning Process fffffa801eeb1700 Image: clussvc.exe

Attached Process N/A Image: N/A

Wait Start TickCount 9209942 Ticks: 3811 (0:00:00:59.451)

Context Switch Count 10363 IdealProcessor: 5

UserTime 00:00:00.124

KernelTime 00:00:00.078

Win32 Start Address 0x000007ff24521290

Stack Init fffff880089b6c90 Current fffff880089b6060

Base fffff880089b7000 Limit fffff880089b1000 Call 0

Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

fffff880`089b60a0 fffff803`494af54b nt!KiSwapContext+0x76

(Inline Function) ` nt!KiSwapThread+0xfa (Inline Function @ fffff803`494af54b)

fffff880`089b61e0 fffff803`494ae38f nt!KiCommitThreadWait+0x23b

fffff880`089b62a0 fffff803`494f1746 nt!KeWaitForSingleObject+0x1cf

fffff880`089b6330 fffff803`494d36d1 nt!ExfAcquirePushLockExclusive+0x16d

(Inline Function) ` nt!ExAcquirePushLockExclusive+0x62 (Inline Function @ fffff803`494d36d1)

fffff880`089b63c0 fffff803`49447a62 nt!LOCK_WORKING_SET+0x79

fffff880`089b63f0 fffff803`49448218 nt!MiLockCode+0xc2

fffff880`089b6500 fffff803`4944839a nt!MiLockImageSection+0x58

fffff880`089b6550 fffff803`49806f58 nt!MiLockPagableImageSection+0xda

fffff880`089b6590 fffff880`013759c0 nt!MmLockPagableSectionByHandle+0x68

(Inline Function) ` NDIS!ndisReferencePackage+0x22 (Inline Function @ fffff880`013759c0)

fffff880`089b65c0 fffff803`498822e8 NDIS!ndisDeviceControlIrpHandler+0x98

(Inline Function) ` nt!IoCallDriverWithTracing+0x1b (Inline Function @ fffff803`498822e8)

(Inline Function) ` nt!IopCallDriverReference+0x9b (Inline Function @ fffff803`498822e8)

fffff880`089b67b0 fffff803`49841d23 nt!IopSynchronousServiceTail+0x158

fffff880`089b6880 fffff803`4985b406 nt!IopXxxControlFile+0xde0

fffff880`089b6a20 fffff803`4946b453 nt!NtDeviceIoControlFile+0x56

fffff880`089b6a90 000007ff`244e2bda nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`089b6b00)

00000053`a6c9f1f8 00000000`00000000 0x000007ff`244e2bda

Because kernel component has not received heartbeat in 1 minute it bugchecked the machine with USER_MODE_HEALTH_MONITOR (9e).

As far as I remember this was fixed and a fix was backported to Windows Server 2012 R2. I do not remember if it was ported back to Windows Server 2012 (9200.17528.amd64fre.win8_gdr.150928-0600), which you are running.

Here is what I suggest to do.

Make sure that you have all the latest patches.
If you do and issue is still reproducing, then I suggest to call to Microsoft support and present them this analysis and the dump. They should be able to provide you a patch, and if necessary they will initiate backport of the fix to Windows Server 2012.

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, October 21, 2015 4:16 PM}

Thank you so much for taking a look at it at providing suggestions. I suspected it had something to do with backups due to the time frame I was seeing it happen. I thought I had applied all relevant patches and hotfixes but I will revisit that and make sure everything is applied and investigate further from there.

Thanks again for taking your time to look at this.

_{Friday, December 4, 2015 4:30 AM}

Hi Vladimir,

Would you please help me out with taking a look a dump-file ?

So far I have this:

Opened log file 'c:\temp\debuglog.txt'

8: kd> .sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols

************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*c:\symbols*http://msdl.microsoft.com/download/symbols
8: kd> .reload;!analyze -v;r;kv;lmnt;.logclose;q
Loading Kernel Symbols
...............................................................
................................................................
..................
Loading User Symbols

Loading unloaded module list
............
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

USER_MODE_HEALTH_MONITOR (9e)
One or more critical user mode components failed to satisfy a health check.
Hardware mechanisms such as watchdog timers can detect that basic kernel
services are not executing. However, resource starvation issues, including
memory leaks, lock contention, and scheduling priority misconfiguration,
may block critical user mode components without blocking DPCs or
draining the nonpaged pool.
Kernel components can extend watchdog timer functionality to user mode
by periodically monitoring critical applications. This bugcheck indicates
that a user mode health check failed in a manner such that graceful
shutdown is unlikely to succeed. It restores critical services by
rebooting and/or allowing application failover to other servers.
Arguments:
Arg1: fffffa80f7862060, WatchdogSourceRhsCleanup
Monitors that RHS process goes away when
terminating on graceful exit
Arg2: 00000000000004b0, WatchdogSourceRhsResourceDeadlockBugcheckNow
RHS was asked to immediately bugcheck machine
on resource deadlock
Arg3: 0000000000000000, WatchdogSourceRhsExceptionFromResource
Resource has leaked unhandled exception from an entry point,
RHS is terminating and this watchdog monitors that
process will go away
Arg4: 0000000000000000, WatchdogSourceRhsUnhandledException
Unhandled exception in RHS.
RHS is terminating and this watchdog monitors that
process will go away
5 - WatchdogSourceRhsResourceDeadlock
Monitors that RHS process goes away when
terminating on resource deadlock
6 - WatchdogSourceRhsResourceTypeDeadlock
Monitors that RHS process goes away when
terminating on resource type deadlock
7 - WatchdogSourceClussvcUnhandledException
Unhandled exception in clussvc.
clussvc is terminating and this watchdog monitors that
process will go away
8 - WatchdogSourceClussvcBugcheckMessageRecieved
Another cluster node has send message asking to bugcheck this node.
9 - WatchdogSourceClussvcWatchdogBugcheck
User mode watchdog has expired and created netft watchdog
to bugchecked the node.
10 - WatchdogSourceClussvcIsAlive
Cluster service sends heartbeat to netft every 500 millseconds.
By default netft expects at least 1 heartbeat per second.
If this watchdog was triggered that means clussvc is o not getting
CPU to send heartbers.
101 - WatchdogSourceRhsResourceDeadlockPhysicalDisk
102 - WatchdogSourceRhsResourceDeadlockStoragePool
103 - WatchdogSourceRhsResourceDeadlockFileServer
104 - WatchdogSourceRhsResourceDeadlockSODAFileServer
105 - WatchdogSourceRhsResourceDeadlockStorageReplica
106 - WatchdogSourceRhsResourceDeadlockStorageQOS
107 - WatchdogSourceRhsResourceDeadlockStorageNFSV2
Values above are subclasses of WatchdogSourceRhsResourceDeadlock
For the resource types that are known to cause RHS termination hangs
we isolate them to separate codes so we can quickly triage dumps.
201 - WatchdogSourceRhsResourceTypeDeadlockPhysicalDisk
202 - WatchdogSourceRhsResourceTypeDeadlockStoragePool
203 - WatchdogSourceRhsResourceTypeDeadlockFileServer
204 - WatchdogSourceRhsResourceTypeDeadlockSODAFileServer
205 - WatchdogSourceRhsResourceTypeDeadlockStorageReplica
206 - WatchdogSourceRhsResourceTypeDeadlockStorageQOS
207 - WatchdogSourceRhsResourceTypeDeadlockStorageNFSV2
Values above are subclasses of WatchdogSourceRhsResourceTypeDeadlock
For the resource types that are known to cause RHS termination hangs
we isolate them to separate codes so we can quickly triage dumps.

Debugging Details:

DUMP_CLASS: 1

DUMP_QUALIFIER: 401

BUILD_VERSION_STRING: 7601.18798.amd64fre.win7sp1_gdr.150316-1654

SYSTEM_MANUFACTURER: HP

SYSTEM_PRODUCT_NAME: ProLiant BL460c Gen8

SYSTEM_SKU: 641016-B21

BIOS_VENDOR: HP

BIOS_VERSION: I31

BIOS_DATE: 12/20/2013

DUMP_TYPE: 1

BUGCHECK_P1: fffffa80f7862060

BUGCHECK_P2: 4b0

BUGCHECK_P3: 0

BUGCHECK_P4: 0

PROCESS_NAME: rhs.exe

IMAGE_NAME: rhs.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 0

MODULE_NAME: rhs

FAULTING_MODULE: 0000000000000000

CPU_COUNT: 10

CPU_MHZ: ce4

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 2d

CPU_STEPPING: 7

CPU_MICROCODE: 6,2d,7,0 (F,M,S,R) SIG: 710'00000000 (cache) 710'00000000 (init)

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0x9E

CURRENT_IRQL: 2

ANALYSIS_SESSION_HOST: ATC-WJUMP

ANALYSIS_SESSION_TIME: 12-03-2015 18:10:10.0807

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

STACK_TEXT:
fffff880`0253f3d8 fffff880`0512d6a5 : 00000000`0000009e fffffa80`f7862060 00000000`000004b0 00000000`00000000 : nt!KeBugCheckEx
fffff880`0253f3e0 fffff800`0209c91c : fffff880`0253f4c0 00000000`00000001 00000000`40390088 fffff880`014cbc94 : netft!NetftWatchdogTimerDpc+0xb9
fffff880`0253f430 fffff800`0209c7b6 : fffff880`0513a100 00000000`010322f9 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x6c
fffff880`0253f4a0 fffff800`0209c69e : 00000268`d86300d3 fffff880`0253fb18 00000000`010322f9 fffff880`0251b4a8 : nt!KiProcessExpiredTimerList+0xc6
fffff880`0253faf0 fffff800`0209c487 : 000000c6`cac035c1 000000c6`010322f9 000000c6`cac035cf 00000000`000000f9 : nt!KiTimerExpiration+0x1be
fffff880`0253fb90 fffff800`0208874a : fffff880`02517180 fffff880`025222c0 00000000`00000001 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff880`0253fc40 00000000`00000000 : fffff880`02540000 fffff880`0253a000 fffff880`0253fc00 00000000`00000000 : nt!KiIdleLoop+0x5a

STACK_COMMAND: kb

THREAD_SHA1_HASH_MOD_FUNC: 816f11a87ad4e6cc4ea0c0bae49953386adf648a

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 5d45c8c68fd994a4bdace855a86f47ca470766fe

THREAD_SHA1_HASH_MOD: 62ffe22e249d6ed7a775c30201445f402c1cebbf

FOLLOWUP_NAME: MachineOwner

IMAGE_VERSION:

FAILURE_BUCKET_ID: X64_0x9E_IMAGE_rhs.exe

BUCKET_ID: X64_0x9E_IMAGE_rhs.exe

PRIMARY_PROBLEM_CLASS: X64_0x9E_IMAGE_rhs.exe

TARGET_TIME: 2015-11-23T12:43:44.000Z

OSBUILD: 7601

OSSERVICEPACK: 1000

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK: 274

PRODUCT_TYPE: 3

OSPLATFORM_TYPE: x64

OSNAME: Windows 7

OSEDITION: Windows 7 Server (Service Pack 1) Enterprise TerminalServer SingleUserTS

OS_LOCALE:

USER_LCID: 0

OSBUILD_TIMESTAMP: 2015-03-17 00:02:04

BUILDDATESTAMP_STR: 150316-1654

BUILDLAB_STR: win7sp1_gdr

BUILDOSVER_STR: 6.1.7601.18798.amd64fre.win7sp1_gdr.150316-1654

ANALYSIS_SESSION_ELAPSED_TIME: 15d0

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:x64_0x9e_image_rhs.exe

FAILURE_ID_HASH: {c259dc81-ac67-ae55-7e62-cf1c10f90628}

Followup: MachineOwner

rax=0000000000000000 rbx=fffffa80f9c091d0 rcx=000000000000009e
rdx=fffffa80f7862060 rsi=fffff88005139110 rdi=fffff8800253f400
rip=fffff80002090a40 rsp=fffff8800253f3d8 rbp=0000000000000000
r8=00000000000004b0 r9=0000000000000000 r10=fffff8000201e000
r11=fffffa806e4a4c10 r12=fffff8800513a0e0 r13=fffff8800253fb18
r14=fffff880025222c0 r15=fffff88002517180
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000246
nt!KeBugCheckEx:
fffff800`02090a40 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:fffff880`0253f3e0=000000000000009e
Child-SP RetAddr : Args to Child : Call Site
fffff880`0253f3d8 fffff880`0512d6a5 : 00000000`0000009e fffffa80`f7862060 00000000`000004b0 00000000`00000000 : nt!KeBugCheckEx
fffff880`0253f3e0 fffff800`0209c91c : fffff880`0253f4c0 00000000`00000001 00000000`40390088 fffff880`014cbc94 : netft!NetftWatchdogTimerDpc+0xb9
fffff880`0253f430 fffff800`0209c7b6 : fffff880`0513a100 00000000`010322f9 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x6c
fffff880`0253f4a0 fffff800`0209c69e : 00000268`d86300d3 fffff880`0253fb18 00000000`010322f9 fffff880`0251b4a8 : nt!KiProcessExpiredTimerList+0xc6
fffff880`0253faf0 fffff800`0209c487 : 000000c6`cac035c1 000000c6`010322f9 000000c6`cac035cf 00000000`000000f9 : nt!KiTimerExpiration+0x1be
fffff880`0253fb90 fffff800`0208874a : fffff880`02517180 fffff880`025222c0 00000000`00000001 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff880`0253fc40 00000000`00000000 : fffff880`02540000 fffff880`0253a000 fffff880`0253fc00 00000000`00000000 : nt!KiIdleLoop+0x5a
start end module name
fffff800`01faf000 fffff800`01fb9000 kdcom kdcom.dll Mon Jul 13 21:31:07 2009 (4A5BDFDB)
fffff800`0201e000 fffff800`02606000 nt ntkrnlmp.exe Tue Mar 17 00:02:04 2015 (5507A73C)
fffff800`02606000 fffff800`0264f000 hal hal.dll Sat Nov 20 08:00:25 2010 (4CE7C669)
fffff880`00c00000 fffff880`00c52e00 mfewfpk mfewfpk.sys Fri Jan 30 13:42:08 2015 (54CBD080)
fffff880`00c53000 fffff880`00c80000 mrxsmb mrxsmb.sys Tue Jan 06 20:49:30 2015 (54AC90AA)
fffff880`00c81000 fffff880`00cd0000 mcupdate_GenuineIntel mcupdate_GenuineIntel.dll Sat Nov 20 08:03:51 2010 (4CE7C737)
fffff880`00cd0000 fffff880`00ce4000 PSHED PSHED.dll Mon Jul 13 21:32:23 2009 (4A5BE027)
fffff880`00ce4000 fffff880`00d42000 CLFS CLFS.SYS Tue Mar 03 22:21:12 2015 (54F67A28)
fffff880`00d42000 fffff880`00da5000 storport storport.sys Sat Nov 20 05:35:02 2010 (4CE7A456)
fffff880`00da5000 fffff880`00dd6000 be2iscsi be2iscsi.sys Sun Aug 11 04:57:21 2013 (520751F1)
fffff880`00e00000 fffff880`00e3a000 involflt involflt.sys Tue Aug 05 22:35:00 2014 (53E19454)
fffff880`00e3a000 fffff880`00e5e000 hpeaadsm hpeaadsm.sys Mon Jan 10 00:07:38 2011 (4D2A941A)
fffff880`00e5e000 fffff880`00e89000 HpCISSs2 HpCISSs2.sys Mon Oct 28 11:10:30 2013 (526E7E66)
fffff880`00e9d000 fffff880`00f5d000 CI CI.dll Sat Nov 20 08:12:36 2010 (4CE7C944)
fffff880`00f78000 fffff880`00fd4000 volmgrx volmgrx.sys Sat Nov 20 04:20:43 2010 (4CE792EB)
fffff880`00fd4000 fffff880`00feb000 HpSAMD HpSAMD.sys Tue Apr 20 14:32:18 2010 (4BCDF332)
fffff880`01000000 fffff880`01060000 NETIO NETIO.SYS Tue Nov 26 05:21:01 2013 (5294760D)
fffff880`01060000 fffff880`0107a000 mountmgr mountmgr.sys Sat Nov 20 04:19:21 2010 (4CE79299)
fffff880`01082000 fffff880`01175000 NDIS NDIS.SYS Sat Nov 20 04:23:30 2010 (4CE79392)
fffff880`01175000 fffff880`011d3000 msrpc msrpc.sys Sat Nov 20 04:21:56 2010 (4CE79334)
fffff880`011d3000 fffff880`011f9000 msdsm msdsm.sys Fri May 31 23:58:14 2013 (51A97156)
fffff880`01200000 fffff880`0122b000 mpio mpio.sys Thu Nov 29 22:55:16 2012 (50B82E24)
fffff880`0122b000 fffff880`01240000 partmgr partmgr.sys Sat Mar 17 01:06:09 2012 (4F641BC1)
fffff880`01240000 fffff880`01249000 compbatt compbatt.sys Mon Jul 13 19:31:02 2009 (4A5BC3B6)
fffff880`01249000 fffff880`01255000 BATTC BATTC.SYS Mon Jul 13 19:31:01 2009 (4A5BC3B5)
fffff880`01260000 fffff880`01322000 Wdf01000 Wdf01000.sys Fri Jun 21 23:13:05 2013 (51C51641)
fffff880`01322000 fffff880`01332000 WDFLDR WDFLDR.SYS Wed Jul 25 22:29:04 2012 (5010AB70)
fffff880`01332000 fffff880`01389000 ACPI ACPI.sys Sat Nov 20 04:19:16 2010 (4CE79294)
fffff880`01389000 fffff880`01392000 WMILIB WMILIB.SYS Mon Jul 13 19:19:51 2009 (4A5BC117)
fffff880`01392000 fffff880`0139c000 msisadrv msisadrv.sys Mon Jul 13 19:19:26 2009 (4A5BC0FE)
fffff880`0139c000 fffff880`013cf000 pci pci.sys Sat Nov 20 04:19:11 2010 (4CE7928F)
fffff880`013cf000 fffff880`013dc000 vdrvroot vdrvroot.sys Mon Jul 13 20:01:31 2009 (4A5BCADB)
fffff880`013dc000 fffff880`013f1000 volmgr volmgr.sys Sat Nov 20 04:19:28 2010 (4CE792A0)
fffff880`01418000 fffff880`01588000 ql2300 ql2300.sys Thu Mar 07 13:19:46 2013 (5138DA42)
fffff880`01588000 fffff880`01593000 amdxata amdxata.sys Fri Mar 19 12:18:18 2010 (4BA3A3CA)
fffff880`01593000 fffff880`015df000 fltmgr fltmgr.sys Sat Nov 20 04:19:24 2010 (4CE7929C)
fffff880`0161f000 fffff880`016f4700 mfehidk mfehidk.sys Fri Jan 30 13:41:34 2015 (54CBD05E)
fffff880`016f5000 fffff880`01767000 cng cng.sys Thu Jan 29 22:17:08 2015 (54CAF7B4)
fffff880`01767000 fffff880`01793000 ksecpkg ksecpkg.sys Fri Apr 03 22:15:42 2015 (551F494E)
fffff880`01793000 fffff880`017dc000 fwpkclnt fwpkclnt.sys Fri Apr 04 21:23:21 2014 (533F5B09)
fffff880`017dc000 fffff880`017e9000 TDI TDI.SYS Sat Nov 20 04:22:06 2010 (4CE7933E)
fffff880`017e9000 fffff880`017ff000 intelppm intelppm.sys Mon Jul 13 19:19:25 2009 (4A5BC0FD)
fffff880`01800000 fffff880`0180a000 Fs_Rec Fs_Rec.sys Wed Feb 29 22:41:06 2012 (4F4EEFD2)
fffff880`0180a000 fffff880`01812000 hpqilo3whea hpqilo3whea.sys Fri Feb 12 18:33:27 2010 (4B75E547)
fffff880`01812000 fffff880`01822000 vmstorfl vmstorfl.sys Sat Nov 20 04:57:30 2010 (4CE79B8A)
fffff880`01829000 fffff880`019cb000 Ntfs Ntfs.sys Fri Apr 12 07:54:36 2013 (5167F5FC)
fffff880`019cb000 fffff880`019e6000 ksecdd ksecdd.sys Fri Apr 03 21:58:36 2015 (551F454C)
fffff880`019e6000 fffff880`019f7000 pcw pcw.sys Mon Jul 13 19:19:27 2009 (4A5BC0FF)
fffff880`01a00000 fffff880`01bff000 tcpip tcpip.sys Fri Apr 04 21:26:44 2014 (533F5BD4)
fffff880`01c00000 fffff880`01c22000 tdx tdx.sys Sat Nov 20 04:21:54 2010 (4CE79332)
fffff880`01c26000 fffff880`01c72000 volsnap volsnap.sys Sat Nov 20 04:20:08 2010 (4CE792C8)
fffff880`01c72000 fffff880`01c7a000 spldr spldr.sys Mon May 11 12:56:27 2009 (4A0858BB)
fffff880`01c7a000 fffff880`01c98000 mup mup.sys Tue Jan 06 20:48:27 2015 (54AC906B)
fffff880`01c98000 fffff880`01cae580 mfedisk mfedisk.sys Fri Jan 30 13:42:39 2015 (54CBD09F)
fffff880`01caf000 fffff880`01cb8000 hwpolicy hwpolicy.sys Sat Nov 20 04:18:54 2010 (4CE7927E)
fffff880`01cb8000 fffff880`01cce000 disk disk.sys Mon Jul 13 19:19:57 2009 (4A5BC11D)
fffff880`01cce000 fffff880`01cfe000 CLASSPNP CLASSPNP.SYS Sat Nov 20 04:19:23 2010 (4CE7929B)
fffff880`01cfe000 fffff880`01d1c000 bowser bowser.sys Tue Feb 22 23:55:04 2011 (4D649328)
fffff880`01d41000 fffff880`01d61000 DpmFilter DpmFilter.sys Mon Sep 17 06:52:03 2012 (505700D3)
fffff880`01d61000 fffff880`01d6f000 ClusDisk ClusDisk.sys Sat Nov 20 04:42:24 2010 (4CE79800)
fffff880`01d6f000 fffff880`01d77000 InCDskFl InCDskFl.sys Tue Aug 05 22:49:45 2014 (53E197C9)
fffff880`01d77000 fffff880`01d80000 Null Null.SYS Mon Jul 13 19:19:37 2009 (4A5BC109)
fffff880`01d80000 fffff880`01d8e000 vga vga.sys Mon Jul 13 19:38:47 2009 (4A5BC587)
fffff880`01d8e000 fffff880`01db3000 VIDEOPRT VIDEOPRT.SYS Mon Jul 13 19:38:51 2009 (4A5BC58B)
fffff880`01db3000 fffff880`01dc3000 watchdog watchdog.sys Mon Jul 13 19:37:35 2009 (4A5BC53F)
fffff880`01dc3000 fffff880`01dcc000 RDPCDD RDPCDD.sys Mon Jul 13 20:16:34 2009 (4A5BCE62)
fffff880`01dcc000 fffff880`01dd5000 rdpencdd rdpencdd.sys Mon Jul 13 20:16:34 2009 (4A5BCE62)
fffff880`01dd5000 fffff880`01dde000 rdprefmp rdprefmp.sys Mon Jul 13 20:16:35 2009 (4A5BCE63)
fffff880`01dde000 fffff880`01de9000 Msfs Msfs.SYS Mon Jul 13 19:19:47 2009 (4A5BC113)
fffff880`01de9000 fffff880`01dfa000 Npfs Npfs.SYS Mon Jul 13 19:19:48 2009 (4A5BC114)
fffff880`04c01000 fffff880`04c46000 netbt netbt.sys Sat Nov 20 04:23:18 2010 (4CE79386)
fffff880`04c46000 fffff880`04ccf000 afd afd.sys Fri May 30 02:45:48 2014 (5388291C)
fffff880`04ccf000 fffff880`04cd8000 wfplwf wfplwf.sys Mon Jul 13 20:09:26 2009 (4A5BCCB6)
fffff880`04cd8000 fffff880`04cfe000 pacer pacer.sys Sat Nov 20 05:52:18 2010 (4CE7A862)
fffff880`04cfe000 fffff880`04d0d000 netbios netbios.sys Mon Jul 13 20:09:26 2009 (4A5BCCB6)
fffff880`04d0d000 fffff880`04d2a000 serial serial.sys Mon Jul 13 20:00:40 2009 (4A5BCAA8)
fffff880`04d2a000 fffff880`04d45000 wanarp wanarp.sys Sat Nov 20 05:52:36 2010 (4CE7A874)
fffff880`04d45000 fffff880`04d59000 termdd termdd.sys Sat Nov 20 06:03:40 2010 (4CE7AB0C)
fffff880`04d59000 fffff880`04daa000 rdbss rdbss.sys Tue Jan 06 20:49:42 2015 (54AC90B6)
fffff880`04daa000 fffff880`04db6000 nsiproxy nsiproxy.sys Mon Jul 13 19:21:02 2009 (4A5BC15E)
fffff880`04db6000 fffff880`04dc1000 mssmbios mssmbios.sys Mon Jul 13 19:31:10 2009 (4A5BC3BE)
fffff880`04dc1000 fffff880`04dd0000 discache discache.sys Mon Jul 13 19:37:18 2009 (4A5BC52E)
fffff880`04dd0000 fffff880`04def000 dfsc dfsc.sys Tue Jan 06 20:48:10 2015 (54AC905A)
fffff880`04def000 fffff880`04e00000 blbdrive blbdrive.sys Mon Jul 13 19:35:59 2009 (4A5BC4DF)
fffff880`04e16000 fffff880`04e92000 be2nd62 be2nd62.sys Sat Dec 07 01:04:36 2013 (52A2BA74)
fffff880`04e92000 fffff880`04ea4000 usbehci usbehci.sys Tue Nov 26 20:41:11 2013 (52954DB7)
fffff880`04ea4000 fffff880`04efa000 USBPORT USBPORT.SYS Tue Nov 26 20:41:11 2013 (52954DB7)
fffff880`04efa000 fffff880`04f09000 hpqilo3core hpqilo3core.sys Wed May 22 17:41:45 2013 (519D3B99)
fffff880`04f09000 fffff880`04f4a000 g200eHm g200eHm.sys Wed Apr 11 16:24:18 2012 (4F85E872)
fffff880`04f4a000 fffff880`04f59000 hpqilo3chif hpqilo3chif.sys Sat Nov 23 21:26:20 2013 (529163CC)
fffff880`04f59000 fffff880`04f66000 usbuhci usbuhci.sys Tue Nov 26 20:41:05 2013 (52954DB1)
fffff880`04f66000 fffff880`04f7e000 IPMIDrv IPMIDrv.sys Sat Nov 20 05:04:53 2010 (4CE79D45)
fffff880`04f7e000 fffff880`04f8a000 serenum serenum.sys Mon Jul 13 20:00:33 2009 (4A5BCAA1)
fffff880`04f8a000 fffff880`04fa8000 i8042prt i8042prt.sys Mon Jul 13 19:19:57 2009 (4A5BC11D)
fffff880`04fa8000 fffff880`04fb7000 kbdclass kbdclass.sys Mon Jul 13 19:19:50 2009 (4A5BC116)
fffff880`04fb7000 fffff880`04fc6000 mouclass mouclass.sys Mon Jul 13 19:19:50 2009 (4A5BC116)
fffff880`04fc6000 fffff880`04fd5000 tpm tpm.sys Mon Jul 13 19:21:48 2009 (4A5BC18C)
fffff880`04fd5000 fffff880`04fdf000 acpipmi acpipmi.sys Sat Nov 20 04:30:42 2010 (4CE79542)
fffff880`04fdf000 fffff880`04fef000 CompositeBus CompositeBus.sys Sat Nov 20 05:33:17 2010 (4CE7A3ED)
fffff880`05000000 fffff880`05018000 rspndr rspndr.sys Mon Jul 13 20:08:50 2009 (4A5BCC92)
fffff880`0501e000 fffff880`05060000 cpqteam cpqteam.sys Wed Dec 11 05:30:40 2013 (52A83ED0)
fffff880`05060000 fffff880`05076000 AgileVpn AgileVpn.sys Mon Jul 13 20:10:24 2009 (4A5BCCF0)
fffff880`05076000 fffff880`0509a000 rasl2tp rasl2tp.sys Sat Nov 20 05:52:34 2010 (4CE7A872)
fffff880`0509a000 fffff880`050a6000 ndistapi ndistapi.sys Mon Jul 13 20:10:00 2009 (4A5BCCD8)
fffff880`050a6000 fffff880`050d5000 ndiswan ndiswan.sys Sat Nov 20 05:52:32 2010 (4CE7A870)
fffff880`050d5000 fffff880`050f0000 raspppoe raspppoe.sys Mon Jul 13 20:10:17 2009 (4A5BCCE9)
fffff880`050f0000 fffff880`05111000 raspptp raspptp.sys Sat Nov 20 05:52:31 2010 (4CE7A86F)
fffff880`05111000 fffff880`0512b000 rassstp rassstp.sys Mon Jul 13 20:10:25 2009 (4A5BCCF1)
fffff880`0512b000 fffff880`05147000 netft netft.sys Mon Jul 13 19:34:34 2009 (4A5BC48A)
fffff880`05147000 fffff880`05152000 rdpbus rdpbus.sys Mon Jul 13 20:17:46 2009 (4A5BCEAA)
fffff880`05152000 fffff880`05153480 swenum swenum.sys Mon Jul 13 20:00:18 2009 (4A5BCA92)
fffff880`05154000 fffff880`05197000 ks ks.sys Sat Nov 20 05:33:23 2010 (4CE7A3F3)
fffff880`05197000 fffff880`051a9000 umbus umbus.sys Sat Nov 20 05:44:37 2010 (4CE7A695)
fffff880`051a9000 fffff880`051b5000 Dxapi Dxapi.sys Mon Jul 13 19:38:28 2009 (4A5BC574)
fffff880`051b5000 fffff880`051c3000 monitor monitor.sys Mon Jul 13 19:38:52 2009 (4A5BC58C)
fffff880`051c3000 fffff880`051e6000 luafv luafv.sys Mon Jul 13 19:26:13 2009 (4A5BC295)
fffff880`051e6000 fffff880`051fb000 lltdio lltdio.sys Mon Jul 13 20:08:50 2009 (4A5BCC92)
fffff880`05200000 fffff880`0522b000 dump_HpCISSs2 dump_HpCISSs2.sys Mon Oct 28 11:10:30 2013 (526E7E66)
fffff880`0522f000 fffff880`05289000 usbhub usbhub.sys Tue Nov 26 20:41:36 2013 (52954DD0)
fffff880`05289000 fffff880`0529e000 NDProxy NDProxy.SYS Sat Nov 20 05:52:20 2010 (4CE7A864)
fffff880`0529e000 fffff880`052eea00 mfeavfk mfeavfk.sys Fri Jan 30 13:43:31 2015 (54CBD0D3)
fffff880`052ef000 fffff880`052fab00 mfeaacsk mfeaacsk.sys Fri Jan 30 13:48:09 2015 (54CBD1E9)
fffff880`052fb000 fffff880`05307100 mfeplk mfeplk.sys Fri Jan 30 13:48:10 2015 (54CBD1EA)
fffff880`05308000 fffff880`0535fb80 mfeaack mfeaack.sys Fri Jan 30 13:48:06 2015 (54CBD1E6)
fffff880`05360000 fffff880`053dfc80 mfefirek mfefirek.sys Fri Jan 30 13:44:56 2015 (54CBD128)
fffff880`053e0000 fffff880`053ee000 crashdmp crashdmp.sys Mon Jul 13 20:01:01 2009 (4A5BCABD)
fffff880`053ee000 fffff880`053f8000 dump_diskdump dump_diskdump.sys Sat Nov 20 05:33:56 2010 (4CE7A414)
fffff880`06e00000 fffff880`06ea6000 peauth peauth.sys Mon Jul 13 21:01:19 2009 (4A5BD8DF)
fffff880`06ea6000 fffff880`06eb1000 secdrv secdrv.SYS Wed Sep 13 09:18:38 2006 (4508052E)
fffff880`06ebe000 fffff880`06f0c000 mrxsmb10 mrxsmb10.sys Tue Jan 06 20:48:48 2015 (54AC9080)
fffff880`06f0c000 fffff880`06f30000 mrxsmb20 mrxsmb20.sys Tue Jan 06 20:48:38 2015 (54AC9076)
fffff880`06f30000 fffff880`06ff9000 HTTP HTTP.sys Tue Feb 24 22:17:55 2015 (54ED3EE3)
fffff880`07c00000 fffff880`07c98000 srv srv.sys Thu Apr 28 23:06:06 2011 (4DBA2B1E)
fffff880`07c98000 fffff880`07ca7000 tssecsrv tssecsrv.sys Sat Nov 20 06:04:09 2010 (4CE7AB29)
fffff880`07cc3000 fffff880`07cf4000 srvnet srvnet.sys Thu Apr 28 23:05:35 2011 (4DBA2AFF)
fffff880`07cf4000 fffff880`07d06000 tcpipreg tcpipreg.sys Sat Nov 20 05:51:48 2010 (4CE7A844)
fffff880`07d06000 fffff880`07d3f000 RDPWD RDPWD.SYS Fri Apr 27 23:55:20 2012 (4F9B6A28)
fffff880`07d57000 fffff880`07dc0000 srv2 srv2.sys Thu Apr 28 23:05:46 2011 (4DBA2B0A)
fffff880`07dc0000 fffff880`07dee000 rdpdr rdpdr.sys Sat Nov 20 06:06:41 2010 (4CE7ABC1)
fffff880`07dee000 fffff880`07df9000 tdtcp tdtcp.sys Thu Feb 16 23:57:32 2012 (4F3DDE3C)
fffff880`0b472000 fffff880`0b47d000 asyncmac asyncmac.sys Mon Jul 13 20:10:13 2009 (4A5BCCE5)
fffff960`00040000 fffff960`00363000 win32k win32k.sys Mon May 25 13:08:15 2015 (556356FF)
fffff960`00400000 fffff960`0041e000 dxg dxg.sys Mon Jul 13 19:38:28 2009 (4A5BC574)
fffff960`00710000 fffff960`0071a000 TSDDD TSDDD.dll Mon Jul 13 20:16:34 2009 (4A5BCE62)
fffff960`00860000 fffff960`00896000 G200eHd G200eHd.dll Wed Apr 11 16:24:18 2012 (4F85E872)
fffff960`00af0000 fffff960`00b38000 RDPDD RDPDD.dll Sat Nov 20 06:05:00 2010 (4CE7AB5C)

Unloaded modules:
fffff880`0b4ee000 fffff880`0b55f000 spsys.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00071000
fffff880`0b5b9000 fffff880`0b5e6000 mfeapfk.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0002D000
fffff880`0b58c000 fffff880`0b5b9000 mfeapfk.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0002D000
fffff880`0b55f000 fffff880`0b58c000 mfeapfk.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0002D000
fffff880`0b47d000 fffff880`0b4ee000 spsys.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00071000
fffff880`0b401000 fffff880`0b472000 spsys.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00071000
fffff880`07d06000 fffff880`07d33000 mfeapfk.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0002D000
fffff880`07d06000 fffff880`07d57000 mfeavfk01.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 00051000
fffff880`01cfe000 fffff880`01d0c000 crashdmp.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0000E000
fffff880`01d0c000 fffff880`01d16000 dump_storport.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0000A000
fffff880`01d16000 fffff880`01d41000 dump_HpCISSs2.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0002B000
fffff880`00f5d000 fffff880`00f78000 sacdrv.sys
Timestamp: unavailable (00000000)
Checksum: 00000000
ImageSize: 0001B000
Closing open log file c:\temp\debuglog.txt

_{Friday, December 4, 2015 6:58 PM}

Please share this dump with me.

Regards, Vladimir Petter, Microsoft Corporation

_{Thursday, December 31, 2015 6:04 PM}

Hi Vladimir

sorry to hijack this thread. First time posting

Would you be able to help me decipher my bugcheck crash?

Running exchange 2010 mailbox server on windows 2008r2. Server crashed yesterday with

The bugcheck was: 0x0000009e (0xfffffa8014390320, 0x000000000000003c, 0x0000000000000000, 0x0000000000000000).

the link to the memory dump is here

https://www.dropbox.com/s/oujf7etgx271a63/MEMORY.DMP?dl=0

thank you so much

_{Friday, January 1, 2016 1:42 AM}

Hi, I am getting errors trying to load this dump

THIS DUMP FILE IS PARTIALLY CORRUPT.
KdDebuggerDataBlock is not present or unreadable.

Unfortunately looks like this dump is corrupt.

Regards, Vladimir Petter, Microsoft Corporation

_{Friday, January 1, 2016 4:36 AM}

Have you made nay changes in Network settings like Teaming or change in Teaming mode from LACP to Auto or any thing?

because in this case you might be face problem if you have not restarted the server after making this change.

_{Monday, January 4, 2016 2:57 PM}

that is weird. server is a virtual machine running under vmware esx.

i guess i have to wait for another crash then

thank you for your prompt response

_{Tuesday, January 5, 2016 8:42 AM}

we run a 5 host hyper-v cluster with a SAN as CSV. we've had a few bugcheck E as well caused by netflt.sys .

One of the Hype-V hosts seem to suffer from BugCheck 9E, {ffffe800f6529900, 4b0, 5, 0}

I have uploaded support diagnostics using http://home.diagnostics.support.microsoft.com/SelfHelp

Not sure how to publish them here, but would be nice if somebody experienced from microsoft take a look at it. I am tired of calling Microsoft support as the support has recently degraded a lot and the support is just a waste of time.

_{Tuesday, January 5, 2016 9:33 AM}

Hi, Hopefully someone from support can take a look at this dump. I am in a completely different part of the org - engineering, and do not have access to the support infrastructure.

If you have support case opened then support folks should be able to help you with this dump.

Regards, Vladimir Petter, Microsoft Corporation

_{Tuesday, January 5, 2016 2:59 PM}

If this issue is causing a business outage, a support case really needs to be opened. This is not a support forum so hoping that a support engineer wanders over here (on their personal time, because like any support organization, they do have charge for company time) might take a while. If you are having issues with support, I would request an escalation. If you are paying for support, you need to demand support.

One thing you need to start with is looking at the physical environment running the ESX hosts. Even though they are running ESX, the components must still be certified to run the version of Windows Server running in the VM, in this case, 2008 R2. You can check at windowsservercatalog.com.

. : | : . : | : . tim

_{Thursday, January 28, 2016 1:16 PM}

Hello Vladimir,
Could you please help in reviewing memory dump uploaded at below link?

https://drive.google.com/file/d/0B7HTVM4VNM3SSEVDZElfZmEtejA/view?usp=sharing

one of the server rebooted due to prowershell. please find debug details

CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 0000000000000003, Process
Arg2: fffffa800d393c10, Terminating object
Arg3: fffffa800d393e48, Process image file name
Arg4: fffff80001eed8d0, Explanatory message (ascii)

Debugging Details:

Page e156a not present in the dump file. Type ".hh dbgerr004" for details
Page e1b5e not present in the dump file. Type ".hh dbgerr004" for details

PROCESS_OBJECT: fffffa800d393c10

IMAGE_NAME: csrss.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 0

MODULE_NAME: csrss

FAULTING_MODULE: 0000000000000000

PROCESS_NAME: wsmprovhost.exe

BUGCHECK_STR: 0xF4_wsmprovhost.exe

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff80001f69353 to fffff80001c6a450

STACK_TEXT:
fffffa60`0c598b28 fffff800`01f69353 : 00000000`000000f4 00000000`00000003 fffffa80`0d393c10 fffffa80`0d393e48 : nt!KeBugCheckEx
fffffa60`0c598b30 fffff800`01e82358 : fffffa80`0f23bbb0 fffffa80`0f23bbb0 00000000`1a78e4c0 00000000`00000000 : nt!PspCatchCriticalBreak+0x93
fffffa60`0c598b70 fffff800`01eb5f50 : fffffa80`0f23bbb0 00000000`00000008 00000000`1a78e4c0 fffff800`00000008 : nt! ?? ::NNGAKEGL::`string'+0x110f6
fffffa60`0c598bc0 fffff800`01c69ef3 : fffffa80`0d393c10 fffffa80`0f23bbb0 fffffa60`0c598ca0 00000000`1a78e4c0 : nt!NtTerminateProcess+0xd8
fffffa60`0c598c20 00000000`778e6fda : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`1a78e3b8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x778e6fda

Thanks & Regards,
Asif Iqbal

_{Friday, January 29, 2016 6:35 AM}

Hello Asif,

This thread has been about cluster bugcheking machine with 9e. You have hit a different issue. In your case user mode process csrss.exe is getting terminated. This is a critical OS process, and in response to its termination OS is bugchecking machine with CRITICAL_OBJECT_TERMINATION (f4). It looks like process termination was done by wsmprovhost.exe, which I guess is running some administrative action on behalf of a user with SID S-1-5-21-800109603-1914418702-385554406-17320. You can use PsGetSid (https://technet.microsoft.com/en-us/sysinternals/bb897417.aspx) to lookup user name from the SID.

Regards, Vladimir Petter, Microsoft Corporation

_{Thursday, February 11, 2016 4:09 AM}

Vladimir,

Would you be able to analyze the following minidump?

https://rbs.syncedtool.com/shares/file/4e7c8229e80029/

Please let me know if you need additional information. Thank you very much.

Brett

_{Thursday, February 11, 2016 6:30 AM}

Hello Brett,

You have a minidump. Unfortunately there is no much I can tell from. Please configure you servers to collect full kernel dump (http://blogs.msdn.com/b/xiz/archive/2013/11/04/how-to-setup-to-collect-memory-dump.aspx) and share a dump next time you get a repro.

In your case a resource call took over 5 minutes, and RHS.exe terminated itself. Because process termination did not complete in 20 minutes netft.sys bugchecked machine with 9e. Process termination cannot complete because one of the threads of that process is stuck waiting for the kernel to complete an IO. From my experience most often we find that this IO is stuck somewhere in the storage stack. If you will be able to get full kernel dump for 9e then I should be able to track down why IO is stuck.

I see this dump is from VMware virtual machine. Please check that at the time of failure VMware did not go through some storage problem.

Regards, Vladimir Petter, Microsoft Corporation

_{Monday, July 25, 2016 12:15 PM}

Hi Vladimir,

I have the same issue - 3 Node Hyper-V cluster - Windows 2012 R2, iSCSI Netapp Storage - 4 CSV disk, which are holding the VM`s.

Could you please check, if you have some time the dump I have. I have checked some things in it, but I am not so good with the Dump reading and analyzing:

https://drive.google.com/file/d/0B75QCMrGcihoVnEyY1FGb1g4aEU/view?usp=sharing

Thank you in advance.

_{Tuesday, July 26, 2016 2:12 AM}

Something going on with the system disk (HP disk). IOs are stuck on the disk for 30+, and this disk is being reset. Most likely disk reset is stuck. Try to install latest firmaware and drivers from HP. If issue is not resolved then contact HP support.

Thanks,

Vladimir.

ffffe000b42c3770 [1,2] HP LOGICAL VOLUME Paging Disk

Storage class device ffffe000b42c3770 with extension at ffffe000b42c38c0

Classpnp Internal Information at ffffe000b41f1010

Transfer Packet Engine:

     Packet          Status DL Irp          Opcode Sector/ListId   UL Irp

    ffffe000b8eddb60 Queued ffffe000b8edd840   00   ffffe000b8953640 \Windows\System32\winevt\Logs\Microsoft-Windows-MsLbfoProvider%4Operational.evtx
    ffffe000b8eacd40 Queued ffffe000b8eaca20   00   ffffe8007a1b4010 \Windows\Cluster\clussvc.exe
    ffffe8007849d4c0 Queued ffffe8007a32b010   00   ffffe000b8e5bcf0 \ProgramData\Microsoft\Windows\WER\ReportQueue\NonCritical_7.9.9600.18039_40833763cfe4a66227486a838a61544353f89a8_00000000_e3915ddc\Report.wer
    ffffe8007a199e80 Queued ffffe8007a32e4e0   00   ffffe000b89b68d0 \Windows\System32\winevt\Logs\Microsoft-Windows-FailoverClustering%4Diagnostic.evtx
    ffffe8007bc4c670 Queued ffffe8007a2f5350   00   ffffe000ba9f2010 \Windows\System32\wfp\wfpdiag.etl
    ffffe8007a022960 Queued ffffe8007a3b89d0   00   ffffe8007ceb54e0 \Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\edb.log
    ffffe8007a1cc840 Queued ffffe8007a2f5cf0   00   ffffe80079c1d4e0 \Windows\System32\winevt\Logs\Microsoft-Windows-WMI-Activity%4Operational.evtx
    ffffe8007a1b1e80 Queued ffffe8007a1b1b60   00   ffffe8007cf05340 \LogFile
    ffffe8007a1b19d0 Queued ffffe8007a1b16b0   00   ffffe000ba26a820 \Windows\System32\winevt\Logs\Microsoft-Windows-FailoverClustering%4Diagnostic.evtx
    ffffe80079014e80 Queued ffffe80078cc6010   00   ffffe000bad19cf0 \Windows\System32\wbem\Repository\OBJECTS.DATA

Pending Idle Requests: 0x0

Failed Requests:

           Srb    Scsi
    Opcode Status Status Sense Code Sector/ListId   Time Stamp

      1a     04     02    05 24 00                  07:37:37.124

Classpnp External Information at ffffe000b42c38c0

HP LOGICAL VOLUME 6.64 5001438018152350

    Minidriver information at ffffe000b42c3d80
    Attached device object at ffffe000b44477c0
    Physical device object at ffffe000b44477c0

Media Geometry:

        Bytes in a Sector = 512
        Sectors per Track = 32
        Tracks / Cylinder = 255
        Media Length      = 299966445568 bytes = ~279 GB

-- dt classpnp!_FUNCTIONAL_DEVICE_EXTENSION ffffe000b42c38c0 --

8: kd> !irp ffffe000b8953640
Irp is active with 8 stacks 4 is current (= 0xffffe000b89537e8)
Mdl=ffffd000524757e0: No System Buffer: Thread ffffe000ba251880: Irp stack trace.
cmd flg cl Device File Completion-Context
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000

   Args: 00000000 00000000 00000000 00000000
>[IRP_MJ_WRITE(4), N/A(34)]
           10 e0 ffffe000b42c3770 00000000 fffff801659e7360-00000000 Success Error Cancel
        \Driver\disk partmgr!PmIoCompletion
   Args: 0000a000 00000000 4bf55000 00000000
[IRP_MJ_WRITE(4), N/A(3)]
           10 e0 ffffe000b42c5a40 00000000 fffff801656011b0-ffffe000b41f6a30 Success Error Cancel
        \Driver\partmgr volmgr!VmpReadWriteCompletionRoutine
   Args: 51af0eed0a 00000000 4bf55000 00000000
[IRP_MJ_WRITE(4), N/A(0)]
            0 e0 ffffe000b41f68e0 00000000 fffff8016639c410-ffffe000b4600180 Success Error Cancel
        \Driver\volmgr volsnap!VspRefCountCompletionRoutine
   Args: 0000a000 00000000 51af0eed08 00000000
[IRP_MJ_WRITE(4), N/A(0)]
            0 e1 ffffe000b4600030 00000000 fffff80165c34a14-ffffd000524751c0 Success Error Cancel pending
        \Driver\volsnap Ntfs
   Args: 0000a000 00000000 36055000 00000000
[IRP_MJ_WRITE(4), N/A(0)]
            0 0 ffffe000b3864030 ffffe000b49282c0 00000000-00000000
        \FileSystem\Ntfs
   Args: 0000a000 00000000 000c1000 00000000

THREAD ffffe000ba251880 Cid 0004.2984 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
    ffffd000524751c8 NotificationEvent
IRP List:
    ffffe000b8953640: (0006,0310) Flags: 00060043 Mdl: ffffd000524757e0
Not impersonating
DeviceMap                 ffffc00157a0c320
Owning Process            ffffe000b3816480       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      9579627        Ticks: 133171 (0:00:34:40.796)
Context Switch Count      64043          IdealProcessor: 5
UserTime                  00:00:00.000
KernelTime                00:00:01.359
Win32 Start Address nt!ExpWorkerThread (0xfffff801fe4ce390)
Stack Init ffffd00052475dd0 Current ffffd00052474a70
Base ffffd00052476000 Limit ffffd00052470000 Call 0
Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
ffffd000`52474ab0 fffff801`fe4d24ee nt!KiSwapContext+0x76
ffffd000`52474bf0 fffff801`fe4d1f69 nt!KiSwapThread+0x14e
ffffd000`52474c90 fffff801`fe487563 nt!KiCommitThreadWait+0x129
ffffd000`52474d10 fffff801`65c2e422 nt!KeWaitForSingleObject+0x373
(Inline Function) ` Ntfs!NtfsWaitOnIo+0x1e (Inline Function @ fffff801`65c2e422)
ffffd000`52474da0 fffff801`65c30734 Ntfs!NtfsNonCachedIo+0x412
ffffd000`52475010 fffff801`65c33b63 Ntfs!NtfsCommonWrite+0xd54
ffffd000`52475250 fffff801`653a2b1e Ntfs!NtfsFsdWrite+0x203
ffffd000`52475520 fffff801`653a10c2 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x2ce
ffffd000`524755c0 fffff801`fe4a2d80 fltmgr!FltpDispatch+0xb2
ffffd000`52475620 fffff801`fe4a2baa nt!IoSynchronousPageWrite+0x138
ffffd000`52475660 fffff801`fe51b2cd nt!MiIssueSynchronousFlush+0x66
ffffd000`524756d0 fffff801`fe4a25e5 nt!MiFlushSectionInternal+0x775
ffffd000`524758e0 fffff801`fe45f0ab nt!MmFlushSection+0xa9
ffffd000`524759a0 fffff801`fe45f7cb nt!CcFlushCachePriv+0x493
(Inline Function) ` nt!CcFlushCache+0x1d (Inline Function @ fffff801`fe45f7cb)
ffffd000`52475ab0 fffff801`fe4a63f5 nt!CcWriteBehindInternal+0x17b
ffffd000`52475b40 fffff801`fe4a67ad nt!CcWriteBehind+0x95
ffffd000`52475be0 fffff801`fe4cea2f nt!CcWorkerThread+0x22d
ffffd000`52475c90 fffff801`fe514c10 nt!ExpWorkerThread+0x69f
ffffd000`52475d40 fffff801`fe56e8c6 nt!PspSystemThreadStartup+0x58
ffffd000`52475da0 00000000`00000000 nt!KxStartSystemThread+0x16

   DO ffffe000b44477c0   Ext ffffe000b4447910   Adapter ffffe000b42241a0   Working
   Vendor: HP         Product: LOGICAL VOLUME     SCSI ID: (0, 4, 0)
   Claimed Enumerated
   SlowLock Free   RemLock 13   PageCount 1
   QueueTagList: ffffe000b44479d0      Outstanding: Head ffffe000b4438040 Tail ffffe000b4448040 Timeout -2 (Timed Out)
   DeviceQueue ffffe000b4447a00   Depth: 24   Status: Not Frozen   PauseCount: 1   BusyCount: 0
   IO Gateway: Busy Count 0   Pause Count 0
   Requests: Outstanding 6   Device 4   ByPass 0

[Device-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ffffe8007a2f5350 [SCSI]     ffffe80078803820    n/a         SCSI/WRITE (10)   ffffe000b92e3f40    n/a             60s
ffffe000b8eaca20 [SCSI]     ffffe000b9278e80    n/a         SCSI/READ (10)    ffffe000b6d06b30    n/a             60s
ffffe8007a32b010 [SCSI]     ffffe8007935e230    n/a         SCSI/READ (10)    ffffe000b8cb7010    n/a             60s
ffffe8007a32e4e0 [SCSI]     ffffe800787f5710    n/a         SCSI/WRITE (10)   ffffe8007a152f48    n/a             60s

[Bypass-Queued Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

[Outstanding Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ffffe8007a3b89d0 [SCSI]     ffffe80077e65a10 ffffe000b4438010 SCSI/WRITE (10)   ffffe000b8f1ddf0 ffffe000b44380e0 60s
ffffe80078cc6010 [SCSI]     ffffe80077f01990 ffffe000b4422010 SCSI/WRITE (10)   ffffd000548f02b0 ffffe000b44220e0 60s
ffffe8007a2f5cf0 [SCSI]     ffffe800788c17b0 ffffe000b443c010 SCSI/WRITE (10)   ffffd000212357e0 ffffe000b443c0e0 60s
ffffe8007a1b1b60 [SCSI]     ffffe80077312090 ffffe000b442c010 SCSI/WRITE (10)   ffffe000b4253720 ffffe000b442c0e0 60s
ffffe8007a1b16b0 [SCSI]     ffffe80078823d10 ffffe000b441a010 SCSI/READ (10)    ffffe8007874e420 ffffe000b441a0e0 60s
ffffe000b8edd840 [SCSI]     ffffe000b4714b40 ffffe000b4432010 SCSI/WRITE (10)   ffffd000524757e0 ffffe000b44320e0 60s
ffffe800773256f0 [SCSI]     ffffe000b8c7c770 ffffe000b4448010 RESET LUN         0000000000000000 0000000000000000 0s

[Completed Requests]

IRP SRB Type SRB XRB Command MDL SGList Timeout

ERROR: 7 counted requests > 6 outstanding requests

Regards, Vladimir Petter, Microsoft Corporation

_{Tuesday, July 26, 2016 6:21 AM}

Hi Vladimir,

Thank you so much for your time. You are machine.

I`ll check with HP, because we are already with the latest SPP from HP.

_{Friday, November 11, 2016 5:19 PM}

Hi Vladimir,

I've got the same problem on my Windows 2012R2 HyperV cluster.

I checked the memory dump but could not identify the real problem. Here are the dump information. Could you help me to see what's going wrong ?

10: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Debugging Details:

PROCESS_OBJECT: ffffe0002424b900

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: 0x9E

PROCESS_NAME: System

CURRENT_IRQL: 2

ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre

LAST_CONTROL_TRANSFER: from fffff800202c2468 to fffff802f9f519a0

STACK_TEXT:
ffffd000`cd2bf968 fffff800`202c2468 : 00000000`0000009e ffffe000`2424b900 00000000`00000690 00000000`00000005 : nt!KeBugCheckEx
ffffd000`cd2bf970 fffff800`202c20f2 : 00000000`00000000 00000000`00000001 ffffd000`cd2aa180 00000000`00000000 : netft!NetftProcessWatchdogEvent+0xe4
ffffd000`cd2bf9b0 fffff802`f9ecb3a8 : ffffd000`cd2bfb20 00000000`00000000 ffffe000`240d49f0 fffff800`2013acd5 : netft!NetftWatchdogTimerDpc+0x36
ffffd000`cd2bf9e0 fffff802`f9f554ea : ffffd000`cd2aa180 ffffd000`cd2aa180 ffffd000`cd2b6bc0 ffffe000`2c08e040 : nt!KiRetireDpcList+0x4f8
ffffd000`cd2bfc60 00000000`00000000 : ffffd000`cd2c0000 ffffd000`cd2ba000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x5a

STACK_COMMAND: kb

FOLLOWUP_IP:
netft!NetftProcessWatchdogEvent+e4
fffff800`202c2468 cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: netft!NetftProcessWatchdogEvent+e4

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: netft

IMAGE_NAME: netft.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 5434e860

BUCKET_ID_FUNC_OFFSET: e4

FAILURE_BUCKET_ID: 0x9E_netft!NetftProcessWatchdogEvent

BUCKET_ID: 0x9E_netft!NetftProcessWatchdogEvent

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:0x9e_netft!netftprocesswatchdogevent

FAILURE_ID_HASH: {fc992d70-4714-ccd6-c6b5-601c2a57cb6c}

Followup: MachineOwner

10: kd> !process ffffe0002424b900
PROCESS ffffe0002424b900
    SessionId: 0 Cid: 153c    Peb: 7ff6f6277000 ParentCid: 09cc
    DirBase: 13da7a000 ObjectTable: ffffc0016bb10040 HandleCount: <Data Not Accessible>
    Image: rhs.exe
    VadRoot ffffe00022e05380 Vads 87 Clone 0 Private 592. Modified 12. Locked 0.
    DeviceMap ffffc0015d00c3a0
    Token                             ffffc00165ce9060
    ElapsedTime                       02:00:11.093
    UserTime                          00:00:00.031
    KernelTime                        00:00:00.031
    QuotaPoolUsage[PagedPool]         76408
    QuotaPoolUsage[NonPagedPool]      11248
    Working Set Sizes (now,min,max) (1952, 50, 345) (7808KB, 200KB, 1380KB)
    PeakWorkingSetSize                1891
    VirtualSize                       2097191 Mb
    PeakVirtualSize                   2097193 Mb
    PageFaultCount                    2285
    MemoryPriority                    BACKGROUND
    BasePriority                      13
    CommitCharge                      639

        THREAD ffffe0002424e080 Cid 153c.1654 Teb: 00007ff6f627e000 Win32Thread: fffff901406ddb60 WAIT: (UserRequest) UserMode Non-Alertable
            ffffe8006ee78960 NotificationEvent
            ffffe0002349f080 ProcessObject
            ffffe000031f11b0 NotificationEvent
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      329056         Ticks: 144294 (0:00:37:34.593)
        Context Switch Count      60             IdealProcessor: 23
        UserTime                  00:00:00.000
        KernelTime                00:00:00.046
        Win32 Start Address 0x00007ff6f6ce1c80
        Stack Init ffffd000d503dc90 Current ffffd000d503cfc0
        Base ffffd000d503e000 Limit ffffd000d5038000 Call 0
        Priority 14 BasePriority 13 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        ffffd000`d503d000 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d503d140 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d503d1e0 fffff802`f9e31f43 nt!KiCommitThreadWait+0x129
        ffffd000`d503d260 fffff802`fa1c9929 nt!KeWaitForMultipleObjects+0x403
        ffffd000`d503d320 fffff802`fa1c967a nt!ObWaitForMultipleObjects+0x289
        ffffd000`d503d830 fffff802`f9f5d1b3 nt!NtWaitForMultipleObjects+0xd6
        ffffd000`d503da90 00007ffb`b3cb11fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d503db00)
        0000006d`24a5edc8 00000000`00000000 0x00007ffb`b3cb11fa

        THREAD ffffe8006ee7f800 Cid 153c.1650 Teb: 00007ff6f627c000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
            ffffe8006ee7f750 NotificationEvent
            ffffe8006ee7f6d0 SynchronizationEvent
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      338992         Ticks: 134358 (0:00:34:59.343)
        Context Switch Count      181            IdealProcessor: 25
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ff6f6cdf550
        Stack Init ffffd000d5acec90 Current ffffd000d5acdfc0
        Base ffffd000d5acf000 Limit ffffd000d5ac9000 Call 0
        Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        ffffd000`d5ace000 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d5ace140 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d5ace1e0 fffff802`f9e31f43 nt!KiCommitThreadWait+0x129
        ffffd000`d5ace260 fffff802`fa1c9929 nt!KeWaitForMultipleObjects+0x403
        ffffd000`d5ace320 fffff802`fa1c967a nt!ObWaitForMultipleObjects+0x289
        ffffd000`d5ace830 fffff802`f9f5d1b3 nt!NtWaitForMultipleObjects+0xd6
        ffffd000`d5acea90 00007ffb`b3cb11fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5aceb00)
        0000006d`252bf678 00000000`00000000 0x00007ffb`b3cb11fa

        THREAD ffffe8006ee84080 Cid 153c.16d8 Teb: 00007ff6f627a000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
            ffffe8006dd2f060 SynchronizationEvent
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      329056         Ticks: 144294 (0:00:37:34.593)
        Context Switch Count      16             IdealProcessor: 27
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ffbb3c533a0
        Stack Init ffffd000d5506c90 Current ffffd000d5506760
        Base ffffd000d5507000 Limit ffffd000d5501000 Call 0
        Priority 15 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        ffffd000`d55067a0 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d55068e0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d5506980 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
        ffffd000`d5506a00 fffff802`fa1eeec2 nt!KeWaitForSingleObject+0x2c0
        ffffd000`d5506a90 fffff802`f9f5d1b3 nt!NtWaitForSingleObject+0xb2
        ffffd000`d5506b00 00007ffb`b3cb0c8a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5506b00)
        0000006d`2533f6a8 00000000`00000000 0x00007ffb`b3cb0c8a

        THREAD ffffe8006ee86080 Cid 153c.16c8 Teb: 00007ff6f6275000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
            ffffe8006ee7f590 SynchronizationEvent
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      329056         Ticks: 144294 (0:00:37:34.593)
        Context Switch Count      15             IdealProcessor: 31
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ffbaa0c1dc0
        Stack Init ffffd000d54fcc90 Current ffffd000d54fc760
        Base ffffd000d54fd000 Limit ffffd000d54f7000 Call 0
        Priority 15 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        ffffd000`d54fc7a0 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d54fc8e0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d54fc980 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
        ffffd000`d54fca00 fffff802`fa1eeec2 nt!KeWaitForSingleObject+0x2c0
        ffffd000`d54fca90 fffff802`f9f5d1b3 nt!NtWaitForSingleObject+0xb2
        ffffd000`d54fcb00 00007ffb`b3cb0c8a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d54fcb00)
        0000006d`2549fe58 00000000`00000000 0x00007ffb`b3cb0c8a

        THREAD ffffe8006ee87080 Cid 153c.1708 Teb: 00007ff6f6273000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
            ffffe0002bd3b360 SynchronizationEvent
        IRP List:
            ffffe80072954cf0: (0006,0310) Flags: 00000884 Mdl: 00000000
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      365873         Ticks: 107477 (0:00:27:59.328)
        Context Switch Count      132            IdealProcessor: 33
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ff6f6cbe880
        Stack Init ffffd000d563ac90 Current ffffd000d5638e60
        Base ffffd000d563b000 Limit ffffd000d5635000 Call 0
        Priority 14 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
*** ERROR: Module load completed but symbols could not be loaded for EX64.SYS
*** ERROR: Module load completed but symbols could not be loaded for SRTSP64.SYS

        Child-SP          RetAddr           Call Site
        ffffd000`d5638ea0 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d5638fe0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d5639080 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
        ffffd000`d5639100 fffff802`fa290399 nt!KeWaitForSingleObject+0x2c0
        ffffd000`d5639190 fffff802`fa1e55b7 nt!IopAcquireFileObjectLock+0x85
        ffffd000`d56391e0 fffff802`f9f5d1b3 nt!NtReadFile+0x8d7
        ffffd000`d56392d0 fffff802`f9f55600 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5639340)
        ffffd000`d56394d8 fffff800`1f4bc73c nt!KiServiceLinkage
        ffffd000`d56394e0 fffff800`1f4bc89d EX64+0x273c
        ffffd000`d5639550 fffff800`1f4bdf87 EX64+0x289d
        ffffd000`d56395a0 fffff800`1f4c0087 EX64+0x3f87
        ffffd000`d56395f0 fffff800`1f4c0468 EX64+0x6087
        ffffd000`d56396b0 fffff800`1f4c0350 EX64+0x6468
        ffffd000`d5639840 fffff800`1f4c0b35 EX64+0x6350
        ffffd000`d5639910 fffff800`1f4c0e0d EX64+0x6b35
        ffffd000`d5639a50 fffff800`1f4c2a4c EX64+0x6e0d
        ffffd000`d5639af0 fffff800`1f4c888e EX64+0x8a4c
        ffffd000`d5639cb0 fffff800`1f4c971b EX64+0xe88e
        ffffd000`d5639e00 fffff800`1f4c991e EX64+0xf71b
        ffffd000`d5639ee0 fffff800`1ed5240f EX64+0xf91e
        ffffd000`d5639f40 fffff800`1ed525c7 SRTSP64+0x7940f
        ffffd000`d5639fa0 fffff800`1eda031d SRTSP64+0x795c7
        ffffd000`d563a000 fffff800`1ed9e198 SRTSP64+0xc731d
        ffffd000`d563a070 fffff800`1ed9debc SRTSP64+0xc5198
        ffffd000`d563a0f0 fffff800`1ed08718 SRTSP64+0xc4ebc
        ffffd000`d563a150 fffff800`1ed08a40 SRTSP64+0x2f718
        ffffd000`d563a180 fffff800`1ed406b1 SRTSP64+0x2fa40
        ffffd000`d563a1b0 fffff800`1e6fc9d7 SRTSP64+0x676b1
        ffffd000`d563a200 fffff800`1e6fd14d fltmgr!FltpPerformPostCallbacks+0x2d7
        ffffd000`d563a2d0 fffff800`1e6fbbc1 fltmgr!FltpPassThroughCompletionWorker+0x7d
        ffffd000`d563a340 fffff800`1e724349 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x371
        ffffd000`d563a3e0 fffff802`fa240b91 fltmgr!FltpCreate+0x339
        ffffd000`d563a490 fffff802`fa1ecc39 nt!IopParseDevice+0x6c1
        ffffd000`d563a6b0 fffff802`fa1eaa63 nt!ObpLookupObjectName+0x6b9
        ffffd000`d563a830 fffff802`fa2637ab nt!ObOpenObjectByName+0x1e3
        ffffd000`d563a960 fffff802`fa263438 nt!IopCreateFile+0x36b
        ffffd000`d563aa00 fffff802`f9f5d1b3 nt!NtCreateFile+0x78
        ffffd000`d563aa90 00007ffb`b3cb119a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d563ab00)
        0000006d`2561de18 00000000`00000000 0x00007ffb`b3cb119a

        THREAD ffffe8006ee8b080 Cid 153c.1774 Teb: 00007ff6f614a000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
            ffffe8006d89d710 NotificationEvent
            ffffe8006ec92d60 NotificationEvent
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      436170         Ticks: 37180 (0:00:09:40.937)
        Context Switch Count      421            IdealProcessor: 39
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ffbb30f0b70
        Stack Init ffffd000d5adfc90 Current ffffd000d5adefc0
        Base ffffd000d5ae0000 Limit ffffd000d5ada000 Call 0
        Priority 14 BasePriority 13 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        ffffd000`d5adf000 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d5adf140 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d5adf1e0 fffff802`f9e31f43 nt!KiCommitThreadWait+0x129
        ffffd000`d5adf260 fffff802`fa1c9929 nt!KeWaitForMultipleObjects+0x403
        ffffd000`d5adf320 fffff802`fa1c967a nt!ObWaitForMultipleObjects+0x289
        ffffd000`d5adf830 fffff802`f9f5d1b3 nt!NtWaitForMultipleObjects+0xd6
        ffffd000`d5adfa90 00007ffb`b3cb11fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d5adfb00)
        0000006d`25a7f8f8 00000000`00000000 0x00007ffb`b3cb11fa

        THREAD ffffe000263ee880 Cid 153c.151c Teb: 00007ff6f614e000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
            ffffe000263eeeb0 Semaphore Limit 0x1
        Waiting for reply to ALPC Message ffffc0015e5b9d00 : queued at port ffffe00023a68e40 : owned by process ffffe00023469900
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      338992         Ticks: 134358 (0:00:34:59.343)
        Context Switch Count      545            IdealProcessor: 31
        UserTime                  00:00:00.031
        KernelTime                00:00:00.031
        Win32 Start Address 0x00007ffbb3c533a0
        Stack Init ffffd000d7fe6c90 Current ffffd000d7fe6400
        Base ffffd000d7fe7000 Limit ffffd000d7fe1000 Call 0
        Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        ffffd000`d7fe6440 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`d7fe6580 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`d7fe6620 fffff802`f9e5a8d0 nt!KiCommitThreadWait+0x129
        ffffd000`d7fe66a0 fffff802`f9e3d9c3 nt!KeWaitForSingleObject+0x2c0
        ffffd000`d7fe6730 fffff802`fa1cb3c9 nt!AlpcpSignalAndWait+0x443
        ffffd000`d7fe67e0 fffff802`fa1ca8c7 nt!AlpcpReceiveSynchronousReply+0x55
        ffffd000`d7fe6840 fffff802`fa1ec520 nt!AlpcpProcessSynchronousRequest+0x377
        ffffd000`d7fe69c0 fffff802`f9f5d1b3 nt!NtAlpcSendWaitReceivePort+0x270
        ffffd000`d7fe6a90 00007ffb`b3cb14ca nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`d7fe6b00)
        0000006d`2569e138 00000000`00000000 0x00007ffb`b3cb14ca

        THREAD ffffe8006ec4e880 Cid 153c.1784 Teb: 00007ff6f6278000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Alertable
            ffffe8006d89ec80 QueueObject
        Not impersonating
        DeviceMap                 ffffc0015d00c3a0
        Owning Process            ffffe0002424b900       Image:         rhs.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      472644         Ticks: 706 (0:00:00:11.031)
        Context Switch Count      329            IdealProcessor: 33
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00007ffbb3c533a0
        Stack Init ffffd000222f9c90 Current ffffd000222f9420
        Base ffffd000222fa000 Limit ffffd000222f4000 Call 0
        Priority 13 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        ffffd000`222f9460 fffff802`f9e32d7e nt!KiSwapContext+0x76
        ffffd000`222f95a0 fffff802`f9e327f9 nt!KiSwapThread+0x14e
        ffffd000`222f9640 fffff802`f9e314cd nt!KiCommitThreadWait+0x129
        ffffd000`222f96c0 fffff802`f9e3104a nt!KeRemoveQueueEx+0x26d
        ffffd000`222f9750 fffff802`f9e306fb nt!IoRemoveIoCompletion+0x8a
        ffffd000`222f9870 fffff802`f9f5d1b3 nt!NtWaitForWorkViaWorkerFactory+0x30b
        ffffd000`222f9a90 00007ffb`b3cb273a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`222f9b00)
        0000006d`2541f3f8 00000000`00000000 0x00007ffb`b3cb273a

_{Saturday, November 12, 2016 12:03 AM}

Please open a case with Microsoft support, and they will be able to analyze the memory dump for you and root cause what went wrong.

Thanks!
Elden

_{Friday, May 19, 2017 8:37 PM}

Hello Vladimir, Sorry for replying to an old thread. But we just had a cluster server stop error with a 9E with the watch dog event. I've run the memory dump through the debugger and I can't find anything obvious about why it reached the timeout / bug-check threshold. I was wondering if someone could look at it with more knowledge in the debugger than I have?

Here is the 7-zip'd file.

https://1drv.ms/u/s!Aj_NX6R0CE7qeXLZMJXSCGH-Lqg

Thanks,

Dean

_{Wednesday, September 27, 2017 2:53 PM}

Hello Vladimir,

Would you be helping me in analyzing a memory dump with bugcheck 19, {0x22, ffffe00017fb3000, 0, 0}?

server is a VM running windows 2012 on Esxi 6.0, it is rebooting after every 2-3 days with above mentioned bugcheck.

Thanks & Regards,

Suchit

_{Wednesday, February 14, 2018 8:42 AM}

HI Vladimir,

Can you please analyze the memory dump file generated.

location: https://ibm.box.com/s/dnvntrh4dg2kd8gexqhbo9yiizg6pqnn

_{Sunday, March 25, 2018 1:20 AM}

You can analyze the memory dump using WinDbg tool here: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools

_{Thursday, October 4, 2018 8:12 PM}

Hi, I am having similar issue. Can I share the dump file?

_{Monday, June 10, 2019 3:11 PM}

HI,Vladimir：

What command did you use for this dump file?I also suspect that it is a vmware disk problem.Thank you！

_{Monday, June 10, 2019 3:23 PM}

Hello Hardy.

If you are looking at bugcheck 9e then first parameter of bugcheck is address of the process this watchdog was for. Start by running "!process <address> 1f". Stepping back a bit, always start with

.symfix

.reload

and then start trying other commands. This will fix symbols for you. Use lml to see what symbols are loaded.

Not going back. Once you dump that process look at the threads, and see if they are blocked on something in kernel. For instance if they are waiting on ERESOURCE then try to get address of ERESOURCE and use !locks <address> to see who owns it. Keep following wait chain.

If you get stuck then you can try your like by checking what is going on in storage using

!storagekd.unit

!storagekd.classext

I am inside Microsoft corpnet so I have access to private symbols, while you might be limited to public symbols only. As such I am not sure if storagekd will work for you. I am also not sure if it ships with public version of windbg. If you are blocked and can find a way to give me access to the dump I can take a look.

Regards,

Vladimir.

Regards, Vladimir Petter, Microsoft Corporation

_{Tuesday, June 11, 2019 4:42 PM}

HI，Vladimir：

Thanks for your help, I have tried many times using windbg to prompt DBGHELP: Timeout to store: c:\symbols*http://msdl.microsoft.com/download/symbols. Please help you analyze the dmp file, thank you!

https://1drv.ms/u/s!ApZTZXkGWOEB2zmFbxqB072hED7A?e=dut0Bx

_{Tuesday, June 11, 2019 6:43 PM}

What you have is a kernel minidump. Minidump would not be helpful for this class of issues because it includes only stack of the thread that triggered bugcheck, and we need all kernel threads and non-paged pool so we can follow threads wait chain. You need to change settings to collect full kernel dump on the next repro https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/enabling-a-kernel-mode-dump-file

Regards,

Vladimir Petter.

Regards, Vladimir Petter, Microsoft Corporation

_{Wednesday, July 24, 2019 5:25 PM}

HI,Vladimir：

I have a system failure, I hope you can help analyze the problem! I hope you can open the link: https://pan.baidu.com/s/1zz6Xkj_qzlwiK2QrbkO2VQ Extraction code: if3l Thank you！

_{Thursday, July 25, 2019 2:48 AM}

HI,Vladimir：

Not sure if you can open the Baidu network disk, I will change it to onedrive: https://1drv.ms/u/s!ApZTZXkGWOEB3DKLkm7MwHd-i_Wu .Thank you!

_{Tuesday, October 1, 2019 8:08 AM}

HI,Vladimir

We have system failure at the same time at two Hyperv Cluster Windows 2012R2. Some hosts in the cluster restarted with bugcheck code 0x0000009E. In memory dump I can onli get this. All host are fw and software up tu date.

I put all dumps on share (mini and some full memory dump)

https://izum-my.sharepoint.com/:f:/g/personal/gorazd_taciga_izum_si/Eg3EH-lS-o9LntvTfxvfLqQBSvOCnEk6hs1FhXmv2lE6Ew?e=a1ii15

I hope you can help analyze the problem.

4: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

USER_MODE_HEALTH_MONITOR (9e)
One or more critical user mode components failed to satisfy a health check.
Hardware mechanisms such as watchdog timers can detect that basic kernel
services are not executing. However, resource starvation issues, including
memory leaks, lock contention, and scheduling priority misconfiguration,
may block critical user mode components without blocking DPCs or
draining the nonpaged pool.
Kernel components can extend watchdog timer functionality to user mode
by periodically monitoring critical applications. This bugcheck indicates
that a user mode health check failed in a manner such that graceful
shutdown is unlikely to succeed. It restores critical services by
rebooting and/or allowing application failover to other servers.
Arguments:
Arg1: ffffe000234e7080, Process that failed to satisfy a health check within the
    configured timeout
Arg2: 000000000000003c, Health monitoring timeout (seconds)
Arg3: 000000000000000a, WatchdogSourceClussvcIsAlive
    Cluster service sends heartbeat to netft every 500 millseconds.
    By default netft expects at least 1 heartbeat per second.
    If this watchdog was triggered that means clussvc is o not getting
    CPU to send heartbers.
Arg4: 0000000000000000

Regards

Gorazd

_{Tuesday, October 1, 2019 6:06 PM}

You have DPC storm on CPU0 from network activity. DPC storm is starving cluster service timer that is triggering sending heart to netft.sys. Since netft.sys has not recieved HB in 60 seconds it bugchecked machine.

To solve this issue you need to setup RSS/vRSS/VMQ to make sure network traffic is equally spread across all CPUs instead of all happening on CPU0.

In Windows Server 2019 we made number of improvements to reduce number of false positives watchdog timeouts.

- We increased timeout to 135 to be larger than commulative DPC timeout 120 seocnds so you would get a bugcheck from commulative DPC watchdog before the cluster watchdog.

- We changed default watchdog action from bugchck to take a LiveDump, terminate process, and extent watchdog to 20 minutes. Only if process termination does not complete in 20 minutes we would bugcheck machine.

CPU#0

Current DPC: NDIS!ndisInterruptDpc (Normal DPC)
Debugger Saved IRQL: 0
Cumulative DPC Time Limit: 120.000 seconds
Current Cumulative DPC Time: 60.156 seconds
Single DPC Time Limit: 20.000 seconds
Current Single DPC Time: 0.000 seconds

DPC History:

     Deferred Routine                                   Start Tick      End Tick        Duration (in seconds)

[00] NDIS!ndisInterruptDpc                              0x3EDB          0x3EDB          -
[01] NDIS!ndisInterruptDpc                              0x3EDB          0x3EDB          -
[02] NDIS!ndisInterruptDpc                              0x3EDB          (DPC Active)    
[03] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[04] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[05] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[06] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[07] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[08] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[09] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[10] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[11] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[12] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[13] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[14] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDA          -
[15] NDIS!ndisInterruptDpc                              0x3EDA          0x3EDB          0.0156

You can also see number of threads are sitting on CPU0 in a ready state for almost 60 seconds

Processor 0: Ready Threads at priority 30
    THREAD ffffe800edf29080  Cid 0c58.1cfc  Teb: 00007ff692a72000 Win32Thread: 0000000000000000 READY on processor 0
    IRP List:
        ffffe0009b376ee0: (0006,0118) Flags: 00060030  Mdl: 00000000
    Not impersonating
    DeviceMap                 ffffc0000260c370
    Owning Process            ffffe0008fc27080       Image:         vmms.exe
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      12261          Ticks: 3830 (0:00:00:59.843)
    Context Switch Count      931            IdealProcessor: 0             
    UserTime                  00:00:00.015
    KernelTime                00:00:00.000
    Win32 Start Address 0x00007ffeeaaf8a10
    Stack Init ffffd0002846dc90 Current ffffd0002846cc80
    Base ffffd0002846e000 Limit ffffd00028468000 Call 0000000000000000
    Priority 30 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`2846ccc0 fffff801`d123519b nt!KiSwapContext+0x76
    (Inline Function) ` nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
    ffffd000`2846ce00 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
    ffffd000`2846ce40 fffff801`d1234c8d nt!KeSetSystemGroupAffinityThread+0xfb
    ffffd000`2846ce90 fffff801`d1234a3d nt!KeGenericProcessorCallback+0xdd
    ffffd000`2846d000 fffff801`166326cc nt!KeFlushQueuedDpcsEx+0x11d
    ffffd000`2846d1a0 fffff801`16632627 vmswitch!VmsOmpPortUninitializeFlowTable+0x1c
    ffffd000`2846d1d0 fffff801`1662b52b vmswitch!VmsOmpPortDeleted+0xcb
    ffffd000`2846d220 fffff801`1661ef22 vmswitch!VmsOmObjectRefCountDecrement+0x2f
    ffffd000`2846d250 fffff801`1661d32f vmswitch!VmsOmPortDelete+0x3fa
    (Inline Function) ` vmswitch!VmsCdpPortDeleteByName+0x24 (Inline Function @ fffff801`1661d32f)
    ffffd000`2846d330 fffff801`1662bcca vmswitch!VmsCdpPortDelete+0x11f
    ffffd000`2846d5c0 fffff801`1533018c vmswitch!VmsCdpDeviceControl+0x416
    ffffd000`2846d5f0 fffff801`15331176 NDIS!ndisDummyIrpHandler+0x88
    ffffd000`2846d620 fffff801`d168e0af NDIS!ndisDeviceControlIrpHandler+0x666
    (Inline Function) ` nt!IoCallDriverWithTracing+0x7e (Inline Function @ fffff801`d168e0af)
    (Inline Function) ` nt!IopCallDriverReference+0x187 (Inline Function @ fffff801`d168e0af)
    ffffd000`2846d810 fffff801`d168f018 nt!IopSynchronousServiceTail+0x32b
    ffffd000`2846d8e0 fffff801`d165e8a6 nt!IopXxxControlFile+0xdb8
    ffffd000`2846da20 fffff801`d13592e3 nt!NtDeviceIoControlFile+0x56
    ffffd000`2846da90 00007ffe`eab507ca nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`2846db00)
    00000082`eeefe958 00000000`00000000 0x00007ffe`eab507ca

    THREAD ffffe800eded4040  Cid 0004.17e8  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
    IRP List:
        ffffe800f2b85ca0: (0006,0358) Flags: 00000404  Mdl: 00000000
    Not impersonating
    DeviceMap                 ffffc0000260c370
    Owning Process            ffffe00084020480       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      13221          Ticks: 2870 (0:00:00:44.843)
    Context Switch Count      426            IdealProcessor: 0             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.000
    Win32 Start Address nt!ExpWorkerThread (0xfffff801d1269430)
    Stack Init ffffd00026d3fc90 Current ffffd00026d3f3c0
    Base ffffd00026d40000 Limit ffffd00026d3a000 Call 0000000000000000
    Priority 30 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`26d3f400 fffff801`d123519b nt!KiSwapContext+0x76
    (Inline Function) ` nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
    ffffd000`26d3f540 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
    ffffd000`26d3f580 fffff801`d1234c8d nt!KeSetSystemGroupAffinityThread+0xfb
    ffffd000`26d3f5d0 fffff801`d1234a3d nt!KeGenericProcessorCallback+0xdd
    ffffd000`26d3f740 fffff801`d1597e64 nt!KeFlushQueuedDpcsEx+0x11d
    ffffd000`26d3f8e0 fffff801`1620529f nt!MmPageEntireDriver+0xa0
    (Inline Function) ` Msfs!MsDereferenceVcb+0x26 (Inline Function @ fffff801`1620529f)
    (Inline Function) ` Msfs!MsCommonClose+0x5b (Inline Function @ fffff801`1620529f)
    ffffd000`26d3f910 fffff801`15474101 Msfs!MsFsdClose+0x6f
    ffffd000`26d3f950 fffff801`d15cde0c fltmgr!FltpDispatch+0xf1
    ffffd000`26d3f9b0 fffff801`d15c1e5c nt!IopDeleteFile+0x128
    ffffd000`26d3fa30 fffff801`d125c9bf nt!ObpRemoveObjectRoutine+0x64
    (Inline Function) ` nt!ObfDereferenceObjectWithTag+0x80 (Inline Function @ fffff801`d125c9bf)
    ffffd000`26d3fa90 fffff801`1549eb9a nt!ObfDereferenceObject+0x8f
    ffffd000`26d3fad0 fffff801`d1269acf fltmgr!FltpManualDeviceAttachWorker+0xc2
    ffffd000`26d3fb50 fffff801`d12ccdc6 nt!ExpWorkerThread+0x69f
    ffffd000`26d3fc00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
    ffffd000`26d3fc60 00000000`00000000 nt!KxStartSystemThread+0x16

Processor 0: Ready Threads at priority 15
    THREAD ffffe000a3bf9080  Cid 0004.2ffc  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
    Not impersonating
    DeviceMap                 ffffc0000260c370
    Owning Process            ffffe00084020480       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      12940          Ticks: 3151 (0:00:00:49.234)
    Context Switch Count      7              IdealProcessor: 0             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.000
    Win32 Start Address nt!EtwpLogger (0xfffff801d16afc78)
    Stack Init ffffd0002bad9c90 Current ffffd0002bad9780
    Base ffffd0002bada000 Limit ffffd0002bad4000 Call 0000000000000000
    Priority 15 BasePriority 15 PriorityDecrement 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`2bad97c0 fffff801`d123519b nt!KiSwapContext+0x76
    (Inline Function) ` nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
    ffffd000`2bad9900 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
    ffffd000`2bad9940 fffff801`d1234c8d nt!KeSetSystemGroupAffinityThread+0xfb
    ffffd000`2bad9990 fffff801`d1234147 nt!KeGenericProcessorCallback+0xdd
    ffffd000`2bad9b00 fffff801`d169d08e nt!KeGenericCallDpc+0x27
    (Inline Function) ` nt!EtwpSynchronizeWithElevatedIrqlLogging+0xe (Inline Function @ fffff801`d169d08e)
    ffffd000`2bad9b40 fffff801`d16afef5 nt!EtwpFreeLoggerContext+0x8e
    ffffd000`2bad9b80 fffff801`d12ccdc6 nt!EtwpLogger+0x27d
    ffffd000`2bad9c00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
    ffffd000`2bad9c60 00000000`00000000 nt!KxStartSystemThread+0x16

Processor 0: Ready Threads at priority 14
    THREAD ffffe800ee157880  Cid 1878.2c2c  Teb: 00007ff71d04c000 Win32Thread: fffff90140740610 READY on processor 0
    Not impersonating
    DeviceMap                 ffffc000078c9eb0
    Owning Process            ffffe0008f9b8900       Image:         vmwp.exe
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      12271          Ticks: 3820 (0:00:00:59.687)
    Context Switch Count      85             IdealProcessor: 0             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.000
    Win32 Start Address 0x00007ffeeaaf8a10
    Stack Init ffffd0002a173c90 Current ffffd0002a1737c0
    Base ffffd0002a174000 Limit ffffd0002a16e000 Call 0000000000000000
    Priority 14 BasePriority 8 PriorityDecrement 16 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`2a173800 fffff801`d12724e7 nt!KiSwapContext+0x76
    (Inline Function) ` nt!KzCheckForThreadDispatch+0x101 (Inline Function @ fffff801`d12724e7)
    (Inline Function) ` nt!KiCheckForThreadDispatch+0x101 (Inline Function @ fffff801`d12724e7)
    ffffd000`2a173940 fffff801`d1233439 nt!KiProcessDeferredReadyList+0x13b
    ffffd000`2a173980 fffff801`d160dcbb nt!KeSetLegacyAffinityThread+0xd9
    ffffd000`2a1739e0 fffff801`d13592e3 nt!NtSetInformationThread+0x9ab
    ffffd000`2a173b00 00007ffe`eab5082a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`2a173b00)
    00000088`03cff018 00000000`00000000 0x00007ffe`eab5082a

Processor 0: Ready Threads at priority 12
    THREAD ffffe00085b60880  Cid 0004.01c4  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
    Not impersonating
    DeviceMap                 ffffc0000260c370
    Owning Process            ffffe00084020480       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      12261          Ticks: 3830 (0:00:00:59.843)
    Context Switch Count      7542           IdealProcessor: 0             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.062
    Win32 Start Address nt!ExpWorkerThread (0xfffff801d1269430)
    Stack Init ffffd00077b31c90 Current ffffd00077b31700
    Base ffffd00077b32000 Limit ffffd00077b2c000 Call 0000000000000000
    Priority 12 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`77b31740 fffff801`d123519b nt!KiSwapContext+0x76
    (Inline Function) ` nt!KzCheckForThreadDispatch+0x134 (Inline Function @ fffff801`d123519b)
    ffffd000`77b31880 fffff801`d1234e7b nt!KiCheckForThreadDispatch+0x153
    ffffd000`77b318c0 fffff801`158023c2 nt!KeSetSystemGroupAffinityThread+0xfb
    (Inline Function) ` vmbusr!WdmlibKeSetSystemGroupAffinityThread+0x20 (Inline Function @ fffff801`158023c2)
    ffffd000`77b31910 fffff801`15817d24 vmbusr!PncPullMessage+0x4e
    ffffd000`77b31980 fffff801`d126a920 vmbusr!XPartReceiveMessageWorkItem+0x7c
    ffffd000`77b31ae0 fffff801`d1269acf nt!IopProcessWorkItem+0x80
    ffffd000`77b31b50 fffff801`d12ccdc6 nt!ExpWorkerThread+0x69f
    ffffd000`77b31c00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
    ffffd000`77b31c60 00000000`00000000 nt!KxStartSystemThread+0x16

Processor 0: Ready Threads at priority 8
    THREAD ffffe00085e49880  Cid 0004.01f8  Teb: 0000000000000000 Win32Thread: 0000000000000000 READY on processor 0
    Not impersonating
    DeviceMap                 ffffc0000260c370
    Owning Process            ffffe00084020480       Image:         System
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      12261          Ticks: 3830 (0:00:00:59.843)
    Context Switch Count      5              IdealProcessor: 0             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.000
    Win32 Start Address NDIS!ndisReceiveWorkerThread (0xfffff801152aedd0)
    Stack Init ffffd000775c6c90 Current ffffd000775c67d0
    Base ffffd000775c7000 Limit ffffd000775c1000 Call 0000000000000000
    Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           Call Site
    ffffd000`775c6810 fffff801`d13c7417 nt!KiSwapContext+0x76
    ffffd000`775c6950 fffff801`d134c2af nt!KiQuantumEnd+0x52b
    ffffd000`775c69a0 fffff801`d134eb69 nt!KiDispatchInterruptContinue+0x16
    ffffd000`775c69d0 fffff801`d125daa7 nt!KiDpcInterrupt+0x2a9 (TrapFrame @ ffffd000`775c69d0)
    ffffd000`775c6b68 fffff801`152aefc1 nt!KzLowerIrql+0x7
    ffffd000`775c6b70 fffff801`d12ccdc6 NDIS!ndisReceiveWorkerThread+0x1f1
    ffffd000`775c6c00 fffff801`d1350e66 nt!PspSystemThreadStartup+0x18a
    ffffd000`775c6c60 00000000`00000000 nt!KxStartSystemThread+0x16

Regards, Vladimir Petter, Microsoft Corporation

_{Thursday, October 3, 2019 6:04 AM}

Thanks for your help

Regards

Gorazd

_{Monday, February 24, 2020 4:07 PM}

Techei, I am a developer on the clustering team. if the problem persists then you can share dump file with me (c:\windows\memory.dmp). I will take a look what exactly caused the bugcheck. if you do not want to share it with the world then give me your email, I'll reply with my email and you can share this dump with me. If you prefer work through customer support then they also should be able to look at the dump and tell you what netft is not happy about.

Regards, Vladimir Petter, Microsoft Corporation

Are you still able to do this with me sending you a direct link via email? I've got a server that dumps out with the same error fairly consistently and I can get you the dump file.

Thanks!

_{Monday, February 24, 2020 5:17 PM}

Sure, happy to help. You can send me a link to vladp72 [at] yahoo [dot] com.

Thanks,
Vladimir.

Regards, Vladimir Petter, Microsoft Corporation

_{Sunday, July 19, 2020 10:51 PM}

John_Alliance I'm having the same issue with 2012 R2. Could you help me out?

Share via

netft.sys is the cause for the bugchk blue screen on the server Windows 2008 R2 Datacenter

Question

All replies (71)

Additional resources