Troubleshooting driver issues for Windows Server 2019

Question

_{Thursday, May 9, 2019 11:07 PM}

I'm likely out of my league here but I'm guessing this is a good place to start as a lot here have expertise in driver related issues.

I'm trying to figure out why the network driver that we're using is spontaneously stopping in the middle of the night.

Details:
We're using a virtio NetKVM on Windows Server 2019 using bhyve.
The driver works fine and then is disabled and says "No drivers are installed for this device." yet the files are installed and the network works again after rebooting.  It also seems to be in a D3 power state but when it's working it's at D0, wasn't sure if this was just due to the driver being disabled or something else, but so far as I can tell this driver does not support any power save mode.
I cannot find any Event Viewer or otherwise logs indicating why the driver stopped (I might not be looking in the right place) only logs showing the results of the network going away.

Any ideas as to how I might debug this issue or get Windows to report why it stopped this driver or why the driver stopped on its own?

Thank you.

All replies (13)

_{Wednesday, May 22, 2019 10:00 PM ✅Answered}

The problem was solved, as for an answer to my question, logman and traceview are tools that may or may not work depending on the support in the driver and the settings configured.  It turns out I wasn't able to get either to work because the older version we were using didn't yet have support for those tools.  The newer version did support it and worked.
https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/logman
[https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/traceview

](https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/traceview)Also found a useful set of slides for debugging drivers here that the developer of the driver pointed me to that describe some of the tools used:
https://www.slideshare.net/YanVugenfirer/windows-guestdebugging-kvmforum2012?qid=86d63495-1094-46a9-a8e1-9e76d14d2ea2&v=&b=&from_search=4

Finally the issue appears to be a problem in the virtualization software we're using and we're testing a fix they supplied.  Though to be fair neither of those tools helped here since things were "fine" from the driver aspect.

_{Thursday, May 9, 2019 11:12 PM}

I'd check here and with manufacturer about support for Server 2016.

https://www.windowsservercatalog.com/

also check the manufacturer chipset has been installed. The chipset allows the operating system to find and use devices integrated into the mother board.

Regards, Dave Patrick ....
Microsoft Certified Professional
Microsoft MVP [Windows Server] Datacenter Management

Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

_{Thursday, May 9, 2019 11:20 PM}

Thanks Dave, they do say that it is compatible with the version we're using.   I just realized a typo, meant to say Server 2019, have 2016 on the brain.  LOL

As for the other part this same driver works fine on 2012 R2 and on 2016, it's 2019 that's causing us problems.  Though there could be some other variable at play here.

So Windows Server doesn't log any information about drivers crashing/disabling themselves anywhere in Event Viewer?  My searches thus far have come up empty.

_{Thursday, May 9, 2019 11:32 PM}

You could check Device Manager for clues (devmgmt.msc) Any logging would be the responsibility of the driver developer. May have to ask developer for help with this.

Regards, Dave Patrick ....
Microsoft Certified Professional
Microsoft MVP [Windows Server] Datacenter Management

Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

_{Thursday, May 9, 2019 11:42 PM}

Yep, device manager was where I found the information I reported about the power state and the driver state.  Nothing else of interest there.

I'm working with the developer as well (well haven't yet heard back), but was hoping there was more debug/logging information I could give them from the server being logged by the OS as well.

Thanks for your help.

_{Thursday, May 9, 2019 11:49 PM}

You can check here in case the developer has provided for some logging.

/en-us/windows-hardware/drivers/kernel/writing-to-the-system-event-log

Regards, Dave Patrick ....
Microsoft Certified Professional
Microsoft MVP [Windows Server] Datacenter Management

Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

_{Friday, May 10, 2019 3:26 PM}

Hi,

Thank you for posting in Microsoft TechNet Forum.

What message have you got from the developer?

I will accompany you to the successful resolution of the problem.

Best regards,

Hurry

Please remember to mark the reply as an answer if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]

_{Saturday, May 11, 2019 2:50 PM}

Hi,

Thank you for posting in Microsoft TechNet Forum.

What message have you got from the developer?

I will accompany you to the successful resolution of the problem.

Best regards,

Hurry

Currently working on that as we're back in the data gathering phase.  This was their response:
https://github.com/virtio-win/kvm-guest-drivers-windows/issues/396

It appears that they're saying that the drivers should work, though we're using a new virtualization software here called bhyve, so we are in the process of trying to rule that in/out.  It takes a while as we need to create images and this was the first one we've created using that technology.  Previously we were using qemu.

I also found an issue where Device Manager was reporting a different driver than the one that the Diagnostics seemed to indicate was running, so testing that out by totally removing the drivers and replacing them with the ones that are working on 2012 R2 and 2016, though those are again using qemu.  EDIT - This test of ensuring that the working drivers were there failed to work, so that wasn't the issue.

We're likely going to have to also create a ticket with the virtualization software vendor we're currently using if it does point to bhyve being a potential root cause.

Thank you for your help, hoping we can provide a resolution one way or another.

Still any more data we can gather from Windows as to why network is stopping would be great.

The other night we had a different outcome, network still stopped in the middle of the night, but this time the driver remained enabled and the power state was still at D0 so it seems the driver remained running but perhaps the underlying virtualization turned off our router or what ever tech that stopped the network. 

Sorry, new to debugging this type of network issue, I've handled plenty of others but this one is more difficult as the logging is pretty sparse in Windows for this problem so figuring out the root cause is difficult.

Thanks,
Mike

_{Monday, May 13, 2019 12:03 PM}

Hi,

Thank you for your feedback.

I'm sorry that I'm not an expert on the network issue. But I will try my best to help you.

Had you run the Internet Connection troubleshooter when network stopped working?

or you can try to post it with network issue again.

Best regards,

Hurry

Please remember to mark the reply as an answer if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]

_{Monday, May 13, 2019 11:20 PM}

We found a couple of issues that were fixed in a more recent update of the virtualization software.  We've update and we're testing this overnight to see if the issue is resolved.

If not I will try to to get more information on tracing drivers using logman.  Though we're leaning towards this likely being an issue at a lower level than Windows and the driver.
/en-us/windows-server/administration/windows-commands/logman

_{Wednesday, May 15, 2019 1:37 AM}

Hi,

Thank you for your feedback.

Sincerely hope that your problem was solved successfully.

If you have any questions, please feel free to contact me.

Best regards,

Hurry

Please remember to mark the reply as an answer if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]

_{Wednesday, May 22, 2019 11:00 PM}

Please indicate whether you still need help with the network driver.

If you do some logs can be collected and some troubleshooting steps performed.

_{Friday, May 24, 2019 8:24 AM}

Hi,

I'm glad to hear that your problem is solved successfully.

Please mark it as an answer so that people in need of help in the future can refer to it.

Wish you a happy life.

Best regards,

Hurry

Please remember to mark the reply as an answer if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]