Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Monday, November 10, 2014 6:29 PM
Hi,
As x64 bit servers has more virtual address space for page and non-page pool memory, event id 2020 and 2019 does not get logged by "srv.sys". However if a faulty driver is present, it will leak memory from both pools and eventually will hang the server.
To make sure, I have recreated the issue (leaking paged and non-paged pool memory) using Mark's "NOTMYFAULT" in "Server 2008 SP2 Ent. x64 Bit". The server got hanged but never logged 2020 and 2019.
It seems to me like a kind of drawback for troubleshooters in such scenarios!
In my lab environment I knew the that there was a memory leak and the tag is "leak" but in real world, how to begin with such server hang scenarios. I know capturing a complete memory dump and analyzing the same may give a solution, but that is bit 'techy' and frankly, does not always works with me. (I'm not good at memory dump analyzing).
If we need to use PerfMon, then what are the counters that will indicate me that there is potential memory leak and its corresponding driver.
In Server 2000 and 2003s, I have used PoolMon however not sure how to use it in Server 2008 and above.
I have also tried to use 'Sysinternal' VMMaap and RAMMap however was not able to find anything significant. Process Explorer shows a steady growth however it is not very much alarming.
** Please help me with the identification of memory leak in x64 bit machines without looking into the memory dump.
Thank you for your kind attention.
Kuntal K. Basu Windows Server Performance Expert
All replies (7)
Friday, November 14, 2014 10:56 PM ✅Answered | 1 vote
Hi Kuntal,
Here is what you can do generally to begin with any "Server Hang" issue.
1. Create a PerfMon counter with following command:
Logman.exe create counter Srv.Hnag.2Min -o "C:\PerfLogs\Srv.Hnag.2Min.blg" -f bincirc -v mmddhhmm -max 1024 -c "\Memory\" "\Network Interface(*)\" "\Paging File(*)\" "\PhysicalDisk(*)\" "\Processor(*)\" "\Process(*)\" "\Server\" "\System\" -si 00:02:00
- May change the sample interval as per your need.
2. Analyze the PerfMon log in every 24/48 hours. To find a trend of depilation of any resource. If there is memory leak, you would see something similar like this:-
In above picture non-page-pool was leaking.
3. Now you know what is causing your server to become hang. Next question would 'why'?
Download PoolMon from Windows Driver Kit or you may get it with Server 2003 Support Tools and Resource Kit as well. Poolmon does not have dependencies on other modules in this folder; you can copy it to your other computers when you need to investigate pool usage. Refer PoolMon Startup Command.
I guess you know what next !!
Cheers :)
Do refer following posts:
Troubleshooting Pool Leaks Part 1 – Perfmon
Troubleshooting Pool Leaks Part 2 – Poolmon
Tuesday, November 11, 2014 5:52 PM
Friends,
I have also posted the same in Syinternals Forum; http://forum.sysinternals.com/topic30945_post144450.html#144450
I will try my best to keep a manual sync in both forums.
Thanks.
Kuntal K. Basu Windows Server Performance Expert
Wednesday, November 12, 2014 5:35 AM | 1 vote
Hi,
I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
Thanks for your understanding and support.
Wednesday, November 12, 2014 5:19 PM
Thank you for your kind attention, I appreciate your initiative.
Kuntal K. Basu Windows Server Performance Expert
Thursday, November 13, 2014 10:34 AM | 1 vote
Hi,
Thank you for your post.
From your description, I learn that you want to know if there is a way to troublshoot memory leak in x64 bit machines without looking into the memory dump.
Please let me know if I have misunderstood anything.
Based on my knowledge and research, to trouleshoot such issue, we can add the related counters in Perfomance Monitor.
For example:
Memory\Free System Page Table Entries
Memory\Pool Non-Paged Bytes
Memory\Pool Paged Bytes
Memory\Pages per Second
Regarding the detailed information, please refer to the following article.
http://technet.microsoft.com/en-us/magazine/2008.08.pulse.aspx
Meanwhile, you may have a look at the following blog.
Troubleshooting Pool Leaks Part 7 – Windows Performance Toolkit
As for PoolMon, it should be used on Microsoft Windows XP and later versions of Windows, you can refer to this link.
http://msdn.microsoft.com/en-us/library/ff550442(v=vs.85).aspx
Hopefully the information is helpfu.
Please let me know if you have any questions. Thanks for your time.
Best Regards,
Sophia Sun
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
Thursday, November 13, 2014 5:51 PM
Thank you Sophia for your attention.
Yes, you have got me right.
For our partners if a server hangs and they/we need to reboot it, in order to get it back (during business hours) creates a question mark on our credibility to maintain business uptime.
The same also disgrace the point of capturing a dump during server hang (during business hours). And again I'm not very good in memory dump analysis (even if I take the risk).
To avoid that I/we may reboot the server everyday or may be every week during off hours but that is only a dumb workaround.
I hope you have understood my point here.
Thank you again.
Kuntal K. Basu Windows Server Performance Expert
Sunday, November 16, 2014 5:21 PM
Thank you Sophia and Ankita for your help.
I found following links very useful:
2. Troubleshooting Pool Leaks Part 1 – Perfmon
3. Troubleshooting Pool Leaks Part 2 – Poolmon
Apart from that the article posted by Ankita Singh Basu is very useful.
Thanks.
Kuntal K. Basu Windows Server Performance Expert