High non-paged pool memory usage on S2D failover clusters - refs.sys?

Question

High non-paged pool memory usage on S2D failover clusters - refs.sys?

Peter Camps 20

After several updates in August 2026 (Windows updates (KB5101372), secure boot certificates and HPE SPP P95170_001_gen10spp-2026.05.00.00-SPP2026050000.2026_0527.9 we are experiencing what looks like a memory leak on all Windows 2025 Core Hyper-V servers in S2D failover clusters. The non-paged pool is steadily increasing and eventually leads to a non-responsive host. After a reboot the non-paged pool starts at about 5 GB and eventually grows to several hundreds GB.

Poolmon shows after reboot about 1 week ago:

User's image

Refs.sys shows high usage. The version of refs.sys was 10.0.26100.33158. After discovering the problems we started installing KB5122871 and got refs.sys version 10.0.26100.33438. But the problem is still unresolved. At this moment we need to reboot the Hyper-V servers every few weeks to bring the non-paged pool usage back down.

We are using HPE Proliant DL380 Gen10 Plus servers with NVMe drives, Intel(R) Ethernet Network Adapter E810-XXVDA4 NIC's.

All storage is configured as 3-way mirror with CSVFS_ReFS

Windows Server Core 2025

But even in my small (virtual) test cluster i can see that the non-paged pool is steadily increasing. This is a nested Hyper-V cluster with no third party drivers.

Anybody experiencing the same issue? Or have a resolution for this problem?

Peter Camps

Chris Draper 20 Reputation points

2026-10-20T21:04:10.6933333+00:00

I have two Hyper-V clusters that are experiencing this issue. Two node clusters using HPE ProLiant DL385 servers. Windows Server 2025 Core. Two CSVs per cluster backed by ReFS. The growth in non-paged pool memory seems to only happen to the owner of the CSVs. I have a case open with Microsoft unified support.

Rolling reboots is unfortunately the answer until we get something from Microsoft.
Peter Camps 20 Reputation points

2026-10-21T07:12:19.12+00:00

Hi @Chris Draper thanks for your reply. Is it possible for you to share your Microsoft case id? I hope to be able to speak to an account manager of our distributor today. If all goes well I hope to open a case with Microsoft myself. Maybe it helps if I can share your case id.

Hope to hear from you.
Chris Draper 20 Reputation points

2026-10-22T00:58:40.48+00:00

TrackingID#2610140040006028 Case is currently with the performance team. Microsoft support is so painful these days. Still in the data collection stage because you can't convince them of the specifics of any issue. Will update with any new information.
Chris Draper 20 Reputation points

2026-11-03T19:39:35.16+00:00

Case is ongoing.

ReFS registry tuning hasn't changed the behavior. ReFS non-paged pool keeps growing in proportion to the workload on the node that owns the CSV, and it isn't released.

Waiting for the storage team for next steps

Rolling reboots of nodes is still the only remediation.
Peter Camps 20 Reputation points

2026-11-04T08:07:47+00:00

I have had a tip from Darryl van der Peijl (Splitbrain) about a forumpost at Veeam and a possible solution that was suggested. See https://forums.veeam.com/viewtopic.php?f=2&t=96912&start=300#p577432

I have tried this solution in my small test environment and saw some strange things that leads me to say that this is not a solution for our problems. I did not implement this in our production environment.

I am still trying to open a case with Microsoft but am running into several barriers.

Hope to have an update soon.

6 answers

Your answer

Chris Draper 20 Reputation points

2026-10-20T21:04:10.6933333+00:00

I have two Hyper-V clusters that are experiencing this issue. Two node clusters using HPE ProLiant DL385 servers. Windows Server 2025 Core. Two CSVs per cluster backed by ReFS. The growth in non-paged pool memory seems to only happen to the owner of the CSVs. I have a case open with Microsoft unified support.

Rolling reboots is unfortunately the answer until we get something from Microsoft.
Peter Camps 20 Reputation points

2026-10-21T07:12:19.12+00:00

Hi @Chris Draper thanks for your reply. Is it possible for you to share your Microsoft case id? I hope to be able to speak to an account manager of our distributor today. If all goes well I hope to open a case with Microsoft myself. Maybe it helps if I can share your case id.

Hope to hear from you.
Chris Draper 20 Reputation points

2026-10-22T00:58:40.48+00:00

TrackingID#2610140040006028 Case is currently with the performance team. Microsoft support is so painful these days. Still in the data collection stage because you can't convince them of the specifics of any issue. Will update with any new information.
Chris Draper 20 Reputation points

2026-11-03T19:39:35.16+00:00

Case is ongoing.

ReFS registry tuning hasn't changed the behavior. ReFS non-paged pool keeps growing in proportion to the workload on the node that owns the CSV, and it isn't released.

Waiting for the storage team for next steps

Rolling reboots of nodes is still the only remediation.
Peter Camps 20 Reputation points

2026-11-04T08:07:47+00:00

I have had a tip from Darryl van der Peijl (Splitbrain) about a forumpost at Veeam and a possible solution that was suggested. See https://forums.veeam.com/viewtopic.php?f=2&t=96912&start=300#p577432

I have tried this solution in my small test environment and saw some strange things that leads me to say that this is not a solution for our problems. I did not implement this in our production environment.

I am still trying to open a case with Microsoft but am running into several barriers.

Hope to have an update soon.

Answer 1

Fedor Travinsky 10

We see the same MSPi non-paged pool leak on a Windows Server 2025 S2D cluster (4 × Dell R640, CSVFS_ReFS, two CSVs: 9 TB NVMe mirror + 18 TB hybrid). Build 26100.33438 (KB5122871), refs.sys 10.0.26100.33438, so the September CU does not fix it for us either.

One node reached 40,066,531 MSPi allocations = 152.8 GiB non-paged pool after 24.8 days uptime (384 GB RAM); MSPi was 99% of all NP pool.

1. Growth is tied to CSV ownership

Owning one CSV: ~6 GB/day. Owning both: ~17 GB/day (~18,000 allocs vs ~3,000 frees per minute).
Nodes owning no CSV: MSPi stays near zero (0.01 GB on a node running 12 VMs).
After moving the CSVs off a node, growth stops immediately (0 allocs/frees per minute), but the memory is never released. Only a reboot frees it.

2. Live kernel dump (LiveKD -ml) of the affected node

!poolused 2 MSPi: 40,066,531 allocs, 164,108,436,752 bytes, Minstore generic/untagged allocation, refs.sys
Every allocation is a large page allocation of exactly 0x1000 bytes.
24 blocks sampled across ~1.48M hits from !poolfind MSPi -nonpaged -large are byte-for-byte identical: 512 bytes of data, the rest zero, no pointers to code or kernel objects.
Those 512 bytes are a GPT protective MBR (LBA 0):

Offset	Bytes	Meaning
0x000	`33 C0 8E D0 BC 00 7C …`	Windows MBR boot code
0x1B8	`00 00 00 00`	Disk signature 0 (GPT)
0x1C2	`EE`	Partition type: GPT protective
0x1C6	`01 00 00 00`	Starting LBA 1
0x1CA	`FF FF FF FF`	Size: disk > 2 TiB
0x1FE	`55 AA`	Boot signature

So it looks like refs.sys (Minstore) repeatedly reads sector 0 of a large GPT disk, most likely the CSV virtual disk, into a 4 KB non-paged buffer that is never freed. At ~6 GB/day that is roughly 17 reads/sec per CSV on the coordinator.

How to check your own nodes

poolmon -b                                        # MSPi Nonp bytes per node
Get-ClusterSharedVolume | ft Name, OwnerNode      # compare with CSV owners
!poolused 2 MSPi                                  # in a dump
!poolfind MSPi -nonpaged -large                   # then db <addr> on a few hits

Interim mitigation: watch MSPi per node, move CSVs away and reboot before NP pool gets critical. A live dump is available if Microsoft wants it.

Fedor Travinsky 10 Reputation points

2026-10-21T12:11:32.6666667+00:00
Update – root cause narrowed down

We traced the MSPi allocations with stacks (xperf -on PROC_THREAD+LOADER+POOL -stackwalk PoolAlloc+PoolFree -PoolTag MSPi, 11.5 min, 0 lost events). 95% of the outstanding MSPi bytes come from one stack, and none of those allocations are ever freed:

ReFS.SYS!operator new ReFS.SYS!utl::vector<utl::byte>::reserve ReFS.SYS!_VCB::ReadBootSectorForMount ReFS.SYS!RefsMountVolume ReFS.SYS!RefsCommonFileSystemControl ReFS.SYS!RefsFspDispatch nt!ExpWorkerThread

So every time something triggers a mount attempt and ReFS is offered a device, it reads sector 0 into a 4 KB non-paged buffer, decides "not my volume", and leaks the buffer. That is why every leaked page contains a GPT protective MBR (LBA 0).

What triggers the mount attempts in our case: Dell OpenManage Server Administrator 11.1

An ETW FileIoCreate trace showed dsm_sa_datamgr64.exe (DSM SA Data Manager) opening every \Device\HarddiskN\DRN about 38 times per second per disk (~520 opens/s in total). On a node that has clustered Storage Spaces virtual disks attached (CSV owner, and the owner of the SDDC Group with the ClusterPerformanceHistory disk), these opens turn into ReFS mount attempts. That gives ~6 GB/day of MSPi per attached disk.

Stop test on an affected node, MSPi net growth:

State MSPi growth

DSM SA Data Manager running 9.2 MB/min

Service stopped (2 min) 0.4 MB/min

Service started again (after 4 min) 9.9 MB/min

Corrections to my previous comment

The leak follows the node that has the clustered disks attached: the CSV owner and the SDDC Group / ClusterPerformanceHistory owner, not only the "CSV coordinator".

The PMBR size field 0xFFFFFFFF does not mean ">2 TiB". Windows writes it for every GPT disk. The leaked sector matched our 24 GB ClusterPerformanceHistory disk byte-for-byte (CHS end bytes FE 7F 86).

Timeline

No leak on 26100.32522 (refs.sys).

Slow growth on 32995 (June CU).

~10× faster from the July CU KB5099536 (26100.33158), which is the only CU in the last 6 months with a large refs.sys change (+106 KB) and the first one that ships KernelFSFormatter.sys (also contains the MSPi tag).

Still present on 33296 and 33438.

Summary: the bug is in refs.sys (the buffer allocated in _VCB::ReadBootSectorForMount is not freed on a failed mount). Any agent that frequently opens raw disk devices turns it into a fast leak. In our case that agent is Dell OMSA. @Peter Camps – with HPE SPP it may be worth checking the HPE agents (AMS, SSA, storage/health agents) the same way:

xperf -on PROC_THREAD+LOADER+FILE_IO+FILE_IO_INIT -stackwalk FileCreate -f fc.etl (wait 2 minutes) xperf -d fc-merged.etl xperf -i fc-merged.etl -o fc.csv -a dumper → group FileIoCreate rows by process for \Device\HarddiskN\DRN

Workaround: stop/disable the DSM SA Data Manager service (dcstor64) on cluster nodes (hardware monitoring via iDRAC/iSM instead), then drain and reboot nodes to release the already leaked pool. A Microsoft support case has the stacks and a live dump available.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Fedor Travinsky 10 Reputation points

2026-10-21T15:24:50.3066667+00:00
Follow-up: root cause of the MSPi leak on our cluster

Environment: Windows Server 2025, 4-node S2D cluster (Dell PowerEdge R640), CSVFS on ReFS, OS build 26100.33438 (09-2026 CU, KB5122871), refs.sys 10.0.26100.33438.

1. Where the leak is

Pool tag MSPi is ReFS Minstore "generic/untagged allocation" (refs.sys). On one node: 40,066,531 outstanding allocations, 164,108,436,752 bytes (152.8 GiB), 99% of non-paged pool after 24.8 days uptime. Almost every allocation is a 0x1000-byte large-page allocation.

A WPT pool trace (tag filter MSPi, stack walk on PoolAlloc/PoolFree, 0 lost events) shows that 95% of outstanding MSPi bytes come from a single path, and none of those allocations are freed:

nt!ExpWorkerThread

ReFS!RefsFspDispatch

ReFS!RefsCommonFileSystemControl (IRP_MJ_FILE_SYSTEM_CONTROL / IRP_MN_MOUNT_VOLUME)

ReFS!RefsMountVolume

ReFS!_VCB::ReadBootSectorForMount

ReFS!utl::vector<utl::byte>::reserve / _SetCapacity

ReFS!operator new → nt!ExAllocatePool2 (tag MSPi, 4 KB)

On a node with two affected disks this path ran about 42 times per second, which matches the observed growth.

2. What is in the leaked buffers

From a live kernel dump: 24 MSPi pages sampled across ~1.48 million hits of !poolfind MSPi -nonpaged -large. All are byte-for-byte identical: 512 bytes of data followed by zeros, no pointers.

The 512 bytes are a GPT protective MBR (boot code 33 C0 8E D0 BC 00 7C, partition type 0xEE, start LBA 1, signature 55AA). The CHS end bytes match LBA 0 of the node's ClusterPerformanceHistory virtual disk exactly.

So each leaked page is the boot-sector buffer ReFS read from the whole-disk device (partition 0) during a mount attempt, which ReFS then rejected without freeing the buffer.

3. What triggers the mount attempts

ETW FileIoCreate and Process Monitor captures points to a vendor hardware monitoring app (dsm_sa_datamgr64.exe, service "DSM SA Data Manager") looping over every HarddiskN\DRN device. Per disk, about 10–17 times per second:

CreateFile: Desired Access Generic Read/Write, Disposition Open, Options Non-Directory File, Share Read/Write

IOCTL_STORAGE_GET_DEVICE_NUMBER

Close

Because the open requests data access, it is not a direct device open. The I/O manager mounts a file system on the device, offering it to the registered file systems, ReFS included. On a clustered Storage Spaces virtual disk, ReFS reads LBA 0 in ReadBootSectorForMount, rejects the volume, and leaks the 4 KB buffer.

At ~17 opens/s per disk: 4 KB × 17 × 86,400 ≈ 6.07 GB/day, which is exactly the per-disk growth we observe.

Stop/start test on an affected node (MSPi net growth):

service running: 9.2 MB/min

service stopped: 0.4 MB/min

service restarted: 9.9 MB/min

4. Scope

Only nodes with clustered virtual disks attached leak: the CSV owner and the owner of the SDDC Group (ClusterPerformanceHistory disk). When those roles move (cluster event 1641), the growth moves with them. Leaked memory is only released by a reboot.

The agent never opens one of our two CSV disks, and that disk's owner does not leak.

The same agent opens the boot disk and the S2D physical pool disks with identical flags at the same rate, but they do not produce MSPi growth. Why only clustered virtual disks hit this path is not yet clear.

5. Regression window

From ClusterNode.Memory.Usage.Host history and per-node install dates:

26100.32995 (June 2026, KB5094125): slow growth only, ~0.5–0.8 GB/day per node.

26100.33158 (July 2026, KB5099536): ~10× increase. This is the only cumulative update in six months with a large refs.sys change (+106 KB), and the first that ships KernelFSFormatter.sys, which also contains the MSPi tag.

Still present on 26100.33296 (KB5120233) and 26100.33438 (KB5122871 / OOB KB5129235; identical refs.sys).

6. Notes for this thread

The residual ~0.4 MB/min with the agent stopped suggests that other software opening disk devices with data access, possibly in-box components, triggers the same path at a lower rate. That may be what Peter sees in the nested test cluster without third-party software.

The ReFS trim registry values (RefsEnableLargeWorkingSetTrim, RefsEnableInlineTrim, RefsNumberOfChunksToTrim) and CSV BlockCacheSize are unlikely to help: the leaked pages are mount-time boot-sector buffers, not metadata cache.

7. How to check your own nodes

Watch MSPi Nonp bytes per node in poolmon and compare with the owners of the CSVs.

Stop disk-polling agents (hardware monitoring, storage management, inventory, backup) for a few minutes and watch the MSPi rate.

To identify the opener: ETW FileIoCreate (or Process Monitor) filtered on HarddiskN\DRN, looking for opens with read/write access.

8. Workaround

Stop any drive polling services on cluster nodes.

Wait for Microsoft to pickup the issue. (Can provide stacks, traces and a live dump on demand).

Answer 2

Andy Taylor 0

Hi @Peter Camps Did you get anywhere with this? We're having exactly the same issue with a 2 node 2025 Datacenter Hyper-V cluster. CSV Ownership seems to trigger this increasing MSPi allocation. We're seeing approx 1GB an hour until the host becomes unresponsive.

Peter Camps 20 Reputation points

2026-10-20T09:58:39.5433333+00:00

Hi @Andy Taylor , unfortunately I still don't have a solution. I am trying to get a Microsoft case opened via our SPLA distibutor, but they need to assign a consultant to my case to continue this at MS.

In the mean time I have installed the latest HPE Gen10 SPP - P99498_001_gen10spp-2026.09.00.00-SPP2026090000.2026_1012.25.iso, also installed the latest Windows updates but this did not solve my problems.

In the meantime I am keeping an eye on our clusternodes non-paged pool byte usage and start rebooting them when necessary.

I also noticed that S2D clusters with a lot of CSV's the MSPI allocation is increasing a lot faster than clusters with just a few CSV's.

Just let me know when you find a solution, I hope I can get through to Microsoft.

Answer 3

Peter Camps 20

Yesterday i upgraded the firmware and drivers of the Intel(R) Ethernet Network Adapter E810-XXVDA4 to 5.00 (firmware) and 1.21.114.0 (drivers) on a 2-node cluster. But looking at the graphs today it is still slowly growing.

At this moment I am growing out of options. There is a new update (KB5126646) which again introduces a new refs.sys but I installed this on 1 cluster and it changed de refs.sys version but not the issues we are having.

Peter

0 comments

Answer 4

I have opened a case with our distributor (cannot use our MS Premier support) and they advised me to change/add a few registry settings.

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "RefsEnableLargeWorkingSetTrim" -Value 1 -Type DWord

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "RefsNumberOfChunksToTrim" -Value 16 -Type DWord

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "RefsEnableInlineTrim" -Value 1 -Type DWord

I have done this on 2 Failover clusters and have been monitoring the non-paged pool usage. But it looks like (after a week) that it does not change the behaviour. It is still steadily growing, most of the nodes start at about 6 GB when rebooted, and now after the registry changes and a reboot on october 8 they have grown to 23 - 42 GB of non-paged pool usage. The graph shows a steady growth,

Has anybody seen this in their environment? I am stuck at the moment.

Peter

Answer 5

Hi Peter Camps,

Your poolmon.exe capture shows that the MSPi pool taghas nearly 12 million unfreed allocations consuming roughly 49 GB of Non-Paged Pool (Nonp) RAM after one week. You reproduced this exact Non-Paged Pool growth in a nested virtual Hyper-V cluster with no third-party drivers, the HPE Gen10 SPP firmware and Intel E810-XXVDA4 drivers are ruled out as root causes. While KB5101372 (the Servicing Stack Update bundled with the July 2026 Cumulative Update KB5099536, build 10.0.26100.33158) and the September 8, 2026 Cumulative Update KB5122871 (build 10.0.26100.33438) updated refs.sys, neither resolved this leak, and the September 14, 2026 Out-of-Band update (KB5129235, build 10.0.26100.33451) only addresses Remote Desktop and USB audio issues. This seems an unresolved Microsoft bug in refs.sys, there is no permanent fix yet, and you will need to wait for an official Microsoft Cumulative Update that patches the MSPi leak.

While waiting for an official Microsoft fix, you can configure Microsoft's documented ReFS metadata trimming parameters under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem on every cluster node by creating the REG_DWORD values RefsEnableLargeWorkingSetTrim set to 1, RefsEnableInlineTrim set to 1, and RefsNumberOfChunksToTrim set to 16 or 32, followed by a rolling node reboot. These parameters force refs.sys to unmap inactive metadata streams inline and at every checkpoint rather than relying on lazy garbage collection; however, because they primarily trim the ReFS metadata working set rather than freeing leaked MSPi Non-Paged Pool pages, they may only modestly reduce memory pressure. To preserve Non-Paged Pool headroom and prevent Srv Event ID 2019 pool exhaustion hangs, check (Get-Cluster).BlockCacheSize in PowerShell to ensure the CSV Block Cache (which also allocates from Non-Paged Pool RAM) is kept conservative, continue scheduling rolling node reboots via Suspend-ClusterNode -Drain, and log a support case with Microsoft Support with your poolmon.exe output and a non-disruptive kernel live dump (Get-StorageDiagnosticInfo or livekd) to escalate the refs.sys defect.

Hope this answer has brought you some useful information. If it did, please hit “accept answer”. Should you have any questions, feel free to leave a comment.

VPHAN

Peter Camps 20 Reputation points

2026-10-02T13:18:55.23+00:00

Hi VPHAN,

Thanks for your reply, I dropped this case at our distributor and hope to get an (Microsoft) answer for this issue. Our current Premier Support contract cannot be used because of an ongoing usage conflict.

I also saw the 3 registry settings for ReFS metadata trimming parameters, but was unsure if this would be a "solution" for our CSVFS_ReFS volumes. I have seen these settings a lot for ongoing ReFS issues on Veeam repository servers, was not sure if this could be implemeted on our Hyper-V hosts without creating other issues.

I am going to test this first in my virtual cluster, see what is happening. Of course I won't be able to simulate the load like we have on the live clusters but maybe it will give me new insights.

(Get-Cluster).BlockCacheSize = 1024 on all nodes

Thanks for your reply

Peter

State	MSPi growth
DSM SA Data Manager running	9.2 MB/min
Service stopped (2 min)	0.4 MB/min
Service started again (after 4 min)	9.9 MB/min

High non-paged pool memory usage on S2D failover clusters - refs.sys?

6 answers

Follow-up: root cause of the MSPi leak on our cluster

1. Where the leak is

2. What is in the leaked buffers

3. What triggers the mount attempts

4. Scope

5. Regression window

6. Notes for this thread

7. How to check your own nodes

8. Workaround

Your answer