Get-Childitem with millions of files

Question

_{Friday, September 12, 2014 8:01 AM}

Hello all,

My task is to create a report with all subfolders in a given path, sorted after the newest LastWriteTime on any file in the folder.
The ultimate reason is to determine which folders have been unused for long enough to move to offline storage.

So far I've come up with this:

Get-ChildItem $folderpath | Where {$_.PsIsContainer} | foreach-object { Get-ChildItem $_ -Recurse -ErrorAction SilentlyContinue | Where {!$_.PsIsContainer} | Select Name,DirectoryName, LastWriteTime, Mode | Sort LastWriteTime -descending | select -first 1}|export-csv $drive:\report.csv

I know this works, I've tested it on a couple of folders.
The problem is that the folder I really want to run the report for contains approximately 6.5 million files in many thousands of subfolders. Memory usage goes through the roof, server gets unresponsive, I have to kill the script before users get angry.

I suppose that PowerShell tries to create an array in memory before actually piping and sorting, and that's the reason for the memory problem.

Is there a smarter way to do it?

All replies (4)

_{Friday, September 12, 2014 9:15 AM ✅Answered | 2 votes}

It is all on 1 line, everything is going through the pipeline, it has to do each part of the pipeline before it`s gets to the next part of the pipeline.

Do each each folder separately, here is a function to step through a folder structure recursively. Customize it for your needs. The maximum amount of resources used will be limited to one folder.

function Recurse-Folders([string]$path) {

  Set-Location -Path $Path
  
  #  $L = Get-Location
  #  Do something at this location
    
  [String[]]$Folders  = Get-ChildItem "." -Directory
  ForEach ($F in $Folders)
  {
    Recurse-Folders($F)
    Set-Location -Path ".."
  }
}

_{Friday, September 12, 2014 9:03 AM}

Im not suprised you have high resources usage..  :)

_{Friday, September 12, 2014 9:15 AM}

Why so many pipelines, sort etc

Get-childitem $folderpath -recurse |?{$_.psiscontainer} |select fullname,lastwritetime |sort  doesn't work for ya?

Why ure looking for files in the folders?

_{Friday, September 12, 2014 11:21 AM}

Thank you, test works perfectly.
I will customize this and get it to write a nice report for those responsible for the state of the file server. They will get the joy of moving their files themselves. :-)