List all files and their respective path with a specific extension

Question

_{Friday, November 26, 2010 11:50 AM}

Hello, another day and another request.

I need to list all files with a specific extension, and get their full path as well.

I do have a script that seems to do the job as follows :

$Dir = get-childitem c:\ -recurse
$List = $Dir | where {$_.extension -eq ".xxx"}
$List |ft fullname |out-file C:\output.txt

My problem is however that I'm running this against 2Tb of data and Powershell seems to be storing everything in memory before writing the output file once it's done parsing everything. Which means I've had to stop the script after a couple of hours and over 3Gb of used RAM.

Anyone got a way of performing this task without hogging all the memory and taking a whole afternoon to run ?

All replies (8)

_{Saturday, November 27, 2010 2:08 PM ✅Answered | 1 vote}

Try this

get-childitem c:\ -recurse -filter *.xxx | select fullname,length | export-csv C:\output\xxxfiles.csv -notype

Shay Levy [MVP]
PowerShay.com
PowerShell Toolbar

_{Friday, November 26, 2010 1:10 PM}

Was able to whip up something thanks to Shay Levi's reply to my other question :

dir "c:\ -rec -Filter *.xxx | select name,@{n='TotalSize(MB)';e={ (($_.group | measure length -sum).sum)/1Mb}}>>C:\output.csv

I can now build up on that one to add the features I need :)

_{Friday, November 26, 2010 1:41 PM | 2 votes}

If you just need of the full paths of files with a specific extension, try this:

cmd /c dir c:\.txt /b /s /a-d | out-file c:\output.txt

I think you'll find a noticable performance difference over using gci on large directory structures.

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

_{Friday, November 26, 2010 2:05 PM}

Hello,

Need full path and size, currently running the following script :

dir "w:\ -rec -Filter *.mdb | select { $_.FullName },{($_.Length/1Mb)} | export-csv C:\output-mdb.csv

Taking some time since the target volume contains almost 2Tb of data but it writes the output on the fly which doesn't clog up the memory. Can't take credit for it though, just took Shay's script and changed it around a bit to suit my needs.

_{Friday, November 26, 2010 2:08 PM}

Ah.  If there was anything in the original post about needing the size, I missed that.  In any case if you need that along with the full paths, the solution I provided will not contain that information.

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

_{Friday, November 26, 2010 2:13 PM}

You didn't miss it, I forgot to add it :)

_{Friday, November 26, 2010 2:18 PM}

FWIW, the problem of eating up large amounts of memory is caused by using cmdlets or functions that don't/can't operate in streaming mode.

There's a good write up on it here:

http://powershell.com/cs/blogs/ebook/archive/2008/11/23/chapter-5-the-powershell-pipeline.aspx

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

_{Saturday, November 27, 2010 1:51 AM}

Hello,

Need full path and size, currently running the following script :

dir "w:\ -rec -Filter *.mdb | select { $_.FullName },{($_.Length/1Mb)} | export-csv C:\output-mdb.csv

Taking some time since the target volume contains almost 2Tb of data but it writes the output on the fly which doesn't clog up the memory. Can't take credit for it though, just took Shay's script and changed it around a bit to suit my needs.

I'm sorry, are you good now or still need help?