8

I'm trying to write a script that would go through 1.6 million files in a folder and move them to the correct folder based on the file name.

The reason is that NTFS can't handle a large number of files within a single folder without a degrade in performance.

The script call "Get-ChildItem" to get all the items within that folder, and as you might expect, this consumes a lot of memory (about 3.8 GB).

I'm curious if there are any other ways to iterate through all the files in a directory without using up so much memory.

3 Answers 3

14

If you do

$files = Get-ChildItem $dirWithMillionsOfFiles
#Now, process with $files

you WILL face memory issues.

Use PowerShell piping to process the files:

Get-ChildItem $dirWithMillionsOfFiles | %{ 
    #process here
}

The second way will consume less memory and should ideally not grow beyond a certain point.

2
  • Thanks for the nice and simple solution. I had always thought pipelining in powershell return the entire result before processing next function.
    – T.Ho
    Commented Sep 5, 2012 at 4:58
  • 2
    This actually still requires O(n) memory, but if it solves the problem then I agree it's the best solution.
    – latkin
    Commented Sep 5, 2012 at 16:53
13

If you need to reduce the memory footprint, you can skip using Get-ChildItem and instead use a .NET API directly. I'm assuming you are on Powershell v2, if so first follow the steps here to enable .NET 4 to load in Powershell v2.

In .NET 4 there are some nice APIs for enumerating files and directories, as opposed to returning them in arrays.

[IO.Directory]::EnumerateFiles("C:\logs") |%{ <move file $_> }

By using this API, instead of [IO.Directory]::GetFiles(), only one file name will be processed at a time, so the memory consumption should be relatively small.

Edit

I was also assuming you had tried a simple pipelined approach like Get-ChildItem |ForEach { process }. If this is enough, I agree it's the way to go.

But I want to clear up a common misconception: In v2, Get-ChildItem (or really, the FileSystem provider) does not truly stream. The implementation uses the APIs Directory.GetDirectories and Directory.GetFiles, which in your case will generate a 1.6M-element array before any processing can occur. Once this is done, then yes, the remainder of the pipeline is streaming. And yes, this initial low-level piece has relatively minimal impact, since it is simply a string array, not an array of rich FileInfo objects. But it is incorrect to claim that O(1) memory is used in this pattern.

Powershell v3, in contrast, is built on .NET 4, and thus takes advantage of the streaming APIs I mention above (Directory.EnumerateDirectories and Directory.EnumerateFiles). This is a nice change, and helps in scenarios just like yours.

8
  • I think using pipeline with Get-ChildItem like manojlds had suggested would achieve the same thing, but thanks for showing me how to use .Net with powershell! :).
    – T.Ho
    Commented Sep 5, 2012 at 5:01
  • Yep, get-childitem | foreach-objetc { ... } will also process only one passed item as a time.
    – x0n
    Commented Sep 5, 2012 at 14:35
  • 2
    See my edit. get-childitem | foreach {...} is only pseudo-streaming, it technically still requires O(n) memory.
    – latkin
    Commented Sep 5, 2012 at 16:49
  • Thanks for the clarification. I'm actually using Powershell V3 on Windows 8, so perhaps that's why it is enumerating through the directory and actually return an FileInfo object for the $input variable. So then, is it true if I were to run this script on Powershell V2 (Windows 7), the $input variable would not be a FileInfo object, but rather an n-size string array?
    – T.Ho
    Commented Sep 5, 2012 at 17:21
  • 1
    No, not quite. All versions will output FileInfo objects. What I'm saying is that in v2, internally a big string array is generated up front, and elements from that array are used one at a time to build FileInfo objects to send down the pipeline. Only one FileInfo is in memory at once, but the array of file paths is stored in its entirety, internally. In v3, no internal array is created at all, everything is fully streamed.
    – latkin
    Commented Sep 5, 2012 at 17:28
0

This is how I implemented it without using .Net 4.0. Only Powershell 2.0 and old-fashioned DIR-command:

It's just 2 lines of (easy) code:

cd <source_path>
cmd /c "dir /B"| % { move-item $($_) -destination "<dest_folder>" }

My Powershell Proces only uses 15MB. No changes on the old Windows 2008 server!

Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.