Share via

Very slow deleting large folder - a work-around

Question

Tuesday, August 9, 2016 4:38 PM

Occasionally I have folders to delete that contain hundreds of thousands of files and sub-folders, sometimes over 100GB of data. It can take a day or longer, if File Manager doesn't crash and I have to start it all over. Really a pain!

Does anyone know why? Is there anything that I can do to my Windows-10 system to make if faster?

I have written a small .BAT file that seems to bypass the normal File Manager to delete these folders using ReboCopy and it is much faster than issuing DOS DEL and RD commands.  I would be willing to share this code, how should I post it? Where? I deleted a folder, and a clone of it and compared the timings: 25 seconds with my BAT file vs 2.5 minutes with File Manager.  Better timings as the files get larger.

Ernest

All replies (14)

Tuesday, August 9, 2016 5:01 PM

Prior to to copy a lot of small files or to remove them, explorer have to enumerate the content of the folder (sub-folders). It could take a while. Enumeration is need to show the progress bar, to control the operations and so on.

To avoid this "redundant" enumeration, we could use any other application that makes it directly - robocopy from your example also does it great.

Tuesday, August 9, 2016 5:28 PM

You post the code in the a code block on the toolbar, or just as plain text;

robocopy e:\empty e:\to_delete /MIR /E

robocopy e:\empty e:\to_delete /MIR /E

Tuesday, August 9, 2016 5:30 PM

Dmitry, you did a little to explain why it is so slow, but did not tell us how to fix it, how to turn off this 'enumeration'. There are may little settings inside Windows, is there one to help with this?

Sunday, August 14, 2016 5:35 AM

Hi,

We cannot skip this because it's designed by default. The workaround could be the scripts that you have.


Technical Explanation

The reason that most methods are causing problems is that Windows tries to enumerate the files and folders. This isn’t much of a problem with a few hundred—or even thousand—files/folders a few levels deep, but when you have trillions of files in millions of folders going dozens of levels deep, then that will definitely bog the system down.

Let’s you have “only” 100,000,000 files, and Windows uses a simple structure like this to store each file along with its path (that way you avoid storing each directory separately, thus saving a some overhead):

struct FILELIST {                   // Total size is 264 to 528 bytes:
  TCHAR         name[MAX_PATH];     // MAX_PATH=260; TCHAR=1 or 2 bytes
  FILELIST*     nextfile;           // Pointers are 4 bytes for 32-bit and 8 for 64-bit
}

Depending on whether it uses 8-bit characters or Unicode characters (it uses Unicode) and whether your system is 32-bit or 64-bit, then it will need between 25GB and 49GB of memory to store the list (and this is a a very simplified structure).

The reason why Windows tries to enumerate the files and folders before deleting them varies depending on the method you are using to delete them, but both Explorer and the command-interpreter do it (you can see a delay when you initiate the command). You can also see the disk activity (HDD LED) flash as it reads the directory tree from the drive.

Please remember to mark the replies as an answers if they help and unmark them if they provide no help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

Sunday, August 14, 2016 8:38 PM

Then it is time to suggest a change - MICROSOFT, Are you listening? There needs to be a QUICK Delete that does not enumerate the files first.

That, and my File Explorer - file manager crashes often when I try to delete large folders. Then the delete display goes dormant (0 files/second) and I have to crash it and start all over.

So far, I have not heard anyone come up with an answer to this issue, not even a good work-around.

Tuesday, August 16, 2016 10:14 AM

Yes, Thanks for your feedback on this issue. 

We can also submit what you advise on Feedback hub which can be searched in Start menu. 

I am proposing  previous reply as "Answered". Please feel free to try it and let me know the result. If the reply is helpful, please remember to mark it as answer which can benefit our Community members

Please remember to mark the replies as an answers if they help and unmark them if they provide no help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

Tuesday, August 16, 2016 1:14 PM

Kate,

I don't know who you work for, Microsoft or someone else and just help out on the forums. You must have a way to communicate with Microsoft. Please tell them that they need to make a QUICK DELETE that can start deleting files while still doing the enumeration that you mention. (And something that will let us know when File Explorer dies, if it relies on File Explorer at all).

Ernest

Tuesday, August 16, 2016 1:24 PM

did you try what Mr Happy suggested?
Robocopy is from Microsoft, so they have a tool to quick delete files.

Tuesday, August 16, 2016 1:43 PM

I have been working on code (including using ROBOCOPY) to provide a faster delete process. I have not completed my development, but if you would like to contact me directly and tell me a little about your situation and needs, I will share what I am working on with you. Read my handle very carefully and you will know how to contact me.

Friday, December 8, 2017 6:49 PM

So now we have to have a workaround just to delete files and folders...    is there a workaround for Windows 10 as well?

Thursday, June 28, 2018 9:07 AM

Hello Dmitry.

I am sorry for the late reply, and with all due respect, no, your Windows Explorer does not have to enumerate all files and folders ahead of time, and it does not have to show a progress bar. It was your choice to make it do that, you can change your mind about that just as well.

On the other hand, of course there is a benefit to having a progress bar, but why force it down a user's throat to either accept the extremely long delays for collecting this information (at some point you *will* have to do something about your file system cache which seems to perform pretty badly, too), or to let them fumble with a command line tool which easily brings the risk of deleting the wrong folder very easily, because they are simply not so familiar with command line usage?

It does not have to be complicated, simply provide a button (or so) "Start deleting now" with tooltip "Deleting will start immediately, but time prediction will possibly be disabled or be inaccurate."

Thursday, June 28, 2018 9:13 AM

Hello Kate.

With all due respect, aren't you explaining why your design and implementation is insufficient for a large number of files and folders?

If so, the answer should be "yes, this should be changed if we want to make this work-flow less painful and as efficient as technically possible", not "it was designed like this at some point and we cannot change it".

You also say "it's designed by default", but it is not a default, it is the only mode of operation in Windows Explorer, isn't it?

Perhaps consider how others are solving the problem. You may find that on macOS after a while of collecting information upfront macOS starts deleting no matter what, and indicating that no proper time prediction is possible at this point yet.

Thursday, June 28, 2018 9:15 AM

The workaround using RoboCopy works on all Windows variants.

Thursday, February 13, 2020 11:31 AM

Thats not true. This is slow only in Win10. 
In Windows 7 and older deletion and copying  is much faster and there is no slow "enumeration" phase.

Just try to delete large directory without Recycle Bin, try shift-del in Totalcms or ancient rmdir /Q /S