CppCon 2015: Niall Douglas “Racing The File System"
Why Stack Overflow recommends it
Stack Overflow users link it when answering why deleting directories, or renaming and replacing files, fails or is not atomic on Windows.
Recommended by 4 people across 6 different months, 2016 to 2023.
What it is
A one-hour CppCon 2015 talk by Niall Douglas on race conditions in the file system, a hazard most programmers overlook while guarding memory with mutexes. It covers portable idioms for avoiding file system races, including on networked Samba shares, and introduces the proposed Boost.AFIO library.
What people said
Strong rec Adrian McCarthy · 2018-03 · Stack Overflow · on “How to prevent my function from delayed deleting in RemoveDirectory() WINAPI?”
The Windows filesystem is not like a Posix one, and this code can still fail because of races. This exact situation is explained in this excellent talk: [this video]
Recommends EmDroid · 2016-02 · Stack Overflow · on “removing directories and all sub-directories in it”
Is this under Windows? In Windows in particular the problem is that if you remove a file or subfolder, it is not removed immediately but asynchronously "sometime in the future". This can take even quite a long time, especially if someone still has some of the file handles open. And even if not, if your program is "too fast" (or the filesystem under heavy load), when you try to remove the parent folder, the contents might not have been actually removed yet, and the call fails. Commonly this is solved by moving each sub-file and sub-folder to some temporary location (on the same partition!), and then deleting it. That way the parent folder will be empty and deletable, even though the contents
Recommends Adrian McCarthy · 2016-09 · Stack Overflow · on “How to remove directory in Windows synchronous”
This video [this video] by Douglas Niall at the 2015 CppCon covers the solution in detail, starting at about 7:30. The idea is to first rename (move) the file or directory to another place (on the same volume), which happens synchronously, and then delete it, which happens asynchronously. Consider this tree: C:\Users\me\ foo\ bar\ obsolete.txt If you try to remove bar after deleting obsolete.txt , it may fail because there can be a delay before obsolete.txt is really deleted. Instead suppose you first move obsolete.txt to C:\Users\me , and give it a temporary name to ensure it doesn't collide with another obsolete.txt in the directory. Maybe you prefix it with a GUID, like 2DCD7863-456C-4B6C-AD84
Recommends Adrian McCarthy · 2017-02 · Stack Overflow · on “GETTING THE ERROR IN VISUAL STUDIO: error C2664: '_chmod' : cannot convert parameter 1 from 'wchar_t [260]' to 'const char *'”
Also note that this recursive approach, which works fine on POSIX systems, won't work reliably on Windows. When `DeleteFile` returns, the file may not yet actually be deleted, so when you then try to remove its directory, that can fail because the directory isn't empty (yet). For details, see [this video] (relevant portion starts about 7:35).
Recommends Adrian McCarthy · 2018-03 · Stack Overflow · on “How to delete a folder in C++?”
It looks like this solution can fail due to filesystem races: DeleteFile is not atomic, which means deleting the directory that contained it can fail because the directory isn't (yet) empty. This talk explains the problem in detail and gives a safer way to delete a directory/tree on Windows: [this video]
Recommends mklement0 · 2018-11 · Stack Overflow · on “Cannot remove item, The Directory is not empty”
Update : Starting with (at least [1] ) Windows 10 version 20H2 (I don't know that Windows Server version and build that corresponds to; run winver.exe to check your version and build), the DeleteFile [LINK] Windows API function now exhibits synchronous behavior on supported file-systems, including NTFS, which implicitly solves the problems with PowerShell's Remove-Item and .NET's System.IO.File.Delete / System.IO.Directory.Delete (but, curiously, not with cmd.exe 's rd /s ). This is ultimately only a timing issue : the last handle to a subdirectory may not be closed yet at the time an attempt is made to the delete the parent directory - and this is a fundamental problem , not restricted to h
Recommends dyp · 2023-11 · Stack Overflow · on “Moving a file using SetFileInformationByHandle”
I'd recommend watching the talk [this video] if you haven't done so already. The issue is that you pass a path as a string to the kernel, and the kernel has to resolve this to an actual file. If you want to ensure that a particular file is deleted, you have to pass the kernel a _handle_ to that file. If you pass a string path instead, you can't ensure that the kernel will operate on the file you had in mind.
Cites Violet Giraffe · 2019-08 · Stack Overflow · on “Is an atomic file rename (with overwrite) possible on Windows?”
The MSDN documentation avoids clearly stating which APIs are atomic and which are not, but Niall Douglas states in his Cppcon 2015 talk [this video] that the only atomic function is SetFileInformationByHandle [LINK] with FILE_RENAME_INFO.ReplaceIfExists set to true. It's available starting with Windows Vista / 2008 Server. Niall is the author of a highly complicated LLFIO library [LINK] and is an expert in file system race conditions so I believe if you're writing an algorithm where atomicity is crucial, better be safe than sorry and use the suggested function even though nothing in ReplaceFile 's description states it's not atomic.
Cites Holli · 2019-09 · Stack Overflow · on “Deleting directory trees in a OS-independent way in Perl 6”
I never really thought about it but apparently the problem is non trivial. [this video] There is a C library named boost which offers platform independent file and directory functions. You could get your feet wet with NativeCall and write a binding for that.I'm sure it's a more pleasant experience than dealing with the windows API.