std::fs::create_dir_all will stack overflow with big paths

[xTachyon]

I tried this code:

std::fs::create_dir_all(big_path);

I expected to see this happen: doesn't stack overflow

Instead, this happened:
std::fs::create_dir_all with a big path with many uncreated directories will stack overflow on Windows. The max path is 32767, which can technically result in ~16383 recursive calls, which will stack overflow the 1mb default stack on Windows.

Meta

rustc --version --verbose:

rustc 1.77.1 (7cf61ebde 2024-03-27)
binary: rustc
commit-hash: 7cf61ebde7b22796c69757901dd346d0fe70bd97
commit-date: 2024-03-27
host: x86_64-pc-windows-msvc
release: 1.77.1
LLVM version: 17.0.6

[ChrisDenton]

I can see two ways of fixing this. Change this so it's not recursive;

rust/library/std/src/fs.rs

Lines 2601 to 2613 in 40dcd79

	fn create_dir_all(&self, path: &Path) -> io::Result<()> {
	if path == Path::new("") {
	return Ok(());
	}
	match self.inner.mkdir(path) {
	Ok(()) => return Ok(()),
	Err(ref e) if e.kind() == io::ErrorKind::NotFound => {}
	Err(_) if path.is_dir() => return Ok(()),
	Err(e) => return Err(e),
	}
	match path.parent() {
	Some(p) => self.create_dir_all(p)?,

Or defer to platform-specific implementations so that they can use the most efficient method for their platform.

[xTachyon]

I did strace mkdir -p .. on a Linux and it doesn't seem to do anything fancier than just calling the mkdir syscall with each component.
Doing the equivalent under Windows (but with procmon) shows repeated calls to CreateFile and CloseFile with each component.

So I'm not sure there's a better way to do this in a platform specific way. Looks like the correct solution would be to just do it iteratively.

As a side note, deleting that huge directory tree (that looks like a/a/a/a/.../a) took 38m18.119s 😰 . Even Windows is handling this terribly.

[ChrisDenton]

On that side note, removing it didn't seem that slow on my machine. I kept pushing a to a PathBuf in a loop, using std::fs::create_dir each time to make the directory tree as deep as possible until it errored (this was incredibly slow). Deleting it with std::fs::remove_dir_all on Windows 11 took and about 3.8s on a ReFs drive and about 3 minutes on an NTFS drive.

[xTachyon]

I'm on W10. Maybe W11 got better at doing this? Or the rm command from MSYS2 doesn't play well with how Windows expects things to be done.

[xTachyon]

@rustbot claim

[asder8215]

Hi, is this issue still being worked on? If not, could I tackle this issue? The iterative approach for this looks similar to a pull request I have seen on the Rust coreutils project: https://github.com/uutils/coreutils/pull/8947/files

As of 1.9.0, we see that create_dir_all(...) function is implemented as such:

fn create_dir_all(&self, path: &Path) -> io::Result<()> {
        if path == Path::new("") {
            return Ok(());
        }

        match self.inner.mkdir(path) {
            Ok(()) => return Ok(()),
            Err(ref e) if e.kind() == io::ErrorKind::NotFound => {}
            Err(_) if path.is_dir() => return Ok(()),
            Err(e) => return Err(e),
        }
        match path.parent() {
            Some(p) => self.create_dir_all(p)?,
            None => {
                return Err(io::const_error!(
                    io::ErrorKind::Uncategorized,
                    "failed to create whole tree",
                ));
            }
        }
        match self.inner.mkdir(path) {
            Ok(()) => Ok(()),
            Err(_) if path.is_dir() => Ok(()),
            Err(e) => Err(e),
        }
}

Instead of recursively calling create_dir_all(...), we can implement what naoNao89 is doing in the linked PR, where we iteratively aggregate all the parent paths (checking for special cases like ".." and ".") into a VecDeque (using it in a stack-like manner so we don't need to call vec.reverse() like how it's done in the PR) and if the tree looks good, we can mkdir on each of those paths. How does that sound?

[xTachyon]

I honestly forgot so hard about this issue, that I did not even remember I made it.

You're free to solve it. I don't know how I unclaim it.