Developers can enumerate directories and files using globbing patterns

[khellang]

I'd like to start a discussion on including a file system globbing API in .NET (Core). If you look at implementations mentioned on Wikipedia, every "mainstream platform" has an entry, but not .NET.

There's quite a few (more or less) successful implementations around (see below), some even from Microsoft, but I think something as fundamental as this, should ship with the framework.

There's already partial globbing support using the following methods

• Directory.GetFiles
• Directory.EnumerateFiles
• Directory.GetFileSystemEntries
• Directory.EnumerateFileSystemEntries
• Directory.GetDirectories
• Directory.EnumerateDirectories

They all have a searchPattern argument, but it lacks support for recursive globs (** aka. "globstar"), brace expansion etc. This can be achieved using the SearchOption argument, but the API is hard to use when you want to support (often user-defined) recursive patterns like /src/**/*.csproj.

I'd ❤️ to hear people's opinions here...

• Is it worth including in the framework?
• Should new APIs be introduced, or can we "level up" the existing searchPattern in the methods mentioned above (without it being a breaking change)?

Examples

• MSBuild by @cdmihai
• Cake by @patriksvensson
• Minimatch (port from JS) by @SLaks
• Microsoft.Extensions.FileSystemGlobbing by @troydai
• Glob by @kthompson

And tons of other implementations...

[nil4]

This article may come in handy: Glob Matching Can Be Simple And Fast Too

[khellang]

This article may come in handy: Glob Matching Can Be Simple And Fast Too

Heh. Reading that article was exactly what prompted me to file this issue 😂

[daveaglick]

I think this would make a great API addition.

I would like to see the implementation tied to an abstraction and not directly to the CoreFx IO classes. Then a specific implementation of the globbing API could be delivered support the CoreFx IO API while leaving it open for other uses.

For example, Microsoft.Extensions.FileSystemGlobbing does a great job with this because you only have to implement two abstract classes to get it two work (DirectoryInfoBase and FileInfoBase - interfaces would be even nicer though).

Some things I think are missing from Microsoft.Extensions.FileSystemGlobbing and would love to see:

• Pattern expansion (being able to specify multiple pattern alternatives with a single string)
• Pattern exclusion (excluding patterns or pattern expansions with a specific syntax, something like !)

For example, pattern expansion can be represented by braces so you can have patterns like:

/src/**/*.{csproj,cs}

which selects all *.csproj files and all *.cs files in the specified folder.

Likewise, pattern exclusion would look like:

/src/**/*.{!csproj,}

which selects all *.* files (notice the comma without anything specified inside the expansion) except *.csproj in the specified folder.

Both of these concepts are (mostly) implemented for Microsoft.Extensions.FileSystemGlobbing in Reliak.FileSystemGlobbingExtensions. There's also a really good implementation of brace expansion in Minimatch.

FWIW, I've also got an implementation for Microsoft.Extensions.FileSystemGlobbing based on combining features from those two libraries here.

[maartenba]

Yes!

[ThatRendle]

Given the shift to CLI, both for the tooling and as a target for .NET Core apps, this would make perfect sense.

[danmoseley]

Is there some kind of quasi standard written down for globbing behavior? Are forward slashes important (they aren't for native Windows apps like MSBuild, but are for Git on Windows probably just undone work)

Does "**" literally match "everything including slash" and "*" literally matches "everything excluding slash"? I believe that's what we did in MSBuild and I think that's Git behavior also.

Could "**" supported be added to the existing File IO API's without unnacceptable breaks, and is that important? I like the idea of also offering it for use in other contexts, eg., Git obviously does not always run globs over the file system.

[danmoseley]

As for perf, of course it's faster to use the native API where possible. I believe for MSBuild we did something like: used the OS if there were no * after the last slash; otherwise cropped the pattern at the first slash after the first * then used the OS to enumerate all files below it into a list then used a regex on the results to handle any subsequent * and **. Curious what Unix shells, perl etc do. It would be easy to check.

[khellang]

Is there some kind of quasi standard written down for globbing behavior?

I don't think there's any formal specification. I guess the best bet is going by something like minimatch's test suite which seems to be pretty in-line with bash, sh, ksh etc.

There's even some comments on compliance with other fnmatch/glob implementations in their README.md

Are forward slashes important (they aren't for native Windows apps like MSBuild, but are for Git on Windows probably just undone work)

When it comes to forward/backward slashes, I think it makes sense to do what node-glob does, which means only use forward slash for glob patterns

[danmoseley]

I think it makes sense to do what node-glob does, which means only use forward slash for glob patterns

Why is that? I assumed that in Git and Sublime on Windows this was just a hangover from their non-Windows heritage. it's certainly handy to be able to paste in a Windows path and add ** on the end or suchlike.

[khellang]

How do you handle escaping if you handle both forward and backward slashes as separators?

[danmoseley]

@JeremyKuhne this is an IO area we might want to invest in..

[TylerLeonhardt]

I'd also like to point out that

Directory.EnumerateFiles(
    folderPath,
    pattern,
    SearchOption.AllDirectories)

isn't good at best effort attempts to enumerate files.

If a path is too long or if you are unauthorized to access a file, the EnumerateFiles will throw and you won't be able to continue enumerating after that throwing file.

[JeremyKuhne]

Note that we're reviewing an extensibility mechanism for enumeration that will allow building globbing solutions. See #24429.