Investigate generating matching strings for test regexes and adding to testbed

[danmoseley]

The regexes we may port in https://github.com/dotnet/corefx/issues/27618 will exercise our parser but not our runner.

Experiment with using REX (https://www.microsoft.com/en-us/research/project/rex-regular-expression-exploration/) to generate interesting strings for each of those interesting patterns. Then preserve some of those strings in the tests - so we have a large body of new positive tests for regex matching.

We could also generate negative tests, for example by making small changes to the positive matching strings until they did not match, and preserving those also. Those should continue to not-match.

@CyrusNajmabadi @ViktorHofer @maryamariyan

[CyrusNajmabadi]

If you add/generate any tests, let me know. I'll add them to the roslyn test suite.

[danmoseley]

@CyrusNajmabadi these would be matching tests not parsing tests. Are those still relevant to you?

[ViktorHofer]

nice!

[CyrusNajmabadi]

Yes. Very relevant. We sitll want to make sure we can successfully parse all legal regex strings that you guys accept.

Basically:

1. If you can parse them, we should be able to as well.
2. If you error while parsing, we should error with the same message.

That can easily be easily tested.

What we also want is: for legal strings, we should be interpretting the input text the same way as you (i.e. what are modifiers, what are comments, what are character classes, etc.). That's harder to validate automatically. But can be done manually.

[danmoseley]

@CyrusNajmabadi I'm not certain we're talking about the same thing. As I understand it your code is all about regex patterns - parsing them successfully or not. Not using them to match. This task is all about generating inputs for those patterns to verify they continue to match or not match as they do today. Are you saying you are interested in testing of executing regexes?

[CyrusNajmabadi]

This task is all about generating inputs for those patterns

Right. i'm saying if you come up with new patterns for "those patterns" that you let me know so i can add tests for those patterns as well.

Thanks!

--

Are you saying you are interested in testing of executing regexes?

Not today. but potentially in the future. Consider, for example, a regex-debugger built into roslyn. We'd want a way to evaluate the users pattern against the input, a step at a time, highlighting what was going on. Right now Regex is too much of a black box to do that. But with the new system i'm building, such a thing would be possible. The concern would mostly be about matching exactly like how you guys match. It would not be good if the Roslyn system showed one thing, but the person got very different results with your system.

I think i maybe saw an interpreter in your code somewhere that i could base my impl on. However, it's tricky given how you build and manipulate your trees in-situ.

[danmoseley]

REX may not be useful: "Rex error: The following constructs are currently not supported: anchors \G, \b, \B, named groups, lookahead, lookbehind, as-few-times-as-possible quantifiers, backreferences, conditional alternation, substitution"

There are plenty of other test sets and generators (even for other regex engines) and other approaches to increase test coverage.

[danmoseley]

Also https://github.com/moodmosaic/Fare ?

[ViktorHofer]

@stephentoub @danmosemsft is this issue still valid? I remember that we ported over a bunch of additional test data, recently. Asking as I know someone who might be interested in helping out.