PyPerformance benchmarks broken on Windows

[gvanrossum]

With the latest PyPerformance (from its main branch at python/pyperformance@098ffc9), on Windows, with the latest CPython main, these benchmarks are failing:

• chameleon
• crypto_pyaes
• django_template
• dulwich_log
• genshi
• html5lib
• mako
• regex_compile
• sqlalchemy_declarative
• sqlalchemy_imperative
• sympy
• tornado_http

I'll try to go into more detail below.

[gvanrossum]

Genshi installs properly but gives this error when running:

Traceback (most recent call last):
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\site-packages\pyperformance\data-files\benchmarks\bm_genshi\run_benchmark.py", line 7, in <module>
    from genshi.template import MarkupTemplate, NewTextTemplate
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'genshi'

[gvanrossum]

When I run the offending command at the REPL I get a perhaps more informative (certainly longer) traceback:

>>> from genshi.template import MarkupTemplate, NewTextTemplate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\gvanrossum\bench\venv\cpython3.11-5d61ebee4deb-compat-f6a835d45d46\Lib\site-packages\genshi\template\__init__.py", line 20, in <module>
    from genshi.template.markup import MarkupTemplate
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\bench\venv\cpython3.11-5d61ebee4deb-compat-f6a835d45d46\Lib\site-packages\genshi\template\markup.py", line 25, in <module>
    from genshi.template.interpolation import interpolate
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\bench\venv\cpython3.11-5d61ebee4deb-compat-f6a835d45d46\Lib\site-packages\genshi\template\interpolation.py", line 33, in <module>
    token_re = re.compile('%s|%s(?s)' % (
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\cpython\Lib\re.py", line 232, in compile
    return _compile(pattern, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\cpython\Lib\re.py", line 284, in _compile
    p = sre_compile.compile(pattern, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\cpython\Lib\sre_compile.py", line 780, in compile
    p = sre_parse.parse(p, flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\cpython\Lib\sre_parse.py", line 963, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\cpython\Lib\sre_parse.py", line 447, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\cpython\Lib\sre_parse.py", line 822, in _parse
    raise source.error('global flags not at the start '
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
re.error: global flags not at the start of the expression at position 994

I don't know the first thing about genshi. But it looks like either it is hitting a bug in re or it is messing with it in a weird way.

According to the debugger, the pattern being compiled is this (this is the repr()):

'[uU]?[rR]?("""|\\\'\\\'\\\')((?<!\\\\)\\\\\\1|.)*?\\1|[ \\f\\t]*((\\\\\\r?\\n|\\Z|#[^\\r\\n]*|((|Br|F|fr|rb|b|Fr|bR|R|Rf|B|BR|FR|RF|Rb|rF|fR|br|u|r|rB|RB|f|U|rf)\'\'\'|(|Br|F|fr|rb|b|Fr|bR|R|Rf|B|BR|FR|RF|Rb|rF|fR|br|u|r|rB|RB|f|U|rf)"""))|(([0-9](?:_?[0-9])*[jJ]|(([0-9](?:_?[0-9])*\\.(?:[0-9](?:_?[0-9])*)?|\\.[0-9](?:_?[0-9])*)([eE][-+]?[0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*[eE][-+]?[0-9](?:_?[0-9])*)[jJ])|(([0-9](?:_?[0-9])*\\.(?:[0-9](?:_?[0-9])*)?|\\.[0-9](?:_?[0-9])*)([eE][-+]?[0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*[eE][-+]?[0-9](?:_?[0-9])*)|(0[xX](?:_?[0-9a-fA-F])+|0[bB](?:_?[01])+|0[oO](?:_?[0-7])+|(?:0(?:_?0)*|[1-9](?:_?[0-9])*)))|(\\r?\\n|(\\~|\\}|\\|=|\\||\\{|\\^=|\\^|\\]|\\[|@=|@|>>=|>>|>=|>|==|=|<=|<<=|<<|<|;|:=|:|/=|//=|//|/|\\.\\.\\.|\\.|\\->|\\-=|\\-|,|\\+=|\\+|\\*=|\\*\\*=|\\*\\*|\\*|\\)|\\(|\\&=|\\&|%=|%|!=))|((|Br|F|fr|rb|b|Fr|bR|R|Rf|B|BR|FR|RF|Rb|rF|fR|br|u|r|rB|RB|f|U|rf)\'[^\\n\'\\\\]*(?:\\\\.[^\\n\'\\\\]*)*(\'|\\\\\\r?\\n)|(|Br|F|fr|rb|b|Fr|bR|R|Rf|B|BR|FR|RF|Rb|rF|fR|br|u|r|rB|RB|f|U|rf)"[^\\n"\\\\]*(?:\\\\.[^\\n"\\\\]*)*("|\\\\\\r?\\n))|\\w+)(?s)'

[gvanrossum]

Okay, this seems to be caused by a re feature that was deprecated and is now gone. It seems to have to do with bpo-47066, GH-31994. We need to fix this on the Genshi side.

[gvanrossum]

Filed edgewall/genshi#66 -- I have no idea how to fix this.

[gvanrossum]

The regex_compile failure is odd. I see this traceback:

Traceback (most recent call last):
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\site-packages\pyperformance\data-files\benchmarks\bm_regex_compile\run_benchmark.py", line 68, in <module>
    regexes = capture_regexes()
              ^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\site-packages\pyperformance\data-files\benchmarks\bm_regex_compile\run_benchmark.py", line 39, in capture_regexes
    import bm_regex_effbot
    ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\site-packages\pyperformance\data-files\benchmarks\bm_regex_compile\bm_regex_effbot.py", line 1
    ../bm_regex_effbot/run_benchmark.py
    ^
SyntaxError: invalid syntax

I suspect that's a file containing a symbolic link. @ericsnowcurrently Since symlinks don't work reliably on Windows, can we just replace that with the contents of the target file?

[gvanrossum]

For chameleon I get a simple traceback:

Traceback (most recent call last):
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\site-packages\pyperformance\data-files\benchmarks\bm_chameleon\run_benchmark.py", line 5, in <module>
    from chameleon import PageTemplate
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'chameleon'

However, when I manually start the Python executable from the (shared) venv and type that same line from chameleon import PageTemplate it works fine.

@ericsnowcurrently Can you look into this? Maybe something's wrong with the venv and possibly whatever is wrong is also wrong on Linux? Otherwise are there things you'd like me to try to debug this? I'm stumped.

[gvanrossum]

FWIW I cannot repro the chameleon failure on macOS, so I suppose it's Windows specific. But I am quite stuck.

[gvanrossum]

Fixed the regex_compile issue in python/pyperformance@aa2360d.

[gvanrossum]

The mako test fails with a similar failure as chameleon -- the import mako fails with ModuleNotFoundError: No module named 'mako'. There were no errors in the install, and manually running the Python from the venv and importing mako succeeds.

[gvanrossum]

Similar for django_template and crypto_pyaes. I feel there's something fundamentally broken on Windows whenever a benchmark depends on an installed package.