__PYVENV_LAUNCHER__ being set creates unexpected problems for child Python instances

[daisylb]

xonfig

Details

❯ xonfig
+------------------+---------------------+
| xonsh            | 0.9.18              |
| Git SHA          | 26475146            |
| Commit Date      | May 5 06:42:28 2020 |
| Python           | 3.8.2               |
| PLY              | 3.11                |
| have readline    | True                |
| prompt toolkit   | 3.0.5               |
| shell type       | prompt_toolkit      |
| pygments         | None                |
| on posix         | True                |
| on linux         | False               |
| on darwin        | True                |
| on windows       | False               |
| on cygwin        | False               |
| on msys2         | False               |
| is superuser     | False               |
| default encoding | utf-8               |
| xonsh encoding   | utf-8               |
| encoding errors  | surrogateescape     |
+------------------+---------------------+

Expected Behavior

When I run python -m venv from within Xonsh, the virtualenv created should be based on the Python I'm invoking, not the Python that Xonsh is running in.

Current Behavior

The virtualenv created is based on the Python that Xonsh is running in.

If I put del $__PYENV_LAUNCHER__ in my .xonshrc, everything works as expected.

I think this may also be the cause of issues I've been having getting Pyenv to compile Python, but I've yet to test this.

Steps to Reproduce

1. Install Xonsh using pipx, (under, say, Python 3.8).
2. Install a different Python version (say, Python 3.7).
3. Use python -m venv to make a virtualenv with the latter.

[daisylb]

Sorry, I bumped Enter by accident, editing the actual details into the issue now.

[daisylb]

In my digging to discover what was going on, I discovered that Python 3's venv creates symlinks to sys._base_executable, which is normally the same as sys.executable but can be overridden by the __PYENV_LAUNCHER__ environment variable.

[AstraLuma]

This sounds like it's installation/configuration specific.

What is which python?

[daisylb]

❯ which python
/Users/leigh/Library/asdf/shims/python

However, if I bypass the asdf shims and invoke the Python binary directly, I get the same behaviour.

Details

❯ ~/Library/asdf/installs/python/3.7.7/bin/python -V
Python 3.7.7
❯ ~/Library/asdf/installs/python/3.7.7/bin/python -m venv asdf-test
❯ asdf-test/bin/python -V
Python 3.8.2
❯ ~/.pyenv/versions/3.7.7/bin/python -V
Python 3.7.7
❯ ~/.pyenv/versions/3.7.7/bin/python -m venv pyenv-test
❯ pyenv-test/bin/python -V
Python 3.8.2

In zsh, without any modifications, I get the expected behaviour:

Details

leigh@sylveon tmp.flIEx7dz % ~/Library/asdf/installs/python/3.7.7/bin/python -V
Python 3.7.7
leigh@sylveon tmp.flIEx7dz % ~/Library/asdf/installs/python/3.7.7/bin/python -m venv asdf-test
leigh@sylveon tmp.flIEx7dz % asdf-test/bin/python -V
Python 3.7.7
leigh@sylveon tmp.flIEx7dz % ~/.pyenv/versions/3.7.7/bin/python -V
Python 3.7.7
leigh@sylveon tmp.flIEx7dz % ~/.pyenv/versions/3.7.7/bin/python -m venv pyenv-test
leigh@sylveon tmp.flIEx7dz % pyenv-test/bin/python -V
Python 3.7.7

if I remove my .xonshrc totally to try to run Xonsh as it comes out of the box, I still get the buggy result, so it's not a xontrib or other customisation changing things.

Details

leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ~/Library/asdf/installs/python/3.7.7/bin/python -V
Python 3.7.7
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ~/Library/asdf/installs/python/3.7.7/bin/python -m venv
 asdf-test
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ asdf-test/bin/python -V
Python 3.8.2
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ~/.pyenv/versions/3.7.7/bin/python -V
Python 3.7.7
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ~/.pyenv/versions/3.7.7/bin/python -m venv pyenv-test
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ pyenv-test/bin/python -V
Python 3.8.2

In zsh, __PYVENV_LAUNCHER__ is unset, but it is set in Xonsh:

Details

leigh@sylveon tmp.2xGaKj62 % echo $__PYVENV_LAUNCHER__

leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ $__PYVENV_LAUNCHER__
'/Users/leigh/.local/pipx/venvs/xonsh/bin/python'

When created under zsh, the python symlink in my venv points to the Python I used to create it, as expected. When created under Xonsh, it points to the Python Xonsh is installed in (which it's getting from the __PYVENV_LAUNCHER__ environment variable):

Details

leigh@sylveon tmp.2xGaKj62 % ls -l pyenv-test/bin/python
lrwxr-xr-x  1 leigh  staff  45 14 May 14:37 pyenv-test/bin/python -> /Users/leigh/.pyenv/versions/3.7.7/bin/python

leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ls -l pyenv-test/bin/python
lrwxr-xr-x  1 leigh  staff  47 14 May 14:36 pyenv-test/bin/python -> /Users/leigh/.local/pipx/venvs/xonsh/bin/python

If I delete __PYENV_LAUNCHER__ from the environment, everything works as expected again:

Details

leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ del $__PYVENV_LAUNCHER__
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ~/.pyenv/versions/3.7.7/bin/python -m venv pyenv-test2
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ pyenv-test2/bin/python -V
Python 3.7.7
leigh@sylveon /var/folders/0h/v25x4ks55g1bb83jvj4jjltc0000gn/T/tmp.ZpNj36Ke $ ls -l pyenv-test2/bin/python
lrwxr-xr-x  1 leigh  staff  45 14 May 14:41 pyenv-test2/bin/python -> /Users/leigh/.pyenv/versions/3.7.7/bin/python

[AstraLuma]

This doesn't sound like a general xonsh problem. It sounds like a potential quirk for how you installed and are launching xonsh.

There's two fixes I see:

• add del $__PYENV_LAUNCHER__ to .xonshrc
• Create a trampoline script that does the same thing.

Without knowing more about when that variable gets used and why, I can't really say that something should be changed in xonsh itself.

[daisylb]

__PYVENV_LAUNCHER__ (not __PYENV_LAUNCHER__, although I keep making that mistake as well!) is an implementation detail of how the venv module works in macOS. From what I can tell it's set by a wrapper around the CPython interpreter and used in interpreter startup.

New versions of Python will unset it after interpreter startup so Python code doesn't see it; see python/cpython#9516. In the related bug report there is a comment describing the behaviour I'm seeing with creating virtualenvs, and miscellaneous other comments describing other issues it causes.

I've tested it in the Python 3 that comes with macOS Catalina, as well as Python 3.7.7 and Python 3.8.2 installed via Homebrew; all three also have the environment variable set in an out-of-the-box Xonsh install, so it's not limited to pyenv.

Details

leigh@Sylveon-2 ~ % /usr/bin/python3 -m xonsh

******************************************************************************
libedit detected - readline will not be well behaved, including but not limited to:
   * crashes on tab completion
   * incorrect history navigation
   * corrupting long-lines
   * failure to wrap or indent lines properly

It is highly recommended that you install gnureadline, which is installable with:
     xpip install gnureadline
******************************************************************************

                                                   Welcome to the xonsh shell (0.9.18)

                                ~ Ever wonder why there isn't a Taco Shell? Because it is a corny idea. ~

------------------------------------------------------------------------------------------------------------------------------------------
xonfig tutorial    ->    Launch the tutorial in the browser
xonfig web         ->    Run the configuration tool in the browser and claim your shell
(Note: Run the configuration tool or create a ~/.xonshrc file to suppress the welcome screen)

leigh@Sylveon-2 ~ $ $__PYVENV_LAUNCHER__
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'

leigh@Sylveon-2 ~ % /usr/local/Cellar/python/3.7.7/bin/python3 -m xonsh

                                                   Welcome to the xonsh shell (0.9.18)

                                                       ~ Break out of your shell ~

------------------------------------------------------------------------------------------------------------------------------------------
xonfig tutorial    ->    Launch the tutorial in the browser
xonfig web         ->    Run the configuration tool in the browser and claim your shell
(Note: Run the configuration tool or create a ~/.xonshrc file to suppress the welcome screen)

leigh@Sylveon-2 ~ $ $__PYVENV_LAUNCHER__
'/usr/local/Cellar/python/3.7.7/bin/python3'

leigh@Sylveon-2 ~ % /usr/local/Cellar/python@3.8/3.8.2/bin/python3 -m xonsh

                                                   Welcome to the xonsh shell (0.9.18)

                                   ~ Nice indeed. Everything's accounted for, except your old shell. ~

------------------------------------------------------------------------------------------------------------------------------------------
xonfig tutorial    ->    Launch the tutorial in the browser
xonfig web         ->    Run the configuration tool in the browser and claim your shell
(Note: Run the configuration tool or create a ~/.xonshrc file to suppress the welcome screen)

leigh@Sylveon-2 ~ $ $__PYVENV_LAUNCHER__
'/usr/local/Cellar/python@3.8/3.8.2/bin/python3'

So from a purely semantic perspective it's not a bug in Xonsh, but rather a CPython bug which is fixed upstream. But CPython versions with the bug will still be around for a while, and the vast majority of macOS users will be using the built-in Python or one installed from either Homebrew or Pyenv, which are all affected.

Plus, the issues it causes are subtle and difficult to debug; I spent more time than I'd like wading through the internals of venv and CPython's C source before I figured out that __PYVENV_LAUNCHER__ was the culprit.

(I also had a number of other issues that I'm now suspecting might be related too, like CPython itself and Postgres both failing to build from source when run from Xonsh, but working fine from zsh or fish, to the point where I was considering giving up and going back to Fish.)

For my own purposes, I've already added del $__PYVENV_LAUNCHER__ to my .xonshrc, but IMO Xonsh should unset it by default; its existence is a bug, it doesn't appear to serve any purpose aside from interpreter startup AFAICT, and when it is present it causes multiple different kinds of subtle problems that don't happen out of the box in other shells.

[gforsyth]

Wow, thanks for tracking this down @excitedleigh ! I'm always using conda environments and they don't seem to set the __PYVENV_LAUNCHER__, but I've confirmed that I do see it if I use the Catalina built-in python3.

I agree that we should unset it at startup.

[scopatz]

PRs welcome!

[telamonian]

A fix for this issue (which unsets __PYVENV_LAUNCHER__ as part of the Python startup process) has been added to cpython 3.9.0 and backported to 3.8 and 3.7.

Not sure which version for 3.8 it was first added to, but I know through testing that 3.7.9 has the __PYVENV_LAUNCHER__ removal fix. So you could reasonably close this issue as "fixed upstream".

For my own purposes, I've already added del $__PYVENV_LAUNCHER__ to my .xonshrc, but IMO Xonsh should unset it by default;

Or you could do that, which would be functionally identical to the fix they added to cpython.

[gforsyth]

Sweet. Thanks for the update @telamonian ! I'm going to close this out since it's been fixed upstream.