Interpreter is not found when using Anaconda

[Ptrskay3]

🐛 Bug Reports

When reporting a bug, please provide the following information. If this is not a bug report you can just discard this template.

🌍 Environment

• Your operating system and version: Windows 10
• Your python version: 3.8.5
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: Anaconda default Python installation
• Your Rust version (rustc --version): 1.53.0-nightly
• Your PyO3 version: 0.13.2
• Have you tried using latest PyO3 main (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: Yes

💥 Reproducing

Please provide a minimal working example. This means both the Rust code and the Python.
The following code compiles, but crashes when using Anaconda distribution (using the default installation).

use pyo3::prelude::*;

fn main() -> PyResult<()>{
    let gil = Python::acquire_gil();
    let py = gil.python();

    let result = py.run("print('it works')", None, None);

    if let Err(ref err) = result {
        println!("{:?}", err);
    }
    Ok(())
}

The output I got:

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\bug\\target\\debug\\bug.exe'
  sys.base_prefix = 'C:\\Users\\balog\\anaconda3'
  sys.base_exec_prefix = 'C:\\Users\\balog\\anaconda3'
  sys.executable = 'C:\\bug\\target\\debug\\bug.exe'
  sys.prefix = 'C:\\Users\\balog\\anaconda3'
  sys.exec_prefix = 'C:\\Users\\balog\\anaconda3'
  sys.path = [
    'C:\\Users\\balog\\anaconda3\\python38.zip',
    '.\\DLLs',
    '.\\lib',
    'C:\\bug\\target\\debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000e6c (most recent call first):
<no Python frame>
error: process didn't exit successfully: `target\debug\bug.exe` (exit code: 1)

The sys executables are pointing to wrong locations, so it seems it couldn't find the Python interpreter. I haven't checked how PyO3 tries to find the interpreter internally, but as far as I know Anaconda doesn't set PYTHONPATH nor PYTHONHOME env variables and that might be the cause that breaks it. Anaconda sets up the CONDA_PREFIX, which can be - at least in my case - set as PYTHONHOME, so it's my temporal workaround:

use pyo3::ffi;
use pyo3::prelude::*;
use widestring::WideCString;

fn main() -> PyResult<()> {
    if let Some(PYTHONHOME) = std::env::var_os("CONDA_PREFIX") {
        unsafe {
            ffi::Py_SetPythonHome(
                WideCString::from_str(PYTHONHOME.to_str().unwrap())
                    .unwrap()
                    .as_ptr(),
            );
        }
    }

    let gil = Python::acquire_gil();
    let py = gil.python();

    let result = py.run("print('it works')", None, None);

    if let Err(ref err) = result {
        println!("{:?}", err);
    }
    Ok(())
}

However I'm not sure it works out-of-the-box.

The output of conda info -a:

     active environment : base
    active env location : C:\Users\balog\anaconda3
            shell level : 1
       user config file : C:\Users\balog\.condarc
 populated config files : 
          conda version : 4.9.2
    conda-build version : 3.20.5
         python version : 3.8.5.final.0
       virtual packages : __cuda=9.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\balog\anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\balog\anaconda3\pkgs
                          C:\Users\balog\.conda\pkgs
                          C:\Users\balog\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\balog\anaconda3\envs
                          C:\Users\balog\.conda\envs
                          C:\Users\balog\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Windows/10 Windows/10.0.19041
          administrator : False
             netrc file : None
           offline mode : False

# conda environments:
#
base                  *  C:\Users\balog\anaconda3

sys.version: 3.8.5 (default, Sep  3 2020, 21:29:08) [...
sys.prefix: C:\Users\balog\anaconda3
sys.executable: C:\Users\balog\anaconda3\python.exe
conda location: C:\Users\balog\anaconda3\lib\site-packages\conda
conda-build: C:\Users\balog\anaconda3\Scripts\conda-build.exe
conda-convert: C:\Users\balog\anaconda3\Scripts\conda-convert.exe
conda-debug: C:\Users\balog\anaconda3\Scripts\conda-debug.exe
conda-develop: C:\Users\balog\anaconda3\Scripts\conda-develop.exe
conda-env: C:\Users\balog\anaconda3\Scripts\conda-env.exe
conda-index: C:\Users\balog\anaconda3\Scripts\conda-index.exe
conda-inspect: C:\Users\balog\anaconda3\Scripts\conda-inspect.exe
conda-metapackage: C:\Users\balog\anaconda3\Scripts\conda-metapackage.exe
conda-render: C:\Users\balog\anaconda3\Scripts\conda-render.exe
conda-server: C:\Users\balog\anaconda3\Scripts\conda-server.exe
conda-skeleton: C:\Users\balog\anaconda3\Scripts\conda-skeleton.exe
conda-verify: C:\Users\balog\anaconda3\Scripts\conda-verify.exe
user site dirs: 

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: base
CONDA_EXE: C:\Users\balog\anaconda3\condabin\..\Scripts\conda.exe
CONDA_EXES: "C:\Users\balog\anaconda3\condabin\..\Scripts\conda.exe"  
CONDA_PREFIX: C:\Users\balog\anaconda3
CONDA_PROMPT_MODIFIER: (base) 
CONDA_PYTHON_EXE: C:\Users\balog\anaconda3\python.exe
CONDA_ROOT: C:\Users\balog\anaconda3
CONDA_SHLVL: 1
CURL_CA_BUNDLE: <not set>
HOMEPATH: \Users\balog
PATH: C:\Users\balog\anaconda3;C:\Users\balog\anaconda3\Library\mingw-w64\bin;C:\Users\balog\anaconda3\Library\usr\bin;C:\Users\balog\anaconda3\Library\bin;C:\Users\balog\anaconda3\Scripts;C:\Users\balog\anaconda3\bin;C:\Users\balog\anaconda3;C:\Users\balog\anaconda3\Library\mingw-w64\bin;C:\Users\balog\anaconda3\Library\usr\bin;C:\Users\balog\anaconda3\Library\bin;C:\Users\balog\anaconda3\Scripts;C:\Users\balog\anaconda3\bin;C:\Users\balog\anaconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\pysprint-cli\bin;C:\Users\balog\.cargo\bin;C:\Users\balog\AppData\Local\Microsoft\WindowsApps;C:\Users\balog\anaconda3\python.exe;.
PSMODULEPATH: C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>

[davidhewitt]

Thanks for the report.

This looks like an issue in Anaconda to me. Searching the issue tracker I found ContinuumIO/anaconda-issues#11439

That issue was only tested on unix, but it looks like exactly the same problem as reported here.

[cgranade]

If you're still running into this, I found a workaround for Windows by calling into the Python FFI to preconfigure the home directory before the Python runtime is initialized. I apologize I'm still a bit new to Rust, so this may not be the nicest way to do things, but it does seem to work at allowing PyO3 to embed Python from conda environments:

pub fn initialize_python() -> PyResult<()> {
    // Due to https://github.com/ContinuumIO/anaconda-issues/issues/11439,
    // we first need to set PYTHONHOME. To do so, we will look for whatever
    // directory on PATH currently has python.exe.
    let python_exe = which::which("python").unwrap();
    let python_home = python_exe.parent().unwrap();

    // The Python C API uses null-terminated UTF-16 strings, so we need to
    // encode the path into that format here.
    // We could use the Windows FFI modules provided in the standard library,
    // but we want this to work cross-platform, so we do things more manually.
    let mut python_home = python_home.to_str().unwrap().encode_utf16().collect::<Vec<u16>>();
    python_home.push(0);
    unsafe {
        pyo3::ffi::Py_SetPythonHome(python_home.as_ptr());
    }

    // Once we've set the configuration we need, we can go on and manually
    // initialize PyO3.
    pyo3::prepare_freethreaded_python();

    Ok(())
}

[Japanuspus]

Root cause seems to be #1871. A robust workaround is to set PYO3_PYTHON to python so that resolving the python executable is handled by the OS (where the environment is active).

[k-wozniak]

@Japanuspus could you please explain what you mean by setting PYO3_PYTHON to python?
I've tried creating Windows System variable PYO3_PYTHON and setting it to C:\Users\user\Anaconda3\python.exe, setting PYTHONHOME AND PYTHONPATH variables etc. but nothing works.

It works complete fine on Linux and Windows without anaconda.

Thanks for help!

[Japanuspus]

@k-wozniak

@Japanuspus could you please explain what you mean by setting PYO3_PYTHON to python? I've tried creating Windows System variable PYO3_PYTHON and setting it to C:\Users\user\Anaconda3\python.exe, setting PYTHONHOME AND PYTHONPATH variables etc. but nothing works.

No longer on Windows, so cannot test, but the idea was to literally set the environment variable PYO3_PYTHON to python i.e., with no explicit path. That way the correct python active in the conda environment will be called...