Repository navigation
Environment markers with different sources for a dependency #7300
Description
-
Poetry version: 1.4.0.dev0, commit ef89e90fc1305e07a62e9e715c91a66c8d7425f7, also happens on 1.3.1
-
Python version: 3.10.9
-
OS version and name: macOS 12.6 (21G115)
-
pyproject.toml: https://github.com/kilnbuild/pythonbuild_examples/blob/main/poetry/pyproject.toml
-
I am on the latest stable Poetry version, installed using a recommended method.
-
I have searched the issues of this repo and believe that this is not a duplicate.
-
I have consulted the FAQ and blog for any relevant entries or release notes.
-
If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption) and have included the output below.
Issue
I'm trying to use two different sources in Poetry for a dependency using markers:
- Apple Silicon
markers="sys_platform == 'darwin' and platform_machine == 'arm64'" - Fallback (
markers="sys_platform != 'darwin' and platform_machine != 'arm64'")
This is because I'm hosting wheels on a source for Apple Silicon, and want to fallback to PyPI for everything else. grpcio for example does not provide wheels for Apple Silicon, so I'm hosting them on a soon-to-be public source (though you can use it now if you want :-)).
If I have a custom mirror for a marker, let's call it "pythonbuild", then can I fall back to PyPI for the other marker, is this supported in Poetry?
Example:
grpcio = [
{version="==1.47.2", markers="sys_platform != 'darwin' and platform_machine != 'arm64'", source="pypi"},
{version="==1.47.2", markers="sys_platform == 'darwin' and platform_machine == 'arm64'", source="pythonbuild"},
]Doing a poetry update -vvv on MacOS ARM64 (using Poetry from master, as of 2022-01-05, commit ef89e90fc1305e07a62e9e715c91a66c8d7425f7), I get the following output (Gist).
Doing this builds the following lock file: lockfile.
During the update process this works, as it looks at the markers and uses the correct source for the dependency, but when I try to install the lock file, it installs from pypi, as that's listed higher in the lock file. That's how it should work in theory, but there should be a way to tell the lockfile which source to use for a dependency.
You can see an example project here that you can use: https://github.com/kilnbuild/pythonbuild_examples/tree/main/poetry (use poetry install on MacOS ARM64 (Apple Silicon, M1/M2 whatever buzzword Apple is using now) to see the issue).
I've read through the following issues:
And this PR:
#6679
And my source looks like this:
[[tool.poetry.source]]
name = "pythonbuild"
url = "https://python.build/simple/"
# tried with true/false for secondary, same result.
default = false
secondary = falseBefore each step, I ensure that I'm starting with a clean environment:
rm -rf ~/Library/Caches/pypoetry .venv poetry.lockI can verify the wheel is being used when using poetry update:
cat .venv/lib/python3.10/site-packages/grpcio-1.47.2.dist-info/WHEELShows
Wheel-Version: 1.0
Generator: bdist_wheel (0.38.4)
Root-Is-Purelib: false
Tag: cp310-cp310-macosx_11_0_arm64
So I should be able to run poetry install now, and have the same results, using the lock file..
rm -rf ~/Library/Caches/pypoetry .venvI can see via network monitoring, it hits https://pypi.org/pypi/grpcio/1.47.2/json for both the update + install, but during the update it downloads from https://wheels.python.build/grpcio-1.47.2-cp310-cp310-macosx_11_0_arm64.whl, where as during the install it does not.
Activity
neersighted commentedon Jan 5, 2023
PTAL @radoering
dimbleby commentedon Jan 5, 2023
update and install aren't very different, it doesn't make a lot of sense that they would install from different places
Are you sure you're not just seeing poetry using the cached wheel - as obtained during update - when it does the install?
You've made it rather hard to tell what's going on by making poetry choose betwen identical wheels in different places - I can't see a reason that anyone would do that in real life.
If you can construct a more plausible scenario in which the two cases give actually different results, then probably it would be clearer whether this is behaving correctly or not
joshuataylor commentedon Jan 6, 2023
I'm removing the cache between runs. So if I was getting a cached wheel, wouldn't it be from the update?
It's not identical wheels. It's a wheel built for MacOS ARM64 on one source, and falling back to PyPI.
That's what this does:
If platform is darwin and machine is arm64, use source
pythonbuild.If platform is not darwin and platform is not arm64, use source
pypi.The reason this is useful is because compiling Python packages on MacOS is a painful experience, especially when they require different packages, or you have multiple projects and need to compile each time.
radoering commentedon Jan 6, 2023
Maybe, I missed something, but I didn't read what happened during
poetry install. IIUC,poetry updateinstalls the expected wheel (verified by the content of the dist-info/WHEEL file). That specific wheel is only available onpythonbuildbut not onpypi, isn't it? What happens when runningpoetry install(after removing the virtual environment)? Is another wheel installed? What's the content of the dist-info/WHEEL file? Or does the command fail?By the way,
"sys_platform != 'darwin' and platform_machine != 'arm64'"is not the inverse of"sys_platform == 'darwin' and platform_machine == 'arm64'". It should probably be"sys_platform != 'darwin' or platform_machine != 'arm64'". However, I don't expect this to be related to the issue.dimbleby commentedon Jun 4, 2023
was always hard to tell what was being reported here: python.build is now dead and so I suppose we'll likely never get anywhere with this.
recommend closing.
github-actions commentedon Feb 29, 2024
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.