Poetry is extremely slow when resolving the dependencies

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Centos 7
Poetry version: 1.0.0
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/qiuwei/a0c7eee89e5e8d75edb477858213c30b

Issue

I created an empty project and run poetry add allennlp. It takes ages to resolve the dependencies.

Could this be due to downloading packages from pypi to inspect their dependencies, when not properly specified?

Could this be due to downloading packages from pypi to inspect their dependencies, when not properly specified?

It seems so. I have checked the detailed log, poetry kept retrying to resolve the dependency for botocore, but without success. So I assume that the dependency can be eventually resolved if enough time is given.

However, is there any way to get around this?

BTW, I also consider it's better to give some warning if there are some dependencies are not properly specified and could not be resolved after a number of attempts.

Hi,

I encounter a similiar problem on my MacOS. Python version used is 3.7.6, Poetry is 1.0.5. I just created a new project with no dependencies so far in pyproject.toml, just initially pytest. It takes ages until the new virtualenv is setup with all 11 packages installed.

Running it with -vvv does not bring any new findings.

Regards, Thomas

Yes, i'm running into the same problem. Resolving dependencies takes forever. I tried to use VPN to get through the GFW, nevertheless, it is still not working. I also tried to change pip source and wrote local source in the toml file, neither works. It's driving me nuts.

same here...😱

Same here. I just created an empty project then ran poetry install and it takes so much time to resolve dependencies.

I'm currently using this workaround:

poetry export -f requirements.txt > requirements.txt
python -m pip install -r requirements.txt
poetry install

It takes a lower time space to install the package locally since all deps are already installed.

Make sure to run poetry shell before to access the created virtual environment and install on it instead of on user/global path.

Poetry being slow to resolve dependencies seems to be a reoccuring issue:

Poetry resolving dependencies is amazingly slow #476 - Poetry resolving dependencies is amazingly slow
Resolving dependencies are slow #819 - Resolving dependencies are slow
Poetry update never finishes resolve and Poetry show --outdated hangs #832 - Poetry update never finishes resolve and Poetry show --outdated hangs
Poetry doesn't resolve on MacOs Mojave #1047 - Poetry doesn't resolve on MacOs Mojave

Maybe there is a dependency conflict.

No conflict. Poetry is slow as hell.

@gagarine

First of all, I want to say there is ongoing work to improve the dependency resolution.

However, there is so much Poetry can do with the current state of the Python ecosystem. I invite you to read https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow to know a little more about why the dependency resolution can be slow.

If you report that Poetry is slow, we would appreciate a pyproject.toml that reproduces the issue so we can debug what's going on and if it's on Poetry's end or just the expected behavior.

@gagarine Could you provide the pyproject.toml file you are using?

Take about 2min to resolve dependencies after adding newspaper3k on a fresh project.
Connexion: 40ms ping and 10Mb/s down.

pyproject.toml

[tool.poetry]
name = "datafox"
version = "0.1.0"
description = ""
authors = ["Me <mail@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.8"
newspaper3k = "^0.2.8"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Re:

this fixed my issue: poetry cache clear --all pypi

I've seen this mentioned a few times, if it works that's great, but why would it work? Wouldn't clearing the cache cause things to take longer? What am I missing?

@tall-josh

@tall-josh Because Poetry had (maybe still has?) a bug where corrupted cache entries make Poetry hang forever when trying to resolve dependencies.

For me this seemed to occur if I Ctrl+C'd Poetry while it was doing an install and it was downloading packages.

I observed this on 1.1.14, so perhaps it's fixed in 1.2+.

Lockfile generation hangs indefinitely when cache is corrupted #5922

Clearing the cache is most likely related to clearing out partial/incomplete/corrupted by concurrent usage downloads that can cause an indefinite hang.

add this to .toml file is work for me
[[tool.poetry.source]] name = 'aliyun.mirrors' url = 'https://mirrors.aliyun.com/pypi/simple/' default = true

Still have same problem on version 1.2.1

Hi all,

This issue has gotten quite long and meandering, with many disparate causes, solutions, fixed issues, perhaps still extant bugs, and many "me too" comments all discussed.

I'm going to close this issue as most of the root causes discussed within have either been solved in 1.2 or 1.3 (the changes in 1.3 are behavior changes not eligible for backport).

If you are having issues with Poetry taking a long time to resolve dependencies, please first open a Discussion or start on Discord, as many of them are related to configuration and large search space (basically, you're creating exponential work for the solver and should tighten your constraints). Tooling to advise the user should be possible (if difficult to develop) in the long run, and anyone interested in tackling this hairy problem should reach out to the team via a Discussion or on Discord.

Past that, please make sure to test with the latest code (both on the 1.2 branch and master branch presently) when trying to reproduce resolver issues as we are making improvements all the time, and your issue may be fixed and pending release already.

Finally, good reproductions are needed for this category of issue. Many times they are related to transient network issues, pathologically bad cases due to decisions made around (low traffic, private) custom package indexes, or a corrupted cache/bad local config. Reproducing in a container with publicly available packages will mean that someone can dissect your issue and possibly fix it. If you can't reproduce it with public packages, but you can with private packages, there are still options -- everything from sharing details with the team in private, to creating 'imitation' repositories to reproduce an issue.

Please refrain from commenting on this issue more if it's not a mitigation/common solution -- "me too" is not very helpful and will send dozens of emails, and if you can reproduce bad performance consistently/in a clean environment it should be an issue. If you're stuck and need help, ask for support using one of the methods mentioned above.

@Kache

@Kache, It appears to search through dependencies depth-first, rather than breadth-first. As a result, you've probably got a something earlier in your pyproject.toml that depends on ddtrace, so the dependency resolver grabbed that version and tried to resolve using that, rather than the ddtrace version you've specified.

I've had some success moving the dependencies I want exact version logic prioritizing earlier in the pyproject.toml file.

(I also disabled IPv6, upgraded to poetry 1.2x, and have reduced the possible space for the troubling aws libraries (boto3 and awscli, for me) so those go at the very end of my dependency file and have only a few recent versions to chew through.

I'm seeing dependency resolution time between 5 and 35 seconds most of the time now.

I have a private repository from gitlab which is know to have some problems with ipv6, I disabled ipv6 too and add this to my .ssh/config

Host gitlab.com
    AddressFamily inet

then sudo service ssh restart

Got a huge improvement in dependencies resolver

I found one workaround:
You can specify requirements more precisely. E.g.: not python="^3.10" but python="^3.10.0"

Reducing the search space is not a workaround, but a common issue (as mentioned in my comment above). That being said, the version you have written should be equivalent (^3.10.0 and ^3.10) -- any differences you noticed with that change are likely unrelated.

Actually, they a not equivalent: ^3.10 means >=3.10,<4 but ^3.10.0 means >=3.10,<3.11
It reduces time from >2000s (I didn't wait more) to 200s

That should expand to python_full_version >= "3.10.0" and python_version < "4.0" -- ^ is not defined to operate differently based on precision as far as I am aware. However, we are generating a lot of traffic on a tangential topic -- let's take it to a Discussion or Discord if you think my explanation is insufficient or incorrect.

Navigation Menu

Repository navigation

Poetry is extremely slow when resolving the dependencies #2094

Issue

250 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Poetry is extremely slow when resolving the dependencies #2094

Description

Issue

Activity

David-OConnor commented on Mar 4, 2020

qiuwei commented on Mar 6, 2020

thmsklngr commented on Mar 6, 2020

yalunren commented on Mar 8, 2020

long2ice commented on Mar 9, 2020

luanfrv0 commented on Mar 29, 2020

luanfrv0 commented on Mar 30, 2020

MartinThoma commented on Apr 6, 2020

aoreki commented on Apr 16, 2020

gagarine commented on Apr 19, 2020

sdispater commented on Apr 19, 2020

gagarine commented on Apr 19, 2020

250 remaining items

tall-josh commented on Sep 15, 2022

johnthagen commented on Sep 15, 2022

neersighted commented on Sep 15, 2022

namekkozZ commented on Sep 20, 2022

zailaib commented on Sep 21, 2022

legioz commented on Sep 27, 2022

neersighted commented on Sep 27, 2022

legioz commented on Sep 27, 2022

ADR-007 commented on Oct 10, 2022

neersighted commented on Oct 10, 2022

ADR-007 commented on Oct 10, 2022

neersighted commented on Oct 10, 2022

petroslamb commented on Nov 21, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions