Repository navigation
Make the lock file more merge-friendly #496
Description
- I have searched the issues of this repo and believe that this is not a duplicate.
Issue
When two devs install dependencies on separate branches, it is very easy to end up merge-conflicted, in particular, the metadata.content-hash key often changes. It is very unclear how to resolve this manually, so I often delete the lockfile (or perhaps just that key) and rebuild it and, basically, hope that it comes out the same.
It seems like in some scenarios that merge conflicts could be resolved automatically based on pyproject.toml. Yarn does this, for instance.
Reacted by nyanpasu64, Andrew Standley, Andrew Baxter, Christopher Dignam, Lê Hoàng Phương, Claudio Jolowicz, Maksim Kalmykov, Bob Whitelock, Jan Škoda, Jordan Hamill and 113 more
Reacted by Oboleninov Anton
Activity
simonpercivall commentedon Dec 4, 2018
+1 Resolving the
content-hashand trivial conflicts in the individual hashes section would be very welcome.nyanpasu64 commentedon Jan 20, 2019
Workaround:
poetry add pathlib2poetry remove pathlib2or some other similar innocuous package.
Will this issue be resolved soon?
Does a missing content-hash hurt security or not?
seansfkelley commentedon Apr 17, 2019
Yarn's implementation of this is way less insane than I was expecting:
Extract the two versions of the conflicted file.
https://github.com/yarnpkg/yarn/blob/a7334da31bf783af7a3efab451589fe7ac34c748/src/lockfile/parse.js#L397
Blindly try to parse the files, and if that's successful, shallowly merge them.
https://github.com/yarnpkg/yarn/blob/a7334da31bf783af7a3efab451589fe7ac34c748/src/lockfile/parse.js#L399
If the merge conflict resulted in a syntax error, it fails. Yarn's lockfile structure is designed to make this easier: it's flat, it's sorted alphabetically and node is okay with duplicated dependencies in the tree.
itajaja commentedon Oct 4, 2019
is
content-hashmandatory though? the main problem I am encountering is that dependabot will have to ALWAYS rebase all the PRs, which could be avoided ifcontent-hashwas not in poetry.lock. in theory shouldn't be necessary, as it can always be computed by the other hashes, right?stale commentedon Dec 3, 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
chdsbd commentedon Dec 3, 2019
I think this is still a valid issue and should not be closed.
cjolowicz commentedon Feb 13, 2020
I have written a stand-alone script as a stopgap. It only goes half the way, resolving the merge conflict for
metadata.content-hash. The remaining conflicts, if any, should be quite trivial to merge manually. Use at your own risk.https://github.com/cjolowicz/scripts/blob/master/python/poetry-merge-lock.py
Update:
Please use poetry-merge-lock from PyPI instead (see comment below). This should allow you to merge without manual conflict resolution, in most cases.
cjolowicz commentedon Feb 19, 2020
Here is a stand-alone tool which should handle most merge conflicts in the lock file:
Install it with pip or pipx:
This tool is in early development. If you're interested in trying it out, please let me know on its issue tracker if you encounter any problems.
@sdispater, @finswimmer : Would you be interested in a PR to add this as a Poetry command?
finswimmer commentedon Feb 20, 2020
Hello @cjolowicz ,
your contribution is very welcome. I find this feature very useful. 👍
However, I cannot promise if and when this can be included. Including new features is decided by @sdispater .
But please go on!
fin swimmer
donbowman commentedon Jul 11, 2020
as long as there is a line
content-hashwe will never be able to conveniently use dependabot. When dependabot runs, it creates 1 branch and merge request per update, w/ the idea that you let your CI run, and then auto-merge.However, because of
content-hash, with poetry, every single one of these branches has a merge conflict and must be manually dealt with, increasing the human time from a few seconds to ~10-minutes per, and, causing another run of the CI to be required.The repository I just tried this on, 5 MR were created. If it uses tox, it takes no human time and 5 CI runs.
If I use poetry it takes 10 CI runs, and a manual clone/checkout/rebase/poetry update hash somehow/push, which took me more than an hour.
Multiple this by my 30 repos, and make it a daily thing to have to deal with and it becomes intractable.
The solution cannot be in a new poetry command to resolve the lock, that would still require all the manual work.
Instead, we need to be able to merge poetry.lock files that are not otherwise conflicting.
Beacuse all these MR are created at once, in parallel, we cannot teach dependabot to do it either, at the time of each MR they all come from the same spot on master so there is no conflict, its only when the first is accepted that we need to rebase the 2nd, and when it is accepted we need to rebase (twice) the 3rd, and so on.
Can we not make the issuance of
content-hashoptional?event without dependabot its troublesome.
Ideas:
so perhaps if there was a --no-output-hash option to the
poetry update --lock?cjolowicz commentedon Jul 11, 2020
@donbowman Doesn't Dependabot rebase your PRs automatically if they conflict? This is what happens for me. I only need to resolve conflicts when rebasing or cherry-picking my own commits, in which case poetry-merge-lock mostly works fine. I sometimes follow it by
poetry add insecure-package && poetry remove insecure-packageto ensure that the lock file is up-to-date. (insecure-package is just an empty dummy package that's never part of my dependency tree.)feat: add --no-hash option to update, preventing metadata.content-hash
54 remaining items
djmattyg007 commentedon Nov 29, 2022
Even if you don’t want renovate to group dependency updates, you can still do things like ask it to only create one PR at a time, for example. This can reduce the burden of wasted CI time. No matter which way you look at it, it will still make your life easier in dealing with this poetry lock file problem.
zyv commentedon Nov 29, 2022
Well, it's a non-non-sequitur.
Right now, if you want a completely automated solution to resolve merge conflicts with Poetry lockfiles, you have to tell dependency bots to "rebase" their PRs every time a PR touching the lockfile is merged. I'm putting "rebase" in quotes, because in the context of dependency bots like dependabot or Renovate it's not about git rebase, but rather about re-locking the lockfile.
So, imagine, you have 4 dependabot PRs. You merge 1 PR, and trigger 3 PRs to re-lock and re-run the CI (not only Poetry, which might take an insignificant amount of time as compared to tests, but everything). Then you merge the 2nd one and the dance repeats.
This is what drains CI-time, not a possibly failing Poetry call, which (if it indeed fails) will actually conserve CI time, because downstream jobs just won't be able to proceed.
So I think your thinking goes into the right direction in terms of saving CI time.
neersighted commentedon Nov 29, 2022
Maybe this is a matter of differing workflows, but as when I use automated tools there is one PR or they are batched, the prospect of "random" failures requiring human intervention during the install stage seems much more likely to waste CI time.
bmitc commentedon Feb 22, 2024
Have Poetry developers looked at other lock file implementations to help with creating a more merge friendly format? What about taking a look at Elixir's
mix.lockformat? I never had any merge issues with it, but I am constantly having issues with even small Poetry projects.mrcljx commentedon Mar 15, 2024
The content-hash remains to be a big point of frustration. When three engineers add dependencies it's a constant race who gets their merge in first. The other two will have to rebase either once or twice to fix the
content-hash.simonpercivall commentedon Mar 22, 2024
The issue I'm most afraid over right now over how the
poetry.lockfile works is updating sub-dependencies that don't touchpyproject.toml. Now, in a pretty big team of mixed-seniority devs, they don't always know how to properly merge and inspect the result, and it's happened more than once that an upgraded sub-dependency gets invisibly downgraded in a merge conflict resolution (even worse, git makes it really hard to see things that change in a merge commit); that is, the dev blindly checks out--theirsor--ours, and then runspoetry lock --no-update.If there was even an imperfect way poetry could decrease the surface of a conflict (and just having a built-in way to resolve
content-hashconflicts would solve like 95% of conflicts), it would go a long way to making it easier to usepoetryin a team. (Sure, we could go back to pinning every sub-dependency inpyproject.toml, but then what's the point.)In a tool that I otherwise really enjoy using, this is a painful pain point.
dimbleby commentedon Mar 22, 2024
since there is no content-hash change when pyproject.toml is unchanged, a "built-in way to resolve
content-hashconflicts" would make exactly no difference in these casesbmitc commentedon Mar 22, 2024
At the moment, I am the only developer managing dependencies with Poetry, and I actually don't know how to manage
poetry.lockat all. On any merge conflict, I just blast away (i.e., delete) thepoetry.lockfile and regenerate it withpoetry installafter fixing any merge conflict inpyproject.toml.I feel Poetry is absolutely essential to use, but I don't know how to manage the
poetry.lockfile even in very simple cases. When a hash changes, I don't know what to do with that without a version change as well, so I am not personally aware of what decision criteria to use when looking at a merge conflict inpoetry.lock. The above method of deleting and regenerating it actually seems to work okay, but it seems like it defeats one of the purposes of the lock file.simonpercivall commentedon Mar 22, 2024
Ah, I explained badly:
pyproject.tomlpyproject.tomllock --no-update, step (1) is now goneBut as been said earlier in the discussion: Of course, to go as far as possible, any solution would need to look at both sides of the merge, and try to solve them together (like the NPM way someone described, or like the now-outdated "poetry-merge-file" package). And this is of course much harder to maintain.
simonpercivall commentedon Mar 26, 2024
@neersighted Have you thought any more about this idea? To me, it sounds ideal if possible.
(Sorry if this became spammy, but there's been so much back and forth in this issue that I felt a moments suggestion/thought may have been lost.)
king-of-poppk commentedon May 2, 2024
This is not just a dependabot issue: if you work on a long-lived branch where you add/remove/upgrade dependencies good luck with rebasing against external dependency upgrades (for instance, merged dependabot PRs)!
Andrew-Chen-Wang commentedon May 16, 2024
This is more for myself, but having to rebase all the time, just accept the incoming rebase:
then re-lock all the lock files. This primarily assumes the development branch's PR is JUST for adding a new package.
SlowMo24 commentedon Mar 31, 2025
As far as I see this is mainly about the hash and the content of the
poetry.lock. I'm annoyed by a smaller issue: Putting the poetry version into the lock-file header basically creates an edit war in our repository:# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.By recording the full version, each developer will change that line if they are on a slightly different version. I suggest to either only record versions that are relevant for the file (i.e. incompatible in creating or reading it) or not recording the version at all.
webknjaz commentedon Apr 1, 2025
PEP 751 was accepted yesterday. Poetry should implement that instead.
king-of-poppk commentedon Apr 1, 2025
See: https://peps.python.org/pep-0751/#hashing-the-contents-of-the-lock-file-itself.