Repository navigation
Support poetry lockfile version 2.0 #761
Description
Overview
The Python poetry lockfile version is going to be updated to v2.0, starting with the poetry v1.3 release. It is not clear when this release will happen. However, the release of poetry v1.2.2 happened on 10 OCT 2022 and the announcement for that release shows that the release includes backwards and forwards compatibility for the new lockfile version for both poetry version 1.2 and 1.3
Additional Details
- It appears that the main difference is that package files and hashes are no longer stored in
metadata.hashesand instead are stored inpackage.files- This should not affect the way the lockfile is parsed for Phylum
- Only version 1.x
poetrylockfiles are currently supported
- Consider adding a new test fixture to account for the new lockfile version format
References:
- PR for lockfile format 2.0
- PR to add forward compatibility for lock file format 2.0, which will be used by Poetry 1.3
Acceptance Criteria
-
poetrylockfile versions 1.x and 2.x are supported for parsing - Tests are updated to include all supported
poetrylockfile versions - Documentation is updated
Activity
cd-work commentedon Oct 18, 2022
Hm I think this also brings up the interesting point of making sure all our other parsers stay up to date. Luckily you use Python every once in a while and I'd imagine a similar "automatic" realization would happen with cargo lockfile updates, but for many of our ecosystems we don't heavily testdrive our parser on the latest package lockfiles.
Would it maybe be worth considering adding simple tests for every supported ecosystem that generates a completely new lockfile (which should ideally use the latest version then), add a package, then try to parse it?
maxrake commentedon Oct 19, 2022
Great idea. It certainly helps more than waiting for new versions to break the parser.
There may still have to be a bit of a manual element involved, though, to keep up with ecosystem and package manager changes. For instance, the Python
poetryv1.2.0 release added subdirectory support for git dependencies, which is a feature that an established test would not have been able to account for.Similar features are added all the time in
poetrybut maybe the cadence is slower on other, more established, package managers. Still, the 80-90% use case can be accounted for with a basic lockfile generation test as described.Maybe something like a "lockfile zoo" repository could be used...or created if one does not already exist...to host the menagerie of lockfiles across all supported ecosystems and versions. It could be public and allow for community submissions for their esoteric use cases. Maybe the various lockfiles get categorized by level of support...kind of like how Rust platforms have tiers with varying levels of test coverage and support. The zoo language and taxonomic classification aspects certainly fit well with the "Phylum" theme...
kylewillmon commentedon Oct 19, 2022
Another potential option for testing this is to pick a few major projects in each ecosystem and download and parse their latest lockfile
maxrake commentedon Oct 20, 2022
Another option is to use the test fixtures from the package manager's own repos. For instance,
poetryhas an entire directory of known good and bad lockfiles and even has one called up_to_date_lock that uses the latest v2.0 lockfile format.Add poetry lockfile v2 support
Add poetry lockfile v2 support
Add poetry lockfile v2 support
Add poetry lockfile v2 support (#780)