add --x-python3-version and --force-x-python3-version options

[dirk-thomas]

Based on discussion ros/rosdistro#5944

Currently the Python 3 Debian package always has a dependency on the Python 3 version of the system where the packaging is run.

This patch adds support for putting X-Python3-Version in the stdeb.cfg file. If the new option is present and gets passed into the source block of the Debian control file. Additionally ${python3:Depends} will be omitted to avoid also adding the explicit dependency on the Python version of the system running the packaging command.

If the new option is not in the stdeb.cfg file the behavior is unchanged.

[p1otr]

please DO NOT merge these changes.

1. ${python3:Depends} is a must for Python 3 packages
2. X-Python3-Version syntax is described here: https://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html and with this patch one can put whatever (s)he wants

[astraw]

@p1otr can you check out the discussion at ros/rosdistro#5944 and comment? Thanks.

[dirk-thomas]

The first commit of this patch which passes through X-Python3-Version should be fine, right? Until now the option was simply ignored by stdeb.

Regarding the second commit (omitting ${python3:Depends}):

@p1otr Can you please describe how I should release a Python package which should only depend on Python 3.2 using X-Python3-Version on a machine which run e.g. Ubuntu Trusty (which has Python 3.3)?

When I release a Python package on an Ubuntu Trusty machine with the following line in the stdeb.cfg file:

X-Python3-Version: >= 3.2

the resulting Debian control file will still contain python3:any (>= 3.3.2-2~) in the Depends line.
Effectively that prevent me from making use of the X-Python3-Version since it get overridden by the default python3:any (>= 3.3.2-2~). Therefore I did not see any other way then removing ${python3:Depends}.

I am happy about any other workaround to make my use case work.

Update: actually even with X-Python3-Version: >= 3.2 set this won't generate the correct dependencies for me.

[p1otr]

about X-Python3-Version: it should match something like this: '^((\s_>=\s_\d.\d+)(,(?=\s_<))?)?(\s_<<\s*\d.\d+)?$

about ${python3:Depends}: removing it will not change the fact that given package was prepared for Python X.Y. In other words: if you build it on Ubuntu X, it doesn't have to work on Ubuntu X-1 (and most probably will not)

[dirk-thomas]

In our use case we do have a Python package which sources work fine with Python 2.6, 2.7, 3.2, 3.3, 3.4 (probably even more but these are the versions we test on).

So you say that there is no way to release this package from a single distro but it must be released once from each distro we want to support? For us that is Ubuntu: O, P, Q, R, S, T, U. That is simply unpractical.

The Debian package will not contain any compiled code - even the .pyc files are generated in a post-install step. Why would one package not fit all distros?

[astraw]

@p1otr The issue is not about building the binary, but about building the source package. The ROS folks wants to make a source package (.dsc et al) on Trusty. This source package should compile on Precise (for example). Other than version incompatibilities (which will fail loudly) I don't see what else can go wrong.

@dirk-thomas I don't think the suggestion is to build N source packages. Merely build one on the oldest supported version.

[p1otr]

@p1otr Can you please describe how I should release a Python package
which only depends on Python 3.2 using X-Python3-Version on a
machine which run e.g. Ubuntu Trusty?

1. if upstream source code needs at least Python 3.X, put ">= 3.X" in
   X-Python3-Version, no matter for which Python version you're building.

2. rebuild your source package (without any changes outside
   debian/changelog, hint: dch) on each Ubuntu version you want to
   support (unless the one built on Ubuntu A works fine on A+1)

[p1otr]

I'd even simplify the regex to ^\s_>=\s_\d.\d+$ as "<< X.Y" can be used in very rare cases (and I really mean rare - 99% of "<< X.Y" on PyPI do not qualify here)

[p1otr]

please also note that building on Ubuntu A+1 and trying it on A will not work, build it on the oldest Ubuntu you support and rebuild it on newer if the old one doesn't work

[dirk-thomas]

I have to disagree with that statement. It is absolutely possible to release a package on e.g. Trusty (X) and in a way that it also works on Precise (X-4) and Utopic (X+1). We have done this for years with Python 2 packages and also for the last year for Python 3 packages.

Of course there are limitations if e.g. the name of the dependencies have changed (which is not the case in our case). Then for sure you need separate packages.

The Debian control file must obviously specify a minimum version which can be satisfied by all targets suites. And that is currently the only limitation. dh_python enforces a versioned dependency (http://anonscm.debian.org/cgit/dh-python/dh-python.git/tree/dhpython/__init__.py#n29). And X-Python3-Version can not relax this dependency but only make it more strict.

Regarding your bullet (1): specifying X-Python3-Version: >= 3.2 and running the release on Trusty will still result in having a dependency on >= 3.3.2-2 (which imo is a bug / shortcoming in dh_python since it overrides what the developer asked for).

[p1otr]

X-Python3-Version is a way to tell dh_python3 what upstream source supports (and it doesn't change if you build it on different distribution)

MINPYCDEP is bumped every time a new Python or Debian's Python feature is used f.e. in maintainer scripts so if you lower this without rebuilding a package on an older Ubuntu/Debian version it will faill sooner or later.

so again: you can build on A and if it works on A+1, A+2, A+3 - that's fine, if it's doesn't work on A+4. you need to rebuild it and then it will work on A+4 and maybe on A+5, A+6, ...
if you build it on A+6 it's very likely it will not work on A+1

[dirk-thomas]

If the developer "knows" that a package works with a specific version of Python he should be able to specify that and override any kind of default. Currently dh_python and stdeb do not allow that.

For our use case where we do "know" what the supported minimum version is (since we do not use any of the newer features introduced in later versions) that is just not flexible enough. It is simply not feasible for N packages to have a chroot / vm around for the minimum distro supported - since they might be different from package to package.

Since the actual inflexible behavior seems to be part of dh_python I assume stdeb can't do anything about it. I agree that leaving out ${python3:Depends} is a hack and not the right way.

Sadly that seems to leave no options for our use case...

[p1otr]

dh_python3 allows that, it's called... X-Python3-Version

">= 3.3.2-2" (taken from MINPYCDEP) is generated due to multi-arch features, it's not because upstream code doesn't work with 3.2, it's because Debian binary package doesn't work with 3.3.2-1. If you lower it, it will most probably work for most of your users, but the one which is using mutli-arch will blame Debian for not generating correct minimum required python3 dependency (a package which provides f.e. py3compile and py3clean scripts used in preinst, postrm, etc.)

If generated Debian binary package uses features from Python 3.3 in maintainer scripts, you cannot override >= 3.3 dependency because your upstream code requires >= 3.2. That's why I do not allow X-Python-Version to override MINPYCDEP. I know, I should generate >= MINPYCDEP only when it's really needed (i.e. split this dependency into dozens based on used features), but I'm lazy and I bump it every time I need new feature in any possible corner case I implement in dh_python3.

Just build it on oldest supported Debian/Ubuntu, test it on newer ones and if it works, no need to rebuild.

PS If you build an app/library with libc 2.19 - do you expect it to work on libc 1.7 as well? It's easier with C, I know (there are sane sonames "infrastructure" there), but still

[astraw]

If the lines that remove ${python3:Depends} are removed (L792-793 in the diff), there's little controversial here as far as I can see. (@p1otr, the X-Python3-Version will still get parsed by the normal Debian machinery, right? If so, then there is no need to write a parser/validator in stdeb.)

Optional removal of ${python3:Depends} is obviously more controversial but at the end of the day a developer can alter their source packages however they want. stdeb can try and make good defaults, but @dirk-thomas is free to write a non-stdeb parser that changes this anyhow. (He would realize this would get his packages rejected from Ubuntu or Debian, but he might still choose to do so.)

So I propose to decouple the two aspects: 1) specify X-Python3-Version as in the original patch and 2) a new CLI switch like --no-python3-depends-evil-hack.

[dirk-thomas]

@p1otr As I mentioned before: while it might seem feasible for a maintainer of a single package to release the package on the oldest supported Ubuntu it simply isn't for us.

Consider N packages with the following supported platforms:

• pkgA: O, P, Q, R, S, T, U
• pkgB: P, Q, R, S, T, U
• pkgC: Q, R, S, T, U
• pkgD: R, S, T, U
• pkgE: S, T, U
• pkgF: T, U
• pkgG: U

That means I have to use 7 (seven) different Ubuntu versions to release Python packages.

@astraw The proposed approach to separate both aspects would be fine. But I do see the disadvantage of removing ${python3:Depends} all together.

It would be much better to only ignore MINPYCDEP if the developer chooses so. @p1otr Wouldn't that be an option to allow the developer to ignore it when he is certain that the toolchain still works with an older version?

[astraw]

@dirk-thomas is there a way to ignore MINPYCDEP while still keeping ${python3:Depends}? Otherwise I'm not sure what can be done in stdeb itself.

[dirk-thomas]

No, dh_python does currently not allow to ignore MINPYCDEP. But I would consider that to be the "cleanest" approach.

After spending more and more time looking into the internals of stdeb and dhpython I figured out the following:

• when specifying X-Python3-Version (and using the first commit of this PR) that version range will be used in the generated python3-rospkg.postinst.debhelper file to compile the Python code in the post-install step for the desired Python versions
• the generated python3-rospkg.substvars file does contain the substitution for the python3:Depends variable, this includes the dependencies of the version range specified by X-Python3-Version as well as the one coming from MINPYCDEP

In our case the MINPYCDEP will contain a higher version number then the lower bound of the version range specified by X-Python3-Version.

Therefore I will try the following:

• remove the second commit of this PR since we all agree that it is a bad hack
• find a way to modify the substitution file to fit our use case (remove the MINPYCDEP entry when X-Python3-Version specifies a different (lower) value)

Again I am looking forward for any kind of feedback on this.

[p1otr]

well, do whatever makes it work for you. I will remove it before uploading to Debian most probably anyway. The proper way to have lower MINPYCDEP is to split it into several minimum dependencies and use the right one whenever given feature is used. Removing minimum required dependency without changing anything else (i.e. things that were the reason for bumping minimum version) is just plain wrong IMHO

[dirk-thomas]

@p1otr I do understand the reason for the minimum version. Can you give me a specific example where this would actually affect one of our packages? Like package python-catkin-pkg uses feature X which when released with my approach will break since it only exists in version Y.

Is there a way for me to find out what kind of features require which version (beside MINPYCDEP) in order to judge for myself if a lower version is a problem or not?

Also the developer doing the release can easily confirm that the released package works (is installable) on all listed suites as expected (if he is using this non-standard potentially dangerous approach).

[dirk-thomas]

I have updated the PR.

The first commit (f167778) adds the command line option --x-python3-version as well as the X-Python3-Version entry to the stdeb.cfg file. The entry is added one-to-one to the Debian control file.

The second commit (09f7c09, 724476d) adds the command line flag --force-x-python3-version. When it is passed the minimum version provided by X-Python3-Version will be "enforced". That means other dependencies of python3:any with different version numbers will be removed from the Debian control file.

[dirk-thomas]

Using this patched stdeb version I was able to release two ROS Python packages (on a Trusty machine) which are installable on anything between Oneiric and Utopic:

• http://repos.ros.org/repos/building/pool/main/p/python3-rospkg/
• http://repos.ros.org/repos/building/pool/main/p/python3-rosdistro/

If you see any flaws with these Debian packages please let me know.

@astraw What is you opinion about this PR? Do you think it is realistic that it will be merge or should we keep (at least) the --force-x-python3-version option in a fork?

[dirk-thomas]

Do you have any feedback on this if both / the first / none of the commits can be considered for merge? I am fine with any answer but would like to move forward with either getting it merged or updating our toolchain to use a fork.

[astraw]

I haven't had a chance to try this yet, but I do intend to merge both commits if I don't find any problems. I hope I can get to this later this week.

[dirk-thomas]

Thanks, that is great. (We are not in a hurry - just wanted to know the direction.)

[dirk-thomas]

@astraw Any chance to look at this in the near future (and probably release a new patch version)? Thank you 😉

[dirk-thomas]

Any updates on getting this merged and released ❔

[dirk-thomas]

Thank you for taking the time to merge this as well as some other PRs.

I saw you just released 0.8.3 which does not yet include these changes. Do you plan a new release soon which will include the merged PRs?

[astraw]

Yes, I finally found a couple hours today to work on stdeb 🤘.

Indeed, pushing out a release with this PR is high on my to-do list. At the moment, I am hoping to do one more release fairly soon (day or days timescale) in the 0.8 series with this PR included.

Switching to pybuild (see the pybuild and pybuild-dev branches) will require dropping support for Ubuntu 12.04, which we still use pretty heavily in my lab. Therefore switching to pybuild will likely be further in the future and get a version bump to (at least) 0.9. But I'm curious how using pybuild will affect ROS -- can you port this PR to the pybuild-dev branch?

[astraw]

I just made release 0.8.4 with this PR included.

For the long-term future, it seems a pybuild-based approach is pretty nice and what I think Piotr would advocate. Therefore, I've been trying to bang the pybuild-dev branch into shape. That said, it'll likely be a while before I could use the pybuild stuff myself. I am curious (no rush) if you can sufficiently port the --force-x-python3-version option to the pybuild-dev branch.

[dirk-thomas]

Thank you for the latest release. We immediately use the new features for our ROS Python packages.

I am not familiar with pybuild. Is it this project: http://sourceforge.net/projects/pybuild/ ? That one does not look very active.

Anyway I am not sure if and when I can spend time on yet another iteration of the Python release tools.

[astraw]

Great, I look forward to ROS packages with source easily available!

pybuild is described here.

[dirk-thomas]

Sorry to bother you again. But can you point me to any sources of pybuild. I couldn't find anything on the Debian page nor on Google. I am curious how active it is developed etc.

[astraw]

The project page for pybuild seems to be https://launchpad.net/ubuntu/+source/dh-python and the repository is at git://anonscm.debian.org/dh-python/dh-python.git

[p1otr]

sources are in dh-python debian source package or in this Git repo:
git://anonscm.debian.org/dh-python/dh-python.git (you can browse it online here: http://anonscm.debian.org/cgit/dh-python/dh-python.git)
I didn't chanage anything recently because we're in a freeze, but I already have some improvements on my TODO list.

PS Please note that python-stdeb package from Debian is already using pybuild although I didn't use what's in pull request (it wasn't available at that time or I didn't notice it)

[astraw]

@p1otr the key commit of the pybuild-dev branch of stdeb is what I pulled from the sources of the 0.8.2 debian python-stdeb package (see f6cf8c6).

[p1otr]

ah. OK.

About supporting older Debian/Ubuntu releases - I already provided Debian Wheezy (current stable, soon to be replaced with Jessie) backport and same can be probably done for Ubuntu (minimum required Python version is 3.2 although I was really tempted to use Python 3.4's features from Jessie :)