Standardise ensure_dir and rmdir

[Julian-O]

1. A new version of Flake8 dropped while I was submitting this PR. There are several inconsequential edits to unrelated files, because Flake8 held my tests at gunpoint until I did it.

2. When p4a wanted to create a directory, it used any of several different techniques:

   • it called os.mkdirs()
   • it called shutil.mkdir()
   • it called util.ensure_dir()
   • it called bootstraps.common.build.build.ensure_dir()
   • it called sh.mkdir()

The latter has terrible performance and isn't cross-platform.

• I modified util.ensure_dir() to log if and only if it made a change.
• I modified all directory creations to use util.ensure_dir().
• I removed any now unnecessary logs and existence checks from the code, so the clients are now simpler.

3. When p4a wants to remove a directory, it used any of several different techniques:
   • it called shutil.rmtree
   • it called sh.rm("-r")
   • it called subprocess.check_output(["mkdir"..])

The latter two have terrible performance and aren't cross-platform.

• I created util.rmdir(). It logs if and only if it makes a change.
• I modified all directory deletions (outside of test code) to use util.rmdir().
• I removed any now unnecessary logs and existence checks from the code, so the clients are now simpler.
• recipe.py had some weird code to delete then create then delete a folder. I simplified it to delete.

4. Whenever I had to touch the imports, I sorted them to make them PEP8 compliant.

5. A couple of the receipes (e.g. sqlite3) weirdly imported shutil indirectly, via toolchain, which caused a problem when toolchain stopped importing it (no longer required). Changed the recipe code to import shutil directly.

Note I only tackled two system calls; I didn't even look at removing single files. There are more to come, but I want to keep each review simple, and this one is already too complex.

[misl6]

libmysqlclient is known to be broken (see: #2808). Can you please add it to BROKEN_RECIPES_PYTHON3 (ci/constants.py) so we can see ✅ on "Updated recipes" checks?

[Julian-O]

I don't pretend to understand what the implications of that request, but I went ahead and did it.

[misl6]

I don't pretend to understand what the implications of that request, but I went ahead and did it.

Build tests are performed on pre-defined demo apps (always) and on updated recipes (when a recipe gets updated).

Since you edited libmysqlclient recipe file, the CI was trying to build it as part of the "Build Updated Recipes" test.
But, unfortunately, this recipe is broken (since a while), and the whole "Build Updated Recipes" CI pipeline failed.

Adding libmysqlclient to BROKEN_RECIPES_PYTHON3 allows us skip libmysqlclient build during the "Build Updated Recipes" test phase.

[misl6]

Hi @Julian-O !

Sorry for the late reply, but I was on vacation 😀📷

Seems we still have some issues to perform the build as boost recipe is not working anymore. (boost is required by libtorrent, which has been edited by this PR)

Similarly to libmysqlclient, can you please add boost and libtorrent to BROKEN_RECIPES_PYTHON3 ?

[misl6]

Uh 😅

[misl6]

Excellent.
LGTM. Thank you!

[NomadDemon]

how about to migrate everything to "pathlib" from python ? :) it would be also nice improvement

[Julian-O]

@NomadDemon: I agree. Given the amount of path-processing involved, pathlib would be a boon.

I am trying to keep my changes small and incremental to make reviewing easier and to reduce the chance that I break anything. I already have a lot on my plate. So I haven't included it here.

But what I have done is make the new Buildops library compatible with path-like parameters, so it is easier to migrate the clients that use it in the future.

[Julian-O]

@NomadDemon: Sorry. Ignore that last sentence. That refers to Buildozer, not Python-For-Android.