Mac OSX binaries and Windows binaries that use GDAL/GEOS/PROJ

[edzer]

I'm trying to bring together several discussions here:

• @s-u has argued against brew builds for most users here: PROJ6/GDAL3 causing "Cannot find proj.db" error fdetsch/MODIS#78 (comment)
• @tim-salabim argued in PROJ6/GDAL3 causing "Cannot find proj.db" error fdetsch/MODIS#78 (comment) that it would be good that OSX and Windows CRAN binaries would have the same (in any case: similar) versions of GDAL/GEOS/PROJ, and for instance provide the same set of GDAL drivers.
• @pat-s and @ateucher have been discussing improvements to OSX/homebrew build instructions for sf README.md: Update macOS install instructions in README sf#1430
• Brian Ripley has compiled static binaries for OSX with modern spatial libraries, see: https://stat.ethz.ch/pipermail/r-sig-mac/2020-June/013599.html ; Brian shared discussions preceding this email with @rsbivand and me.
• Most CI setups (GitHub actions) I've seen for OSX now use brew, but OSX compiles on CRAN is our target

I agree that syncing the binary builds on OSX and Windows in terms of versions and drivers would be marvelous. I have experience with both statically building OSX binaries of sf (the way @s-u does for CRAN) as well as using homebrew, and bad experiences with trying the former while having homebrew libraries installed. I'm hesitant to advice dynamic linking to OSX users who have no clue what "compilation" or "dynamic linking" means, because they will need to recompile/reinstall the R packages when GDAL etc libraries get updated, but probably won't understand why things don't work in the first place when this happens. For those users, statically linked binary R packages seem the best advice.

@jeroen's rwinlib has made it easy for pretty much any user to install dev versions: install rtools and there you go, regardless whether you understand what is going on or not. (It may have led to the large number of CRAN packages linking to GDAL etc). It would be perfect if a static build system with similar simplicity existed for OSX (I don't know if it does). Jeroen pointed me to recipes but I'm not sure what to do there. Can I use that to locally build static builds with library versions I want?

I am confused about:

• what the path is to update spatial libraries used for CRAN OSX builds, who to contact and how (Brian? Simon? PR recipes?),
• how we can sync GDAL driver support on OSX and Windows (HDF4 and OpenJP2 for OSX come to mind),
• and whether we should use static, CRAN-like OSX builds in GitHub actions.

[s-u]

@edzer Thanks for starting this. As for macOS recipes is how a) you can replicate exactly what the macOS CRAN setup is and b) by issuing PRs you can update any libraries that your package needs which will be reflected on the macOS build server. Alternatively you can just let me know what your package needs and I can make those changes myself. My main problem here is that I don't know which features your packages need and which dependencies are required as you can build GDAL with many different options - I can't tell if something is useful or not.

Recipes are not intended for end-users as users will simply download the package from CRAN which uses the libraries from the recipes. If developers want to re-build a package, they can download the library binaries from https://mac.r-project.org/libs-4/ instead of building them themselves.

The macOS recipes setup pre-dates the Windows one by many years. The Windows system was put in place only very recently for R 4.0.0 and I agree that it would be nice to have some consistency, but those are very different operating systems so it may sound easier than it is.

I was playing with a Jenkins setup, but the R support in Jenkins is terrible, so it pretty much has to be done from scratch. I am still hopeful that I can have a script that can replicate the CRAN setup in Jenkins. As for GitHub actions, those are, unfortunately, hopeless as they don't support the necessary macOS versions, so Jenkins is the only viable path I'm aware of.

[s-u]

I spent most of today trying to work-around all those bugs in the spatial libraries that have to do with static linking. All of them are unrelated to macOS, and I'm documenting my progress on the recipes wiki on known issues in libraries. Especially for GDAL the list is pretty long. That said, the recipes now include GDAL 3.1.1, GEOS 3.8.1, PROJ 6.3.1, NetCDF 4.7.4, HDF5 1.12.0 and HDF4 4.2.15.

I only tested rgdal and MODIS which pass checks and sf which doesn't pass its tests, but compiles (this may be related to other packages). I have not rolled the update out to the production CRAN machine yet, I want to run more package checks, but you can download the library binaries from https://mac.R-project.org/libs-4/ as usual.

[mdsumner]

wow, thank you!

@s-u I'm a little reluctant to ask as I'm out of my depth, but - is the workflow in the first post here (to unpack the 10.13 SDK) not viable to emulate at least High Sierra on CRAN?

r-lib/actions#69

[s-u]

@mdsumner no, it is not viable. We tried it before. It's not as good as it sounds, because SDKs don't work for configure tests as you'll still get run-time behavior of the system, so that's why you really want VMs and not just SDKs. That's why binaries compiled on more recent macOS still don't work and why I said Actions are not an option for this, unfortunately. I don't understand why GH killed them, there is no technical reason, they just simply don't care about macOS, so Jenkins is all that's left.

[mdsumner]

ok, thanks very much

[s-u]

BTW: to clarify, the reason the SDK use was added is sort of the inverse - to avoid using the most recent SDKs which are partially broken. So it makes recent Xcode to not break, but the resulting binaries are not guaranteed to work on High Sierra (Xcode has been always funky - Apple likes to supply newer SDKs than the OS which is really odd). So it helps in some cases, but doesn't solve the task of replicating the CRAN setup - for that you need a 10.13 VM.

[mdsumner]

but, fwiw it's not completely pointless right? I feel pretty good that I can get those CRAN binaries and have a full test pass on 10.15:

https://github.com/hypertidy/vapour/runs/825320263?check_suite_focus=true
(it's the first macos entry, the second uses brew)

at least it's teaching me a lot, I wasn't sure about all your configure notes but - all the patches and configure details are now built-in to those binaries, is that accurate?

I have no idea about Jenkins yet, but it's not impossible we have access to these VMs at my work so I will ask around.

[s-u]

@mdsumner no, that's great! I like that the workflow allows to fetch the latest library binaries so, yes, for all intents and purposes it's as close as you can get with Actions. Don't get me wrong, I think it's perfect for testing. What I was referring to was to replicate the CRAN setup exactly to trace any issues that may come from the checks, which is possible only if the CI system allows the use of 10.13 VMs.

[mdsumner]

Nice, understood!

[tim-salabim]

@s-u thanks for all this effort! This is highly appreciated from the Rspatial comunity.
I wonder if there could be a more carved out and aligned process for pooling the expertise of the CRAN build maintainers and package developers. As much as CRAN build maintainers cannot be expected to know the ins and outs of all the system libs and which would be a suitable choice for which package, I assume that most package developers will have next to no idea about the build process and what it entails. I think the recipies repo is a good start, but if I understand corrrectly, this is only for MacOS?
I'd really like to make an argument for trying to align Mac and Windows builds as close as possible (@jeroen) because it will be hard to explain to users why they can do something in one OS but not the other. This will likely also lift (or better shift) workload from package maintainers (there have been sooo many issues regarding gdal and friends in the Rspatial world - though understandably as they've undergone major breaking changes).

Anyway, this is just a few thoughts from a package maintainer that does not really have experience with any of the steps involved in getting things set up in a usable way (I leave all this up to @edzer via sf basically).

I really appreciate all the work that is being done to make this as painless as possible for all parties involved

[mdsumner]

Just one comment to add to @tim-salabim - totally agree about alignment of library versions, but I'm personally less concerned about alignment of available GDAL drivers on Windows and MacOS - I think it's more important to have as many drivers as possible on each, even if one is missing some the other has. I know that might be controversial, because it implies different capabilities and testing requirements.

In my experience over the years the ability to access a new format (for me) was always instructive and helpful, more important than system consistency - but I can see that might be a topic for discussion ;)

[rsbivand]

@tim-salabim Pain is essential to find pinch-points, like: OSGeo/PROJ#2084 and OSGeo/gdal#2672. We also feed back up the component tree when necessary. For both GDAL and PRØJ, @rouault has been very helpful and responded as quickly as possible, so subsequent releases of PRØJ and GDAL will have changes suggested by CRAN.

[tim-salabim]

@rsbivand of course. I am more thinking about catching these pain points early and having a system to avoid them later, hence my "shift" in brackets. Admittedly, I don't really have an idea about the whole process that is necessary to ensure stable useful builds on CRAN. Just thought my amateur views might be helpful (different angle) and wanted to express my gratitude to all people involved

@mdsumner I agree that having a gdal suite that is as complete as possible would be great. In the end that's where the pwoer of gdal lies right

[tim-salabim]

Oh, and as a sidenote and from a very selfish standpoint, I'd love to see GDAL 3.1.x in both MacOS and Windows binaries, as it will make the mapview experience significantly nicer :-)

[pat-s]

I would like to draw attention to the following points which IMO usually do not get discussed:

1. Replicating the CRAN 10.13 setup which builds the binary and uses static versions of system libs
2. Ensuring a robust build process against the most recent stable macOS version with rolling updates of system libs (homebrew)

Most of this thread talks about 1). This is fine as long as people use binaries. However, installation issues from source will not be solved by referring to the CRAN 10.13 setup as nobody will (be able to) replicate this setup. Installing snapshots of system libs is a pain and one cannot simply go back to macOS 10.13.

What I am missing is a robust check system against the most recent macOS version.
Most people run this macOS version (or one major version lower) and also have homebrew installed to manage the system libs (which is perfectly fine as I outlined here already).

The default to check against macOS is either locally or via CI, using the latest stable macOS version.
It is unfortunate that one is pointed to the CRAN setup if something does not work and CRAN itself does no checks against the most recent stable version.
This way, package maintainers stumble across new issues themselves or users have to find out the hard way by opening issues because something does not work.
Currently, CRAN essentially forces you to stay two major versions behind on macOS (in theory) because everything else "is not tested".
This is not even possible if you buy a new machine because you cannot go back in time.

When giving R courses/consulting R, many people have Macs and I am always have to explain them why R packages are so unstable when it comes to packages that use more than just plain R code.
And no, binaries are not the universal solution for every package.
This is more of a general issue than a specific one to the r-spatial build process though.

Building universal binaries across major versions of macOS is hard.
This is why homebrew builds binaries for different macOS versions.
In a dream scenario, this is also what CRAN would do - using the latest stable versions of homebrew formulas.
This would be an easy and robust build process but would of course also require more hardware.

CRAN could also make use of GitHub Actions and ensure a proper build process against the most recent macOS system and the latest homebrew libraries.
This is how most people use R on macOS and it's a pity that only the lower supported limit (right now 10.13) is tested.
It would be better to test against both limits: the lower OS version limit with static library versions for building binaries and a dynamic upper limit against the most recent stable OS and system libs versions.

Usually the r-lib/tidyverse guys or other ambitioned packages maintainers lift all the support for the most recent macOS versions because these usually have a CI test system set up.
However, these are rarely heard when it comes to upstream CRAN adjustments and often enough, custom workarounds against pain points in the CRAN setup need to be made.