PROJ6/GDAL3 causing "Cannot find proj.db" error

[mstrimas]

I'm on Mac OS Mojave and recently updated to proj 6.2.1 and gdal 3.0.2 with homebrew, which resulted in the following error when calling runGdal():

runGdal(product = "MCD12Q1", collection = "006", SDSstring = "01", 
        extent = "austria",
        begin = "2010.01.01", end = "2011.12.31", 
        outDirPath = "data", job = "modis",
        MODISserverOrder = "LPDAAC")
ERROR 1: PROJ: pj_obj_create: Cannot find proj.db
ERROR 1: PROJ: createGeodeticReferenceFrame: Cannot find proj.db
ERROR 1: PROJ: createGeodeticReferenceFrame: Cannot find proj.db
ERROR 1: PROJ: proj_as_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_as_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db
ERROR 1: PROJ: pj_obj_create: Cannot find proj.db
ERROR 1: PROJ: createGeodeticReferenceFrame: Cannot find proj.db
ERROR 1: PROJ: createGeodeticReferenceFrame: Cannot find proj.db
ERROR 1: PROJ: proj_as_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_as_wkt: Cannot find proj.db
ERROR 1: PROJ: proj_create_from_wkt: Cannot find proj.db

A little Googling turned up that it's looking for proj.gb in the PROJ_LIB environment variable, which is getting set to "/Users/mes335/Library/R/3.6/library/rgdal/proj" when MODIS is loaded. Through dumb luck I found that if I just nuke that env variable (Sys.unsetenv("PROJ_LIB")) everything works fine. No idea what's going on here, but would be nice to not have to always unset PROJ_LIB.

[mstrimas]

Update: I've just realized that, despite all the error messages, runGDAL() actually runs just fine and produces the correct output tifs. The error messages are extremely confusing though, so I think it would be worth looking into.

[fdetsch]

The issue is that PROJ_LIB is overridden as soon as rgdal or sf are attached. There is no file proj.db in the PROJ_LIB version shipped with these packages, and hence the error. I need to put some more thoughts in how we can handle this. In the meantime, runGdal() luckily just ignores the error.

[fdetsch]

@mstrimas Can you try again with the latest pkg version?

[mstrimas]

Can't even get to this point anymore, stuck in dependency hell. I have gdal installed with HDF4 support through brew, but still getting the HDF4 driver seems to be lacking error. Looking into checkHdf4Driver() I see it checks with sf::st_drivers(what = "raster")$name. Not clear what version of GDAL that's actually referencing, but it doesn't seem to be picking up the version of GDAL I installed with brew. I also tried explicitly setting the path GDAL in the options file, but doesn't have any impact.

[fdetsch]

I believe it's referencing the GDAL version that ships with sf..

[mstrimas]

Hmm, I'm using the CRAN binary for sf, so I guess that doesn't include HDF4 support. Also, I assume none of the GDAL binaries ship with sf, just the libraries, so does it really make sense to use sf::st_drivers(what = "raster")$name to check for HDF4 support given that that isn't the version that will actually be used to convert the files?

[fdetsch]

I don't quite understand why this wouldn't be the version used to convert the files. Given that sf::st_drivers(what = "raster") does not report HDF4 support in your case, and drivers available depend on the installation of GDAL/OGR (see ?st_drivers), wouldn't this indicate that sf isn't correctly linked to your local GDAL installation? Did you use configure.args during compilation as described here?

[mstrimas]

I've never been able to get those sf install instructions to work for me, so I'm using the CRAN binaries for Mac OS. I suspect this will be the case with almost all sf users, except the few who have the knowledge and interest in compiling from source. So I don't think sf is linking to my system GDAL, e.g. when I load sf it says it's linking to GDAL 2.4.2, but my system GDAL is 2.4.4.

[tim-salabim]

@mstrimas have you tried installing sf from github? This may solve your issue in two ways.

a. If I am not mistaken, this will link to your system GDAL
b. IIRC, there have been efforts to ensure back-compatibility with older versions of GDAL/PROJ/GEOS (though that may already be included in the current CRAN version)

remotes::install_github("r-spatial/sf")

All this is a major hick-up at the moment for many packages that rely on GDAL/PROJ/GEOS as those libs have gone through massive changes very rapidly lately.

[mstrimas]

Yeah, I have the latest versions of PROJ and GDAL installed via brew and have tried following the instructions for install from both CRAN and GitHub from source with no luck. Error is:

checking GDAL: checking whether PROJ is available for linking:... no
Undefined symbols for architecture x86_64:
  "OGRCreateCoordinateTransformation(OGRSpatialReference const*, OGRSpatialReference const*)", referenced from:
      _main in gdal_proj-20d5e8.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
configure: Install failure: compilation and/or linkage problems.
configure: error: cannot link projection code
ERROR: configuration failed for package ‘sf’

More broadly than my specific error, I think it should be expected that most people are using pre-compiled binaries of sf and will therefore not have the package linking to the system libraries.

[tim-salabim]

Well, then I think there's not much you can do but wait until MacOS binaries of sf ship with more recent versions of GDAL/PROJ/GEOS and hope that HDF4 driver is included then.

This seems to me like it should be solved at the point where the MacOS binaries are built for CRAN. I don't know who is in charge of these but the windows binaries which are maintained by @jeroen (I think) include much more recent versions of GDAL/PROJ/GEOS and also the HDF4/5 drivers.

[mstrimas]

Ahh, looked at the source code, I see what's going on. I didn't realize MODIS was using sf::gdal_utils(), I thought it was using the command line gdalwarp via system(). This makes sense now, but annoying since it means this package is unusable by the large majority of Mac OS users until those sf CRAN binaries get fixed.

[tim-salabim]

On the other hand, it takes burden off the maintainers. The problem is not with this package, the problem is with very old MacOs binaries. Maybe this should be raised on r-sig-mac

[fdetsch]

@mstrimas The pkg had been using gdalinfo and gdalwarp via system() calls before version 1.2.0. We would have had to check for the presence of GDAL utilities - and not install without them - if we had wanted to retain the pkg on CRAN in its old form.

Now, quite a lot of users seemed to have problems with installing GDAL w/HDF4 support (Python bindings etc.) in the past, not to mention linking it to MODIS. At the same time, there is now a good number of pkgs out there that does a rather good job at this, making the whole process a lot more error proof and, as @tim-salabim mentions, less troublesome for us to maintain. Hence the decision for sf, which is imported anyway.

So yeah, maybe r-sig-mac would be a good starting point to ensure a flawless experience for users on any OS.

[mstrimas]

Thanks both of you for explaining the thought process! Makes a lot of sense to use sf::gdal_utils() now that I understand. Seems like the thing to do is just to get those Mac OS binaries compiled with HDF4 support. I've posted to r-sig-mac, hopefully that gets the process started. Thanks!