PROJ 6 CRS handling changes impact workflows using +datum= (and other fun)

[rsbivand]

Ongoing changes in the representation of coordinate reference systems in the PROJ, external software used by sf, sp and other R packages, will impact R packages and workflows using these packages. In particular, the changes affect transformation between coordinate reference systems, as for example shown in this presentation at the OpenGeoHub Summer School 2019.

• Invite users/developers to contribute cases

• Add other examples and links, also from other geospatial software.

GDAL RFC 73: https://gdal.org/development/rfc/rfc73_proj6_wkt2_srsbarn.html

Notes from Spatialite

PROJ wiki proj.h adoption status

GRASS-dev thread and PR

PostGIS blog posting

cartography CRS issues

geojsonio geojsonio seems vulnerable to PROJ 6

MODIS getTile() sp subset or raster::crop issues with PROJ 6.1 and GDAL 3.0

MODIStsp PROJ 6 failures in MODIStsp

rangeMapper PROJ 6 failures in rangeMapper

rdefra PROJ 6 failures in rdefra

rnrfa PROJ 6 failures in rnfra

rshapemapper PROJ 6 problems?

Incomplete list of packages with CRAN issues after PROJ 6.2.0 installed on some test machines:

https://cran.r-project.org/web/checks/check_results_foieGras.html
https://cran.r-project.org/web/checks/check_results_plotKML.html
https://cran.r-project.org/web/checks/check_results_rangeMapper.html
https://cran.r-project.org/web/checks/check_results_rnrfa.html
https://cran.r-project.org/web/checks/check_results_sf.html
https://cran.r-project.org/web/checks/check_results_vapour.html

• Provide recommendations for checking/testing PROJ versions in use and expected transformation outcomes.

See also r-spatial/sf#1146 for sf and rgdal reprex with PROJ 4.9.3/GDAL 2.4.2 output from the same file contrasted with PROJ 6.2.0/GDAL 3.0.1.

This issue is based on https://github.com/rsbivand/geostat19_talk; https://rsbivand.github.io/geostat19_talk/Discuss_issue_drafts.html

[rsbivand]

Relevant CDN for serving grid files discussion on https://twitter.com/howardbutler/status/1171778886646022145?s=20. On the client side related to thread: https://lists.osgeo.org/pipermail/proj/2019-September/008825.html.

[rsbivand]

Adding files (scripts, output) map_package_analyses.zip for packages with reverse dependencies (depends, imports, suggests) on rgdal using pkgapi map_package() to try to see which packages use spTransform(), project(), and raster projectExtent() or projectRaster(). About 80 packages of almost 300 in total seem to use projection mechanisms on the sp side. The scripts may be used to set up similar reports on the sf side. Other columns report CRS() and other vulnerable points.

[rsbivand]

See also suggestions on https://github.com/pyproj4/pyproj; https://pyproj4.github.io/pyproj/stable/index.html.

[MichaelChirico]

Hi Roger, much appreciate your reaching out and pointing to all the resources.

However, I feel a bit overwhelmed by it all and am struggling to turn this into action for how to get my package (geohashTools) working with PROJ>=6.

I see this entry in map_package_analyses/df_all.csv for my package:

   raster_projectExtent raster_projectRaster rgdal_checkCRSargs rgdal_CRSargs rgdal_make_EPSG
1:                    0                    0                  0             0               0
   rgdal_project rgdal_spTransform sp_CRS sp_proj4string sp_spTransform sum
1:             0                 0      1              2              1   4

How can I use this to proceed?

Also, is it safe to use rocker/geospatial for testing (i.e., does that image have the proper system libraries in place already)?

Seems rocker/geospatial is still using PROJ 4.9.3, can you recommend any Docker image to use for testing?

[Nowosad]

@MichaelChirico for the purpose of testing PROJ6 in the Geocomputation with R book, I've created a new docker image based on rocker:

FROM rocker/rstudio:3.6.1

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
	gdb \
	git \
	libcairo2-dev \
	libcurl4-openssl-dev \
	libexpat1-dev \
	libpq-dev \
	libsqlite3-dev \
	libudunits2-dev \
	make \
	pandoc \
	qpdf \
	r-base-dev \
        sqlite3 \
	subversion \
	valgrind \
	vim \
	wget

RUN apt-get install -y --no-install-recommends \
	libv8-3.14-dev \
	libjq-dev \
	libprotobuf-dev \
	libxml2-dev \
	libprotobuf-dev \
	protobuf-compiler \
	unixodbc-dev \
	libssh2-1-dev \
	libgit2-dev \
	libnetcdf-dev \
	locales \
	libssl-dev

RUN locale-gen en_US.UTF-8

ENV PROJ_VERSION=6.2.0
# ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

RUN wget http://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz \
  && tar zxf proj-*tar.gz \
  && cd proj* \
  && ./configure \
  && make \
  && make install \
  && cd .. \
  && ldconfig

# install proj-datumgrid:
RUN cd /usr/local/share/proj \
  && wget http://download.osgeo.org/proj/proj-datumgrid-latest.zip \
  && unzip -o proj-datumgrid*zip \
  && rm proj-datumgrid*zip \
  && wget https://download.osgeo.org/proj/proj-datumgrid-europe-latest.zip \
  && unzip -o proj-datumgrid*zip \
  && rm proj-datumgrid*zip \
  && cd -

# GDAL:

ENV GDAL_VERSION=3.0.1
ENV GDAL_VERSION_NAME=3.0.1

RUN wget http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION_NAME}.tar.gz \
  && tar -xf gdal-${GDAL_VERSION_NAME}.tar.gz \
  && cd gdal* \
  && ./configure \
  && make \
  && make install \
  && cd .. \
  && ldconfig

#RUN git clone --depth 1 https://github.com/OSGeo/gdal.git
#RUN cd gdal/gdal \
#  && ls -l \
#  && ./configure \
#  && make \
#  && make install \
#  && cd .. \
#  && ldconfig

# GEOS:
ENV GEOS_VERSION=3.7.2
#
RUN wget http://download.osgeo.org/geos/geos-${GEOS_VERSION}.tar.bz2 \
  && bzip2 -d geos-*bz2 \
  && tar xf geos*tar \
  && cd geos* \
  && ./configure \
  && make \
  && make install \
  && cd .. \
  && ldconfig

RUN install2.r --error \
    remotes

RUN R -e "remotes::install_github('geocompr/geocompkg')"

[MichaelChirico]

@Nowosad great! Is this hosted on dockerhub?

[Nowosad]

It was not... until today;)
There are two versions:

1. https://hub.docker.com/r/jakubnowosad/rspatial_proj6 - with all external libraries (PROJ6, GDAL3, GEOS)
2. https://hub.docker.com/r/jakubnowosad/geocompr_proj6 - the same as above + many R packages for geocomputation (https://github.com/geocompr/geocompkg/blob/master/DESCRIPTION)

[rsbivand]

This is the WIP-vignette from R-Forge/rgdal 1.5-1 rev 888. Comments welcome!
PROJ6_GDAL3.zip

[edzer]

DO make the WKT representations readable using cat and pretty, as in

> library(sf)
Linking to GEOS 3.7.1, GDAL 2.4.2, PROJ 5.2.0
> st_as_text(st_crs(4326))
[1] "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]"
> st_as_text(st_crs(4326), pretty = TRUE)
[1] "GEOGCS[\"WGS 84\",\n    DATUM[\"WGS_1984\",\n        SPHEROID[\"WGS 84\",6378137,298.257223563,\n            AUTHORITY[\"EPSG\",\"7030\"]],\n        AUTHORITY[\"EPSG\",\"6326\"]],\n    PRIMEM[\"Greenwich\",0,\n        AUTHORITY[\"EPSG\",\"8901\"]],\n    UNIT[\"degree\",0.0174532925199433,\n        AUTHORITY[\"EPSG\",\"9122\"]],\n    AUTHORITY[\"EPSG\",\"4326\"]]"
> cat(st_as_text(st_crs(4326), pretty = TRUE), "\n")
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
>

(misses the final newline, but nevertheless)

[rsbivand]

Suggestion: do it yourself and commit to svn. I'm busy, and don't think appearances matter enough. Roger Bivand Falsensvei 32 5063 Bergen tir. 12. nov. 2019, 19.22 skrev Edzer Pebesma <notifications@github.com>:

…

DO make the WKT representations readable using cat and pretty, as in > library(sf) Linking to GEOS 3.7.1, GDAL 2.4.2, PROJ 5.2.0 > st_as_text(st_crs(4326)) [1] "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]" > st_as_text(st_crs(4326), pretty = TRUE) [1] "GEOGCS[\"WGS 84\",\n DATUM[\"WGS_1984\",\n SPHEROID[\"WGS 84\",6378137,298.257223563,\n AUTHORITY[\"EPSG\",\"7030\"]],\n AUTHORITY[\"EPSG\",\"6326\"]],\n PRIMEM[\"Greenwich\",0,\n AUTHORITY[\"EPSG\",\"8901\"]],\n UNIT[\"degree\",0.0174532925199433,\n AUTHORITY[\"EPSG\",\"9122\"]],\n AUTHORITY[\"EPSG\",\"4326\"]]" > cat(st_as_text(st_crs(4326), pretty = TRUE), "\n") GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4326"]] > (misses the final newline, but nevertheless) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ACNZ3BELABVSHPA6HMBLTRDQTLX6JA5CNFSM4IVRDCLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED3JLTY#issuecomment-553031119>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNZ3BFJH27CCQ6XG6TOD3TQTLX6JANCNFSM4IVRDCLA> .

[rsbivand]

All I need is an indication if someone else is attacking this problem and can see obviously better resolutions, or indications (thanks @Nowosad !) of checks on the current state of play. I do not need comments about aesthetics, they do not matter for the task in hand. I'll wrap the output, but I will not change to multiline out from exportToWkt(), because the idea is to present a string that is machine readable. Unless it can be shown that WKT needs to be human-readable (worse editable), I'll keep then as single line strings.

[Robinlovelace]

Just discovered this thread, great to see proactive work to preempt proj6. Just testing my package stplanr with the following command (in case of use/interest to others):

docker run -d -p 8789:8787 -e DISABLE_AUTH=TRUE -v $(pwd):/home/rstudio/  jakubnowosad/geocompr_proj6

[mdsumner]

what are we to use for testing? github.com/rsbivand/sp@master and rgdal master on r-forge?

Correct?

#remotes::install_github("rsbivand/sp")
#install.packages("rgdal", repos="http://R-Forge.R-project.org")

[Robinlovelace]

👍 that's what I did, as documented here: ropensci/stplanr#364 (comment)

[mdsumner]

cool thanks, I'm failing to install rgdal with PROJ 6.1.0 atm but I consider that tangential, got the geocompr_proj6 running now