Service 'trafficops-perl' failed to build: [...] dial tcp: lookup geolite.maxmind.com: No address associated with hostname

[zrhoffman]

I'm submitting a ...

• bug report
• new feature / enhancement request
• improvement request (usability, performance, tech debt, etc.)
• other

Traffic Control components affected ...

• CDN in a Box
• Documentation
• Grove
• Traffic Control Client
• Traffic Monitor
• Traffic Ops
• Traffic Ops ORT
• Traffic Portal
• Traffic Router
• Traffic Stats
• Traffic Vault
• unknown

Current behavior:

The trafficops-perl Docker image for CDN-in-a-Box fails to build because the geolite.maxmind.com subdomain no longer resolves.

Expected / new behavior:

trafficops-perl should build successfully.

Minimal reproduction of the problem with instructions:

docker rmi cdn-in-a-box_perl;
docker-compose build trafficops-perl;

Failed build log follows:

Building trafficops-perl
Step 1/16 : FROM centos:7
 ---> 5e35e350aded
Step 2/16 : LABEL   name="Traffic Ops Perl"         maintainer="dev@trafficcontrol.apache.org"
 ---> Using cache
 ---> cfa2c7d6e139
Step 3/16 : EXPOSE 443
 ---> Using cache
 ---> f20b3afea2b4
Step 4/16 : ENV MOJO_MODE production
 ---> Using cache
 ---> 07c71701c21e
Step 5/16 : RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 &&     rpm --import https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7 &&     rpm --import https://download.postgresql.org/pub/repos/yum/RPM-GPG-KEY-PGDG &&     yum -y update ca-certificates &&     yum-config-manager --add-repo https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64 &&     yum install -y         epel-release         pgdg-redhat-latest &&     yum install -y         bind-utils         cpanminus         expat-devel         gcc-c++         gettext         git         iproute         jq         libcurl         libcurl-devel         libidn-devel         libpcap-devel         mkisofs         net-tools         nmap-ncat         openssl         openssl-devel         perl         perl-core         perl-Crypt-ScryptKDF         perl-DBD-Pg         perl-DBI         perl-DBIx-Connector         perl-Digest-SHA1         perl-JSON         perl-JSON-PP         perl-libwww-perl         perl-TermReadKey         perl-Test-CPAN-Meta         perl-WWW-Curl         postgresql96         postgresql96-devel         postgresql96-libs         tar &&    yum -y clean all
 ---> Using cache
 ---> 9c6488d6690d
Step 6/16 : ADD https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz /
ERROR: Service 'trafficops-perl' failed to build: ADD failed: Get https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz: dial tcp: lookup geolite.maxmind.com: No address associated with hostname

Anything else:

MaxMind no longer publicly hosts their downloads, instead requiring you to register (for free). See: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

This also affects the Traffic Ops post-install script.

Docs will need to be updated to remove references to geolite.maxmind.com.

[ocket8888]

Is the process no longer possible to automate?

Also, I think we added this because TR refused to start without a database. If that's (still?) true, then IMO that requirement should be relaxed, since geolocating clients via IP database lookup is not a requirement for TR to function.

[rawlinp]

Without at least a GeoIP DB, TR will always default to whatever the default lat/long is for geolocating clients (assuming no CZF either). That makes it pretty useless for testing and demo purposes.

It sounds like it can still be automated as long as you create an account and provide the license key via a qparam. So...not sure if we can just create some "shared" Traffic Control account and use that. There's probably a lot of fine print to read. The website says they reserve the right to limit the number of downloads within a limited amount of time.

[rawlinp]

BTW looking at the TR code, it looks like if there is no geolocation.polling.url configured, it will fail to process the snapshot. So basically, that parameter is required, meaning a maxmind DB is required. Relaxing that requirement to make a maxmind DB optional will likely require changes to several places in TR, and while TR is mostly useless without some kind of Geo-IP DB for geolocation, I'm leaning more towards the view of keeping it required.

[zrhoffman]

We already track the db ourself in our repo, along with its license. Any reason not to just reuse what we have?

[rawlinp]

That is a really old version of the DB, but maybe that is enough for CIAB. That is, if the new EULA for downloads doesn't apply to old downloads from before this new EULA existed.

[zrhoffman]

The latest Internet Archive backup (2019-12-27) has the same license. If we can keep the old DB, we can just as easily use this newer one from before the new EULA.

[ocket8888]

"maybe enough for CIAB" is what I was going for with my suggestion. You might not be able to get a perfectly efficient TR instance without that database, but it could still function by falling back to the same static coordinates for every client request. It's not enough to demo or test Traffic Router but it would be enough to demo and at least partially test Traffic Control. Besides, it shouldn't preclude you from using a database in such demos or tests.

We should maybe spend some time looking into that shared account - personally I like that better than having to update a 3rd party file in our own version tracking.

[rawlinp]

I think the shared account is probably a non-starter given the fine print in the new EULA. Our best bet for now is to probably just use the internet archive backup or the "test" DB we already have backed up into our repo.

[zrhoffman]

For users who still want the database, maybe we could make a config var titled "Download Maxmind Database?", use the backup if it's set to "no" (default), then require an account ID and license key if it is set to "yes". As for downloading it, maxmind/geoipupdate lays it out in relatively few lines. Maybe we could do something similar in our postinstall script, replacing the first InstallUtils::execCommand() call in setupMaxMind():

curl \
    --verbose \
    --compressed \
    --user "${MAXMIND_ACCOUNT_ID}:${MAXMIND_LICENSE_KEY}" \
    'https://updates.maxmind.com/geoip/databases/GeoLite2-City/update?db_md5=00000000000000000000000000000000' |
    gzip > /GeoLite2-City.mmdb.gz

But yeah, users would need to supply their own account ID and license key to do that.