Installation guides don't work

[SomeoneElseOSM]

There is https://wiki.openstreetmap.org/wiki/Overpass_API/Installation#Option_2:_Installation_via_bleeding_edge_dev_version_(expert_use) , but that says "The following steps were successfully tested on Ubuntu 14.04 and debian 7.0" so clearly is only of use to software archaeologists.

There is also https://overpass-api.de/full_installation.html which is I presume is based on https://github.com/drolbr/Overpass-API/blob/master/src/html/full_installation.html , but that is not up to date as it says to run "./configure", which is not part of the software that was cloned from github.

There is also https://github.com/drolbr/Overpass-API/tree/master/build , which is empty.

What instructions should I follow to build from a clone of https://github.com/drolbr/Overpass-API on a Debian 13 VPS?

[mmd-osm]

Download the latest release from https://dev.overpass-api.de/releases/ instead of trying to clone from GitHub.

[SomeoneElseOSM]

and then?

[mmd-osm]

https://wiki.openstreetmap.org/wiki/User:ZeLonewolf/Overpass_Installation_Guide is fairly comprehensive and should get you started.

[SomeoneElseOSM]

That's basically where I'd got to, but it does say at the top "This guide is under construction to update for v0.7.61.1.". At https://wiki.openstreetmap.org/wiki/User:ZeLonewolf/Overpass_Installation_Guide#Download_the_Planet there's nothing about how to use an extract, but from https://wiki.openstreetmap.org/wiki/Overpass_API/Installation you can see which script it uses and from that see that it needs a .bz2 file , but https://download.geofabrik.de/bz2.html helps there ("osmium cat myfile.osm.pbf -o myfile.osm.bz2").

The bin directory contents of https://github.com/ZeLonewolf/better-overpass-scripts.git seem to be missing some things such as "update_from_dir", but they can be found in the "other" bin directory, then one populated by the build.

tail -f log/apply_osc_to_db.out 

contains things like

compute_idx_and_geometry: Node 1265708488 used in way 112873880 not found.

which is nowhere near the data extract that I loaded and has not been modified in five years (certainly not in the 23 hours since the Geofabrik extract extract time)

tail -f log/osm_base.out

is busily writing out lines that all say:

File_Error 17 File exists /opt/op/db/osm_base_shadow.lock write_start:1

Will it likely do that until the disk fills, or is that normal?

[SomeoneElseOSM]

Er - it has jut downloaded a minutely diff diff/000/050/999.osc.gz which corresponds to "timestamp=2012-10-17T20:43:02Z". The Geofabrik extract I started from was contained data up to last night, so I'd expect it to go forward from there?

Presumably there should be somewhere to tell it that "the extract that I downloaded corresponds to date XYZ" and set the starting date to there; I don't want it to try and apply minutely diffs from 2012 onwards.

[mmd-osm]

Using extracts is a bit of a special topic, where things may easily break or not work as expected. Some use cases like "attic data" don't work at all. If you're using a Geofabrik extract as a starting point, you might want to switch to daily updates from e.g. https://download.geofabrik.de/europe/united-kingdom-updates/ , rather than pulling in changes from all over the world from https://planet.openstreetmap.org/replication/minute/ that will quickly fill up your disk, and trigger lots of error messages due to missing data.

Note that when you're mixing Geofabrik extracts and Planet OSM diffs, you need to be super careful with the "replicate_id" file in the database directory. Geofabrik extracts have a different way to count diffs, which is not compatible with Planet OSM (and might result in downloading ancient data from 2012 or so).

[SomeoneElseOSM]

(a bit of an aside from the documentation, but) the challenge with using Geofabrik diffs is that they're daily. To trim minutely downloads to the desired area I've always used "trim_osc.py" from https://github.com/Zverik/regional . It can't perform magic (not everything in a minutely diff has a geometry) but it is surprisingly effective.

[mmd-osm]

I don't have any experience with trim_osc.py, and can't tell if it would work. You might hit a previously unknown issue that's breaking the import or causes queries to fail later on. Usually, this means the data is broken beyond repair, and you need to start over with a fresh database import. Be prepared to spend time on something that might fail in the end.

I had some luck running a Geofabrik-only scenario in the past, as discussed in #565. It sounds like you need to have more frequent updates, which rules out this option. Are there any alternatives? I know of SOSM's Planet / Overpass server that serve ".ch only" data with hourly updates -> https://planet.osm.ch/ and https://overpass.osm.ch/

I suppose you still need an up to date planet to generate the diffs and extracts, though I'm not familiar with how exactly the service has been set up.

[SomeoneElseOSM]

In order to get static queries working, what worked was deleting db, diffs and logs, rebooting, and then:

time bin/init_osm3s.sh /home/ajtown/data/north-yorkshire_2026-02-12T21:23:29Z.osm.bz2 "db/" "./" --meta=no

followed by manually editing "db/replicate_id" to contain "6984689" (no quotes), followed by a linefeed. I found the number manually by browsing around https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/minute/006/986/119.state.txt , but of course osmium also has functionality to do things like this automatically - see https://switch2osm.org/serving-tiles/updating-as-people-edit-pyosmium/#Initialising-replication .

Then

bin/osm3s_query 
<query type="nwr"><bbox-query n="54.0" s="53.98" w="-1.09" e="-0.95"/><has-kv k="amenity" v="pub"/></query><print/>
^d

worked

The next bit is Apache. I don't want to run Overpass Turbo locally, but do want to point https://overpass-turbo.eu/ at my Overpasss API server. https://wiki.openstreetmap.org/wiki/User:ZeLonewolf/Overpass_Installation_Guide , which you referred to above, has an apache http (but not https) config in it with,

DocumentRoot /opt/overpass-turbo/dist

in it (which does not exist). Presumably I will need to set up https properly (easy to do) but will also need a different apache config - is there an example of one that works anywhere?

[mmd-osm]

For a quick test, you could use the http version of overpass turbo over at: http://overpass-turbo.eu/
It has some limitations, like no support for loading/storing queries on osm.org.

I'm not sure if a "Let's encrypt" certificate would work for you. Usually, you would have to add another Virtual Host on port 443 in /etc/apache2/sites-enabled/... with SSLEngine On, SSLCertificateFile & SSLCertificateKeyFile set, etc. This should be all fairly standard Apache config, and you should find plenty of blog posts, etc. out there on the web. Calling the Overpass CGI executable files from Apache should be the same as in the HTTP/port 80 config.

[SomeoneElseOSM]

To answer part of my own question here, https (via acme.sh) "just worked". The problem with the apache config in https://wiki.openstreetmap.org/wiki/User:ZeLonewolf/Overpass_Installation_Guide#Web_Server_Configuration is that half of that is relevant to Overpass Turbo (and therefore Overpass API "just doesn't work" at the place you would logically test it up in the document) , but the second half does. The (undocumented) URL to configure an external Overpass turbo with is "https://myservername/api/" - the "/api" must be somehow implemented via cgi.

[SomeoneElseOSM]

I'm currently figuring out how I need to change calls to "apply_osc_to_db.sh" to get that to work. The usage information for that script includes:

--meta:        Metadata handling mode (attic=full history, yes=metadata, no=current only)

I have half an idea what "attic" might mean in terms of Overpass (but I might be wrong); is there a document anywhere that defines what these parameters actually mean? For example what is the difference between "--meta=no" and "--meta=yes"?

Edit: I know about https://wiki.openstreetmap.org/wiki/Attic_Data and https://wiki.openstreetmap.org/wiki/Overpass_API#Meta_data , but the second especially doesn't say exactly what queries won't work with meta=no.

[mmd-osm]

So —meta=yes covers attributes like display name, user id, changeset id, object timestamp and object version number (you can compare out; vs. out meta; output on a public instance to get an idea. In your case, these additional attributes should be simply empty).

Due to GDPR reasons, the publicly available Geofabrik extracts have most of these attributes stripped. To get the full data you need to switch to their internal download area that requires a login with your osm account.

Running a —meta=yes process with the publicly available files might even cause performance issues because the user is defaulted to 0, and the display name is blank. I don’t remember the specifics, it’s been a few years already since last trying this.

The other topic you could check out is area generation, if not done yet. This should be feasible on your extract as well.