[TRACKING] Open Street Map

[parkan]

Tracking for https://github.com/okdistribute/devgrants/blob/25eb4682b305189121b8ae35472bf86a06bf59e5/targeted-grants/open-street-map-ipfs.md

@okdistribute do you have updates on this project?

We have the database working in the browser now on the wasm-async branch:

https://github.com/peermaps/eyros/blob/wasm-async/pkg/main.js

There is still some work to do making the file format more amenable to updates over ipfs since an appended-to file gets a new hash, so I am working on making a file-based linked list format for more efficiently transmitting updates of map data from an existing partial cache.

I'm also working on a scheme to ensure that ingested blocks end up in sufficiently geographically compact pages. Right now the block scheme is sensitive to the locality of ingested features, but there is an opportunity during tree rebuilding and block creation to perform swaps so that the regions expressed by blocks are more compact and the tree is more balanced. This is important so that clients will receive pages local to the bounded region they are looking at.

[okdistribute]

Thanks @parkan and @substack for the update! We're currently working on Deliverable 1 "v1 of the peermaps/ingest library in Rust"

[okdistribute]

I just pushed up a first pass of georender-pack in Rust, which encodes pbf files into a smaller format, currently called peermaps/bufferschema.

Next step is to write some more tests to make sure it's compatible with the existing Node.js implementation, and then integrate it into peermaps/ingest which will be the commandline tool for converting pbf files into an on-disk format. Then I'll look at the ipfs and filecoin apis to see how to best push the files onto the network and record usability notes along the way.

[thibaultmol]

Maybe get into contact with the osm ops team. They might be able to help as well to make this happen https://twitter.com/OSM_Tech/status/1308465290263629825
They've been running the official osm tiles server and are seeking help for hosting. IPFS hosting would be a great solution.

(ps: If it'd be possible to do ipfs clustering or whatever the final solution will be. Please make it something that it's possible to configure the ipfs node to only pin a section of it. So that people who want to contribute some bandwidth and storage but only want to dedicate x amounts of GB's have that option)

[Firefishy]

I am the person who put the @OSM_Tech tweet out. The tile CDN renders on demand, for tile layer 0 to maybe zoom 14 the cache hit ratio is high, but decreases rapidly to zoom 19 where the majority of requests are rendered on demand (~1% hit ratio). This tool is handy for working out the number of map tiles there are: https://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox=-144.227563,-75.111889,233.16404,81.381733 and the likely 3TB storage required over 16.5 billion tiles.

[okdistribute]

Hi all! Thanks for chiming in with all this interesting information and links to existing conversation surrounding maptiles. Just want to give a status update --

We're doing preliminary testing on small metro areas and working on improving the performance for parsing and packing the full planet osm into the peermaps format. We're a bit behind schedule and it's hard to say if we'll have a full working demo of the full planet parsed by the end of the year; the goal is to get it from OSM dump to updated on the peer network daily, but that takes a significant lift of algorithms work.

[autonome]

@okdistribute happy new year! Can you share where things are at?

[okdistribute]

Hi @autonome !

We have made good progress on the ingest cli tool, and the code can be found here. We are blocked currently on a full demo with planet.osm, as the eyros database needs some more bug fixes and tweaks from @substack, including:

• NaN bug @ 50m records
• updating the wasm code and making sure that still works
• double-checking read profile on queries with different input distributions

However, I have enough to work with the current version of the database to compile a report about feasibility and usability when building on both IPFS and Filecoin, just on a smaller metro area. I'll be working on that over the next two weeks (less than part-time) and then report back

[okdistribute]

Hi @autonome , @substack is still working diligently on the database but it's almost ready for primetime. The peermaps ingest pipeline has processed 6.04 billion records in 84h39m, and counting.

Once this is complete the team will create some demos, put it on the network, and finish up the grant. Thanks so much for your patience during this as the team worked out the kinks with the underlying database.

[autonome]

6.04 billion 😱

[whyrusleeping]

@okdistribute is there a way I can pull a copy of the dataset? I have sufficient disk space and bandwidth.