ipfs and pacman

[jbenet]

notes integrating with pacman here

[robcat]

Relevant thread on Arch Linux forums: https://bbs.archlinux.org/viewtopic.php?id=203853

[jbenet]

@robcat thanks! let us know how we can help. we'd love to contribute to making this easy + nice. cc @whyrusleeping

[robcat]

(adding @anatol, Arch Linux developer that has shown interest on the forums)

So, in general ipfs can help in many ways the package distribution (distributed storage, versioning, package signing).

The low hanging fruit is the distributed storage: given a "package entry" in the pacman database, get directly the package from ipfs by hash.
A custom downloader can be already plugged in pacman using the XferCommand configuration variable (some examples)

The big problem is the following: the pacman database includes only SHA256 and MD5 hashes, and no ipfs-style multihashes (that unfortunately cannot be constructed using the already included hashes).

My plan is to build a custom XferCommand script that, querying some kind of service, translates the SHA256 hash into a "standard" ipfs hash, and then does ipfs add -o <cache-path> <ipfs-hash>

This hash translation service can be centralized in this initial stage, but it adds a constant lag to each package download (the next step should be to build and distribute a package database with ipfs-multihashes).

I'm still in search of a more elegant solution that doesn't require a central translation service, ideas are welcome

[fazo96]

A possible solution could be creating a dag with all packages: For example having the repo mirror servers add everything to ipfs and provide the hash of the folder, then deduplication takes care of efficiency and the pacman client only needs that one hash of the latest version of all the packages and the .db repository files 👍

For this to work smoothly we would probably need to patch pacman. Or maybe a tool that wraps pacman but uses IPFS to get the data could be built.

[robcat]

@fazo96 About deduplication: I downloaded the pool of archlinux packages and tried both the chunking algorithms provided by ipfs (fixed blocks and rabin). Apparently there is no detectable deduplication in my case in either case. Can you suggest an effective chunking strategy for xz compressed packages?

About the dag of packages: cool idea, but it requires to manage a fat central mirror that regularly rsyncs and compiles a new dag at every update. Unfortunately I don't have such a server available (but maybe an existing Arch mirror could be interested in running the ipfs daemon?).

[fazo96]

@robcat having an existing mirror run an ipfs daemon would be ideal: by mounting its IPNS publication using fuse, it could store the data directly in IPFS while also being able to read it from the file system and when it updates the files it would automagically update what's on IPFS.

About deduplication, there's probably no trivial way to apply it inside packages but at least two copies of the same package will produce the same hash and thus will become the same file in IPFS, so that the pacman client can get a package from any IPFS node that has a copy without having any configuration telling it the nodes' ip addresses.

This way choosing the best mirror, ranking them etc will not be needed: ipfs will be in charge of downloading the package from the best location. This will mean that updating even when an Internet connection is not there could be implemented if there's a reachable computer that is serving the packages and .db files.

[jbenet]

@fazo96 About deduplication: I downloaded the pool of archlinux packages and tried both the chunking algorithms provided by ipfs (fixed blocks and rabin). Apparently there is no detectable deduplication in my case in either case. Can you suggest an effective chunking strategy for xz compressed packages?

Are they tarballs? Did you try ipfs tar? it imports tars smartly.

@whyrusleeping what is the tar branch that detects tarballs on add?

[robcat]

@jbenet

Did you try ipfs tar? it imports tars smartly.

Cool feature! I didn't know about that. (at the moment it doesn't seem to work for some tars, I'm opening a separate issue)

In the Arch Linux case it's unfortunately not very useful, since the packages are signed only after compression. Distributing the packages in uncompressed form would mean:

• having to recompress the package at installation time to verify the signature (at a great CPU cost)
• guessing which compression settings were used by the developers on their machines, in order to reproduce the same exact tar.xz file

[fazo96]

Well, packages aren't that heavy. I mean, the biggest package I ever downloaded is probably netbeans which is less than 300 MB, and it's wayyyyyy bigger than the average package which is like a few megabytes (quanitity pulled out of thin air).

I don't think there are many packages that share common data except multiple versions of the same package. It would be very nice to figure out how to take advantage of deduplication with the current way pacman packages stuff, but even without that, an IPFS transport for pacman would be a huge step forward in terms of efficiency and repository management.

Just having the official repository publish the hash of the latest version of the packages folder and all the mirrors just automatically pinning that when it updates would propagate changes pretty fast and no one would have to worry about choosing the fastest mirror except bitswap developers 👍

To sum it up my humble suggestion is not to try too hard to fix this problem (applying deduplication to arch packages) now. IPFS is still very useful in this use case, even without that, and in the future pacman could consider packaging stuff differently if IPFS becomes the standard for distribution.

[robcat]

Just having the official repository publish the hash of the latest version of the packages folder and all the mirrors just automatically pinning that when it updates would propagate changes pretty fast and no one would have to worry about choosing the fastest mirror except bitswap developers

@fazo96
I hear you, this would be a pretty good solution.

But the problem is, to follow your plan would mean to convince the central official mirror to:

• hash the whole repository using ipfs
• manage and run an ipfs node (at least to initially seed the new packages)
• use twice the storage space (to store the ipfs blocks of the whole repository)
• sign and publish the ipns periodically (right now there's no central authority, only individual packages are signed)

There is a problem of incentives here: why should the central authority to do all this, if nobody is already using ipfs?

[pbronez]

convince the central official mirror

@robcat shouldn't any mirror work, as long as IPFS users trust it? Anyone could set up a mirror to serve as the bridge to IPFS. Then once people start using it, down the line the official repo could take over that function.

Setting up a new mirror to copy packages into IPFS is closely related to the archival efforts discussed at ipfs-inactive/archives#5

[fazo96]

manage and run an ipfs node (at least to initially seed the new packages)

It's not hard at all to do, there are also docker images.

use twice the storage space (to store the ipfs blocks of the whole repository)

You can mount IPNS with FUSE so that you don't need twice the space

sign and publish the ipns periodically (right now there's no central authority, only individual packages are signed)

Well, if the packages are all in one folder, and that folder is inside the IPNS mountpoint (using FUSE), when a file changes everything gets republished. It's not the best of solution but it's worth a shot (not by the official mirror maintainers of course, but by some interest third party)

@robcat shouldn't any mirror work, as long as IPFS users trust it? Anyone could set up a mirror to serve as the bridge to IPFS. Then once people start using it, down the line the official repo could take over that function.

Yes, that's the point! 👍 I don't see why a random guy can't set up an IPFS mirror (except the actual hassle of downloading every package). Also no need to trust IPFS or the mirror since the packages are signed, if my understanding is correct.

If somebody has the resources needed to set up an IPFS mirror of the arch repositories and some people manage to start using it, it will be a lot easier to get more adoption. I could do it, but I only have 1 Mbit upload bandwidth, for four people and at least 6 devices...

[robcat]

I could do it, but I only have 1 Mbit upload bandwidth, for four people and at least 6 devices...

Ok, so what about this three pronged approach:

• @fazo96 publishes the ipns entry (minimal bandwidth required)
• I keep a non-authoritative mirror with a lot of bandwidth that will sync the most requested packages
• users will put https://ipfs.io/ipns/<faso96-hash>/ at the top of their mirrorlist

My node is already up at 178.62.202.191 and it independently adds all the x86_64 packages from core, extra, multilib, testing and multilib-testing (it syncs to an official arch mirror and adds the packages to ipfs).

@fazo96 you can go ahead and publish the ipns entry, if IPFS works the bulk of the requests will not even hit your node :)

[fazo96]

@robcat I'm very interested in trying this out! :)

However:

• I need the hash to be able to publish something
• You can publish it yourself if you want, or you can mount ipns with FUSE and drop the root of the mirror in <mountpoint>/local so that they are automatically published on your node's IPNS (this is the suggested approach) while also being accessible on the filesystem thus not taking up twice the space! Try publishing the hash of the repo before doing this so that the files will already be there
• If you want to quickly publish to ipns, cd to the root of the repo then run ipfs name publish $(ipfs add -r -q -w . | tail -n1). It will publish the folder you are in and tell you the IPNS name and the hash. You can then tell us the IPNS name or the IPFS hash so we can try 👍
• users shouldn't use ipfs.io as the gateway or all the traffic will have to come from ipfs.io to the users and it doesn't make much sense: they would have to use their local gateway (localhost)
• I have quite some credit on digital ocean thanks to github's student developer pack, and I'm willing to set up a mirror for pacman using IPFS 👍
• I don't know if there's a way to disable the pacman cache, but I hope there is because using this approach would make it a waste of disk space except for downgrading.

Also, another use case for this is local package sharing: I have two computers in LAN and a shitty bandwidth to the internet, and all available solutions to share packages in LAN are pretty ugly compared to this.

If you want we can meet on freenode (I'm Fazo) later today or some other time to try this out 👍 If we succeed (we probably will) we can set up a page on the arch wiki or a github repo that explains how to use IPFS with pacman this way (without any external tools except ipfs)

[jbenet]

If you have troubles with IPNS bug me directly. we'll improve the perf to fix it for you.