Support pebble

[lidel]

TODO

• include https://github.com/ipfs/go-ds-pebble as built-in plugin (similar to badger atm)
• update https://github.com/ipfs/kubo/blob/master/docs/config.md#profiles to include pebble
  • (TBD, sounds like) do we want to allow flatfs (for blocks)+pebble for pins? if so, we need separate profile for applying it only as leveldb replacement, and separate one for /blocks (blockstore)
• update https://github.com/ipfs/kubo/blob/master/docs/datastores.md to include pebble
• (TBD, probably a nice to have but not required) add support to https://github.com/ipfs/ipfs-ds-convert to allow conversion of existing repos?
  • we've decided to not write conversion tools, in practice to switch datastore type, people created new cluster node and transferred data to it, as it was safer and also performed GC of unpinned data for free.
• find a way to deal with ratcheting updates failing (Support pebble  #10347 (comment)) See: feat(pebble): support pinning FormatMajorVersion #10789
  • expose FormatMajorVersion and write docs in docs/datastores.md#pebbleds ?
• add warning to users running badger, nudging them to switch to flatfs or pebble
• replace leveldb with pebble in the default settings
• soft badger deprecation
  • remove badgerv1 profile, so new users won't be able to use it, unless they had preexisting config
  • detect when badgerv1 is used and refuse to start unless KUBO_ALLOW_DEPRECATED_BADGERV1=true is set in env – creates friction, forcing people to notice

Description

Summary

Include pebble as built-in plugin.
It provides meaningful alternative to leveldb as the datastore, and may be better than badger1 as well.

Background

It is 2024, and we still only have flatfs, leveldb and necromancy-level badgerv1 (!) as datastore options.

We got positive feedback about pebble, some examples:

https://ipfscluster.io/documentation/guides/datastore/

Pebble is a high performant
backend from Cochroachdb, used by default in Cluster:

• Proven to work well on very large pinsets.
• Best disk-usage compared to the rest. No need to trigger GC cycles for space reclaim.
• Performance and memory usage seems on par with Badger3, and behaves better than Badger on both counts.
• Behaves correctly with default settings but we bump them up a bit.
• 0-delay startup times, even with very large amounts of data.
• Options support compression (we chose to leave it enabled by default).
• The Pebble project is officially alive and maintained.
• Pebble only runs on 64-bit architectures.
• One key difference with Badger3 is that Pebble stores keys and values
  together and any lookup for a key will also read the values, while Badger3
  can store keys and values separately (i.e. keys only in the index, which can
  be loaded onto memory when small enough).

ipfs/go-ds-pebble#29:

After changing leveldb store to pebble store, the speed of GC has increased by at least dozens of times. So it's not flatfs that's to blame for slow GC, it's leveldb. I also tried leveldb and pebble as blockstore, but the CPU and memory usage is unacceptable.

Right now, to use go-ds-pebble one needs to build external plugin.

Proposed change

Include https://github.com/ipfs/go-ds-pebble in standard kubo build, just like we do with legacy badger1.

This will

• provide meaningful alternative to leveldb and badgerd
• allow community to use it without having to build external plugin and report back

[guojidan]

hi, I want try implement this feature 😄

[lidel]

Triage note:

• given https://ipfscluster.io/documentation/guides/datastore/ is using pebble by default these days, we are considering bringing Pebble as first-class citizen, similar to legacy badger, and remove badger over time.

[lidel]

Triage notes:

• we want to sunset badgerv1, will start warning people when they apply profile it is deprecated
• we want to enable pebble before removing badgerv1
• we need to figure out which pebble settings are worthy exposing via config, prior art in ipfs-cluster will likely be a good starting point
• we want to track release verion of pebble, to avoid the state we ended up with badgerv1