Performance: single (default) RocksDB vs ColumnFamilies

[magicprinc]

If I use RocksDB without ColumnFamilies ~ db.get(key)
then the throughput is amazing 100_000+ op/sec

If I use ColumnFamilies ~ db.get(cfh, key)
then the throughput drops to 1000..10_000 op/sec

After very long consultations with AI, I made such configuration:

dbOptions = new DBOptions()
	.setCreateIfMissing(true)
	.setCreateMissingColumnFamilies(true)
	.setMaxBackgroundJobs(Runtime.getRuntime().availableProcessors() * 2)
	.setUseFsync(false)
	.setUseDirectIoForFlushAndCompaction(true)

tableConfig = new BlockBasedTableConfig();

Cache blockCache = new LRUCache(512 * 1024 * 1024L, 8); // 512MB block cache, 8 shards
tableConfig.setBlockCache(blockCache);
tableConfig.setBlockCacheSize(512 * 1024 * 1024);//<^?

tableConfig.setCacheIndexAndFilterBlocks(true);
//tableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true);
tableConfig.setBlockSize(16 * 1024);// Default: 4K
tableConfig.setFilterPolicy(new BloomFilter(10, false));
//tableConfig.setPartitionFilters(true); ? 40+ mi keys?
tableConfig.setMetadataBlockSize(4096);
return tableConfig;

options = new ColumnFamilyOptions()
	.optimizeLevelStyleCompaction()
	.setCompressionType(CompressionType.ZSTD_COMPRESSION)
	.setBottommostCompressionType(CompressionType.ZSTD_COMPRESSION)
	.setOptimizeFiltersForHits(true)
	.setWriteBufferSize(writeBufferSize * 1024 * 1024)// 128 MB per CF
	.setForceConsistencyChecks(true)
	.setMaxWriteBufferNumber(4)
	.setMinWriteBufferNumberToMerge(2);
	.setTableFormatConfig(createTableConfig);

It is still noticeably slower than no-ColumnFamilies/single-default RocksDB.

Any ideas or tips? 😭🙏

[magicprinc]

Bug/feature:
new tableConfig.setBlockCache(blockCache) DOESN'T work (very slow speed)

old deprecated tableConfig.setBlockCacheSize(512 * 1024 * 1024) works!

Nevertheless it is still slower as no-ColumnFamilies RocksDB usage

[dkan-nyansa]

How many column families does your application actually use? Each CF uses memory resource and will reduce block cache if CF number is high enough.

[magicprinc]

How many column families does your application actually use? Each CF uses memory resource and will reduce block cache if CF number is high enough.

3 😭

[dkan-nyansa]

Please share your code on how you're opening DB and calling db.get(cfh, key). Share key count per CF an average key size and value size.

Also setBlockCacheSize() gives you 3x more total cache (one per CF), while setBlockCache() correctly shares one cache across CFs.

[magicprinc]

Here is the demo
https://github.com/magicprinc/NatsDemo/blob/rocksdb/src/main/java/examples/rocksdb/RocksDBConfig.java

It shows: "old" style cache config is almost twice time faster than "new" style config

Use -DblockCache=true for new

/// "old" t =  64_588 ~ 77_413,76 op/s
/// "new" t = 12_6040 ~ 39_669,95 op/s

[dkan-nyansa]

It's not surprising your "old" is twice as fast because the "new", which uses setBlockCacheSize, will eventually trigger newTableFactoryHandle() 3 times.

3 ColumnFamilyDescriptors
→ 3 createColumnFamilyOptions() calls
→ 3 .setTableFormatConfig(tableConfig) calls
Each setTableFormatConfig() calls tableConfig.newTableFactoryHandle()

As a result, 3 caches of 512MB will be created with old, where as your "new" only shares 512MB amongst the 3 CFs. If you want equivalent performance, you need to make your LRUCache 3 times as big, 3 x 512MB.

[magicprinc]

But! As you can see the only one cf is used in test! So the whole cache can be used in test

[dkan-nyansa]

Fair. I ran your demo with and without -DblockCache=true. I don't see any difference on my local macOS dev machine with the following version

org.rocksdb:rocksdbjni:10.+ -> 10.8.3
OpenJDK 25.0.2 (Temurin-25.0.2+10-LTS)

Not sure if the shard count matters. You have
final LRUCache blockCache = new LRUCache(512 * 1024 * 1024L, 16); // 512MB block cache, 8 shards

16 is numShardBits. You're sharding 512MB into 65,536 shards; each shard is 8kB.