HDDS-8131. Add Configuration for OM Ratis Log Purge Tuning Parameters.

[ivandika3]

What changes were proposed in this pull request?

Currently Ozone Manager enables raft.server.log.purge.upto.snapshot.index by default.
However, for OM cluster with large metadata store, there might be a case where OM leader purge its Ratis logs before a slow follower replicated it to its log. This means that the follower needs to download the whole metadata store from the OM leader. This can be problematic if the metadata store in leader is too large.

We should add two configurations in OM to enable/disable Ratis purge parameters:

• raft.server.log.purge.upto.snapshot.index
  • Disabling this would guarantee that the OM leader will not purge its Ratis log unless all the logs have been replicated to all the followers (through commitIndex).
  • This would effectively means that there shouldn't be a case where the slow follower needs to download the full metadata from the leader. So no snapshot download from follower. For small OM metadata, it can be faster for follower to download the leader's metadata snapshot than normally replicating and applying the outstanding logs.
  • For a very slow follower / downed follower, the OM leader cannot purge the log until the follower catch up to it. This might increase the disk space usage for OM leader.
  • Default would be true to preserve the current OM snapshot behavior
• raft.server.log.purge.preservation.log.num
  • RATIS-1626 introduces logic to preserve the latest n won't-be-purged logs
  • Setting n > 0 while still enabling raft.server.log.purge.upto.snapshot.index should balance a between the cost of preserving & transferring logs and the cost of transferring snapshot.
  • Default would be 0 to preserve the current OM snapshot behavior

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8131

How was this patch tested?

Should have already be covered in Ratis test.

[ivandika3]

Hi @Xushaohong @szetszwo could you take a look?

[ivandika3]

Hi @prashantpogde @hemantk-12 @GeorgeJahad I see that there is already an effort to introduce incremental checkpoint in the OM snapshot process. Our cluster is currently encountering issue in which a slow OM follower has to download a large OM metadata due to the leaders' log being purged. This merge request seeks to circumvent this issue by disabling raft.server.log.purge.upto.snapshot.index so that leader will only purge the log once the followers have replicated the log. Could you take a look?

However this has a risk in which the OM leader disk space could be filled quickly if the follower is down / very slow. Therefore I think the long-term solution would be to integrate incremental checkpoint. May I know what is the progress on the feature? We might be interested in integrating it in our cluster.

[szetszwo]

+1 the change looks good.

[ivandika3]

@szetszwo Thank you for the review and merge.