HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test

[timmylicheng]

What changes were proposed in this pull request?

1. Fix pipeline allocation in non-topology env and prevent pipelines from sharing the same set of datanodes.
2. Add metrics and logs to record when pipeline policy chooses two same set of datanodes for different pipeline for future reference.
3. Add tests for basic reports and mock what pipeline allocation would do given different number of nodes.

(Please fill in changes proposed in this fix)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-1574
(Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)

Please replace this section with the link to the Apache JIRA)

How was this patch tested?

UT

(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)

[xiaoyuyao]

LGTM overall, a few minor issues commented inline.

[timmylicheng]

The acceptance test is fixed in https://issues.apache.org/jira/browse/HDDS-2650

[timmylicheng]

testCloseContainerEventWithRatis doesn't see the same error locally as on CI.

[xiaoyuyao]

Thanks @timmylicheng for the update. The latest change LGTM.

Wrt. the failures in testCloseContainerEventWithRatis. I spend some time and debug it.
After we allow background pipeline thread to create more than one pipelines per datanode, the OZONE_SCM_PIPELINE_NUMBER_LIMIT should always be set properly in the test. Otherwise, this will run indefinitely and timeout the even queue processing for some of those with tight timeout (e.g., 1s in this case). Also, when the limit is reached, we throw exceptions which ends up with a lot of false alarm error logs and the pipeline failure counter incremented. We can fix that in follow up JIRAs.

Here is a proposed fix for testCloseContainerEventWithRatis, which you can include in this PR.

TestCloseContainerEventHandler.java

@@ -24,6 +24,7 @@
 import org.apache.hadoop.hdds.HddsConfigKeys;
 import org.apache.hadoop.hdds.protocol.DatanodeDetails;
 import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
 import org.apache.hadoop.hdds.scm.TestUtils;
 import org.apache.hadoop.hdds.scm.pipeline.MockRatisPipelineProvider;
 import org.apache.hadoop.hdds.scm.pipeline.PipelineProvider;
@@ -67,22 +68,25 @@ public static void setUp() throws Exception {
         .getTestDir(TestCloseContainerEventHandler.class.getSimpleName());
     configuration
         .set(HddsConfigKeys.OZONE_METADATA_DIRS, testDir.getAbsolutePath());
+    configuration.setInt(
+        ScmConfigKeys.OZONE_SCM_PIPELINE_NUMBER_LIMIT, 16);
+
     nodeManager = new MockNodeManager(true, 10);
     eventQueue = new EventQueue();
     pipelineManager =
         new SCMPipelineManager(configuration, nodeManager, eventQueue);
     PipelineProvider mockRatisProvider =
         new MockRatisPipelineProvider(nodeManager,
-            pipelineManager.getStateManager(), configuration);
+            pipelineManager.getStateManager(), configuration, eventQueue);
     pipelineManager.setPipelineProvider(HddsProtos.ReplicationType.RATIS,
         mockRatisProvider);
     containerManager = new
         SCMContainerManager(configuration, nodeManager,
         pipelineManager, new EventQueue());
-    pipelineManager.triggerPipelineCreation();
     eventQueue.addHandler(CLOSE_CONTAINER,
         new CloseContainerEventHandler(pipelineManager, containerManager));
     eventQueue.addHandler(DATANODE_COMMAND, nodeManager);
+    pipelineManager.triggerPipelineCreation();
     // Move all pipelines created by background from ALLOCATED to OPEN state
     Thread.sleep(2000);
     TestUtils.openAllRatisPipelines(pipelineManager);
@@ -93,6 +97,9 @@ public static void tearDown() throws Exception {
     if (containerManager != null) {
       containerManager.close();
     }
+    if (pipelineManager != null) {
+      pipelineManager.close();
+    }
     FileUtil.fullyDelete(testDir);
   }

[timmylicheng]

• if (pipelineManager != null) {

•  pipelineManager.close();
  

• }

Thanks for the efforts!
Looks like this test is not caught in previous PR when I enabled PipelinePlacementPolicy. I'm able to finish the test locally now.

[xiaoyuyao]

Thanks @timmylicheng for the update. The latest change LGTM, +1. Not sure why the acceptance test is failing here. Will take another look tomorrow before merge.

Also, can you open two follow up JIRAs for the other issues?

[timmylicheng]

Thanks @timmylicheng for the update. The latest change LGTM, +1. Not sure why the acceptance test is failing here. Will take another look tomorrow before merge.

Also, can you open two follow up JIRAs for the other issues?

Are you referring to:

1. Use either OZONE_SCM_PIPELINE_NUMBER_LIMIT or OZONE_DATANODE_MAX_PIPELINE_ENGAGEMENT to limit pipeline creation for all tests.

2. Use other method to notify pipeline creation failure due to exceeding max limit than throwing an exception?

[timmylicheng]

Thanks @timmylicheng for the update. The latest change LGTM, +1. Not sure why the acceptance test is failing here. Will take another look tomorrow before merge.

Also, can you open two follow up JIRAs for the other issues?

https://issues.apache.org/jira/browse/HDDS-2756 is creating to track logging issue.

[xiaoyuyao]

bq. Use either OZONE_SCM_PIPELINE_NUMBER_LIMIT or OZONE_DATANODE_MAX_PIPELINE_ENGAGEMENT to limit pipeline creation for all tests.

Yes. I think the default value of 0 will be problematic as it will keep sending createPipeline command to DNs without restriction. A better default should be provided for it to use in production.

[timmylicheng]

OZONE_DATANODE_MAX_PIPELINE_ENGAGEMENT

https://issues.apache.org/jira/browse/HDDS-2772 is created to track this. @xiaoyuyao

[xiaoyuyao]

The acceptance test failures are unrelated tracked by HDDS-2774.