HDDS-6960. EC: Implement the Over-replication Handler by JacksonYao287 · Pull Request #3572 · apache/ozone

JacksonYao287 · 2022-06-30T11:01:58Z

What changes were proposed in this pull request?

Implement the Over-replication Handler

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6960

How was this patch tested?

unit test

umamaheswararao

Thank you @JacksonYao287 for working on this patch. I have few comments to check. Thanks

...src/main/java/org/apache/hadoop/hdds/scm/container/replication/ECOverReplicationHandler.java

umamaheswararao · 2022-07-02T21:47:27Z

...src/main/java/org/apache/hadoop/hdds/scm/container/replication/ECOverReplicationHandler.java

+          try {
+            //the command target node should be in-service and healthy
+            return nodeManager.getNodeStatus(dd)
+                .equals(NodeStatus.inServiceHealthy());


I think we discussed this. This is not covering healthy_readonly?

...src/main/java/org/apache/hadoop/hdds/scm/container/replication/ECOverReplicationHandler.java

...test/java/org/apache/hadoop/hdds/scm/container/replication/TestECOverReplicationHandler.java

umamaheswararao · 2022-07-02T23:16:51Z

...rc/main/java/org/apache/hadoop/hdds/scm/container/replication/ECUnderReplicationHandler.java

 * commands.
 */
-public class ECUnderReplicationHandler implements UnderReplicationHandler {
+public class ECUnderReplicationHandler implements UnhealthyReplicationHandler {


Not sure we will call over replication as unhealthy. Don't have better name in my mind but how about simply ReplicationHandler ?

Since it is common class now, javadoc should represent the generic message. Currently it assumed for underReplication handler.

ReplicationHandler seems a little confusing, it might be considered as the handler to handle replication command.
so i suggest we use unhealthy just for now and change it to an appropriate name after we find a proper one. what about "nonOptimalReplicationHandler"

JacksonYao287 · 2022-07-03T01:21:49Z

@umamaheswararao thanks for the review, will fix the comments later

umamaheswararao

Thanks @JacksonYao287 for updating the patch. Patch almost looks good to me, except a small comment left.

...src/main/java/org/apache/hadoop/hdds/scm/container/replication/ECOverReplicationHandler.java

JacksonYao287 · 2022-07-04T02:37:39Z

Thanks @umamaheswararao for the review , i have update this patch according to your comments, pleas take a look

umamaheswararao · 2022-07-04T07:04:44Z

...in/java/org/apache/hadoop/hdds/scm/container/replication/AbstractOverReplicationHandler.java

+  }
+  /**
+   * Identify a new set of datanode(s) to replicate/reconstruct the container
+   * and form the SCM commands to send it to DN.


Missed to update doc?

sorry for missing , fixed now!

umamaheswararao · 2022-07-04T07:05:36Z

...in/java/org/apache/hadoop/hdds/scm/container/replication/AbstractOverReplicationHandler.java

+import java.util.stream.Collectors;
+
+/**
+ * this class holds some common methods that will be shared among


Typos: this --> This
Implementation -> implementation

umamaheswararao

LGTM

JacksonYao287 · 2022-07-05T01:40:09Z

thanks @umamaheswararao for the review! i have committed this patch to master branch

* master: (46 commits) HDDS-6901. Configure HDDS volume reserved as percentage of the volume space. (apache#3532) HDDS-6978. EC: Cleanup RECOVERING container on DN restarts (apache#3585) HDDS-6982. EC: Attempt to cleanup the RECOVERING container when reconstruction failed at coordinator. (apache#3583) HDDS-6968. Addendum: [Multi-Tenant] Fix USER_MISMATCH error even on correct user. (apache#3578) HDDS-6794. EC: Analyze and add putBlock even on non writing node in the case of partial single stripe. (apache#3514) HDDS-6900. Propagate TimeoutException for all SCM HA Ratis calls. (apache#3564) HDDS-6938. handle NPE when removing prefixAcl (apache#3568) HDDS-6960. EC: Implement the Over-replication Handler (apache#3572) HDDS-6979. Remove unused plexus dependency declaration (apache#3579) HDDS-6957. EC: ReplicationManager - priortise under replicated containers (apache#3574) HDDS-6723. Close Rocks objects properly in OzoneManager (apache#3400) HDDS-6942. Ozone Buckets/Objects created via S3 should not allow group access (apache#3553) HDDS-6965. Increase timeout for basic check (apache#3563) HDDS-6969. Add link to compose directory in smoketest README (apache#3567) HDDS-6970. EC: Ensure DatanodeAdminMonitor can handle EC containers during decommission (apache#3573) HDDS-6977. EC: Remove references to ContainerReplicaPendingOps in TestECContainerReplicaCount (apache#3575) HDDS-6217. Cleanup XceiverClientGrpc TODOs, and document how the client works and should be used. (apache#3012) HDDS-6773. Cleanup TestRDBTableStore (apache#3434) - fix checkstyle HDDS-6773. Cleanup TestRDBTableStore (apache#3434) HDDS-6676. KeyValueContainerData#getProtoBufMessage() should set block count (apache#3371) ... Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java

HDDS-6960. EC: Implement the Over-replication Handler (apache#3572) (cherry picked from commit fac23c9) Change-Id: Ic273820549544731b781b143a810ee45ef0085df

HDDS-6960. EC: Implement the Over-replication Handler

15c8283

JacksonYao287 added the EC label Jun 30, 2022

umamaheswararao reviewed Jul 2, 2022

View reviewed changes

Jackson Yao added 2 commits July 3, 2022 08:30

Merge remote-tracking branch 'origin/master' into HDDS-6960

92623a4

add missing test

602d4fa

fix comment

07719a3

JacksonYao287 requested a review from umamaheswararao July 3, 2022 05:19

umamaheswararao reviewed Jul 4, 2022

View reviewed changes

...src/main/java/org/apache/hadoop/hdds/scm/container/replication/ECOverReplicationHandler.java Outdated Show resolved Hide resolved

fix comment

d7b3e96

Merge remote-tracking branch 'origin/master' into HDDS-6960

10a1998

umamaheswararao reviewed Jul 4, 2022

View reviewed changes

Jackson Yao added 3 commits July 4, 2022 15:13

fix typo error

81bcd2c

triger CI

d1b587f

triger CI

318420a

umamaheswararao approved these changes Jul 4, 2022

View reviewed changes

JacksonYao287 merged commit fac23c9 into apache:master Jul 5, 2022

JacksonYao287 deleted the HDDS-6960 branch July 5, 2022 01:40

Conversation

JacksonYao287 commented Jun 30, 2022

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

umamaheswararao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

umamaheswararao Jul 2, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

umamaheswararao Jul 2, 2022

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 Jul 3, 2022

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 commented Jul 3, 2022

Uh oh!

umamaheswararao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JacksonYao287 commented Jul 4, 2022

Uh oh!

umamaheswararao Jul 4, 2022

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 Jul 4, 2022

Choose a reason for hiding this comment

Uh oh!

umamaheswararao Jul 4, 2022

Choose a reason for hiding this comment

Uh oh!

umamaheswararao left a comment

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 commented Jul 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants