HDDS-10926. Block deletion should update container merkle tree. by errose28 · Pull Request #6875 · apache/ozone

errose28 · 2024-06-27T17:57:06Z

What changes were proposed in this pull request?

When a block is deleted from a container, it should be marked as deleted in the container merkle tree. This tells peer datanodes that the data be skipped if they have not yet processed the deletion but find there is an error with this block. The chunk, block, and container hashes in the tree should not change.

This test also moves some existing test methods to a test util class since more tests, including those in this patch, need to use them.

What is the link to the Apache JIRA

HDDS-10926

How was this patch tested?

Unit tests added

* HDDS-10239-container-reconciliation: HDDS-10372. SCM and Datanode communication for reconciliation (apache#6506) HDDS-10239. Storage Container Reconciliation. (apache#6121)

* HDDS-10887-mt-manager: Rename data tree to container tree

...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java

...c/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

kerneltime · 2024-06-28T20:34:39Z

Will give another look post addressing of review comments and rebase.

…-delete * HDDS-10239-container-reconciliation: HDDS-10887. Implement a basic Merkle Tree Manager. (apache#6778) HDDS-10923. Container Scanner should still scan unhealthy containers. (apache#6809) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

…-delete * HDDS-10239-container-reconciliation: (184 commits) HDDS-10373. Implement framework for capturing Merkle Tree Metrics. (apache#6864) HDDS-11188. Initial setup for new UI layout and enable users to switch to new UI (apache#6953) HDDS-11120. Rich rebalancing status info (apache#6911) HDDS-11187. Fix Event Handling in Recon OMDBUpdatesHandler to Prevent ClassCastException. (apache#6950) HDDS-11213. Bump commons-daemon to 1.4.0 (apache#6971) HDDS-11212. Bump commons-net to 3.11.1 (apache#6973) HDDS-11211. Bump assertj-core to 3.26.3 (apache#6972) HDDS-11210. Bump log4j2 to 2.23.1 (apache#6970) HDDS-11150. Recon Overview page crashes due to failed API Calls (apache#6944) HDDS-11183. Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot (apache#6946) HDDS-11198. Fix Typescript configs for Recon (apache#6961) HDDS-11180. Simplify HttpServer2#inferMimeType return statement (apache#6963) HDDS-11194. OM missing audit log for upgrade (apache#6958) HDDS-10389. Implement a search feature for users to locate open keys within the Open Keys Insights section. (apache#6231) HDDS-10561. Dashboard for delete key metrics (apache#6948) HDDS-11192. Increase SPNEGO URL test coverage (apache#6956) HDDS-11179. DBConfigFromFile#readFromFile result of toIOException not thrown (apache#6957) HDDS-11186. First container log missing from bundle (apache#6952) HDDS-10844. Clarify snapshot create error message. (apache#6955) HDDS-11166. Switch to Rocky Linux-based ozone-runner (apache#6942) ...

This will be handled properly once HDDS-11254 is complete.

errose28 · 2024-07-31T17:06:03Z

For tests to pass I had to add the same metrics unregister workaround from #6945. ~~HDDS-10239~~ HDDS-11254 will properly close the checksum tree manager when everything is connected and this will no longer be necessary.

errose28 · 2024-07-31T19:25:57Z

Run on my fork is green, waiting for verification here as well.

aswinshakil

Thanks for the patch @errose28. LGTM+1

smengcl · 2024-08-06T22:17:20Z

Thanks @errose28 . Nice commit messages btw :D

…p-supervisor Merge conflicts are resolved but the change does not yet build. * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

…rrupt-files * HDDS-10239-container-reconciliation: (183 commits) HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945) HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024) HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043) HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040) HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036) HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003) HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724) HDDS-10926. Block deletion should update container merkle tree. (apache#6875) HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021) HDDS-11272. Statistics some node status information (apache#7025) HDDS-11278. Move code out of Hadoop util package (apache#7028) HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026) HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032) HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990) HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023) HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007) HDDS-11224. Increase hdds.datanode.handler.count (apache#7011) HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012) HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

errose28 added 19 commits June 5, 2024 21:48

Initial outline of checksum manager and merkle tree with proto

bf0dd6e

Merge branch 'HDDS-10239-container-reconciliation' into mt-manager-wip

408cff9

* HDDS-10239-container-reconciliation: HDDS-10372. SCM and Datanode communication for reconciliation (apache#6506) HDDS-10239. Storage Container Reconciliation. (apache#6121)

Add config key for lock stripes

3adbe5a

Keep deleted block list sorted

e02a609

Change chunk checksum proto, add the first passing test

3680a2e

Use config defaults for test setup

0d5800c

Finish tests for ContainerMerkleTree

eb01da4

Add tests for checksum maanger and standardize proto names

f35effd

Updates after reviewing diff

21dd870

Rename checksum manager and file. Fix findbugs and Rat

47cd213

Make checksum file read/write for the API based on files, not protos

6460fe7

Initially mark where blocks will be added

ff55f27

Block deleting task supports updating the file

fffeba6

Basic block delete test passes

4822941

Add test for block delete commands retried

6087fa3

Rename data tree to container tree

9714259

Merge branch 'HDDS-10887-mt-manager' into HDDS-10926-mt-delete

aacac83

* HDDS-10887-mt-manager: Rename data tree to container tree

Use generic container type in checksum manager

3161ad8

Checkstyle

041c5c2

github-actions bot added the container-reconciliation label Jun 27, 2024

kerneltime reviewed Jun 28, 2024

View reviewed changes

...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java Outdated Show resolved Hide resolved

kerneltime reviewed Jun 28, 2024

View reviewed changes

...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java Show resolved Hide resolved

kerneltime reviewed Jun 28, 2024

View reviewed changes

...c/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java Outdated Show resolved Hide resolved

errose28 commented Jul 23, 2024

View reviewed changes

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java Show resolved Hide resolved

errose28 mentioned this pull request Jul 26, 2024

HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. #6945

Merged

errose28 added 3 commits July 26, 2024 13:40

Use block name in task

245844c

Update test description

51e7020

errose28 added 5 commits July 30, 2024 15:32

Move test helper methods to util class

1748205

Add more tests for blocks being written to the file

5d24704

Checkstyle

0145809

Fix rat

52207ce

Add metrics unregister workaround from WIP HDDS-10376

a82c863

This will be handled properly once HDDS-11254 is complete.

errose28 marked this pull request as ready for review July 31, 2024 19:25

kerneltime approved these changes Aug 5, 2024

View reviewed changes

aswinshakil approved these changes Aug 6, 2024

View reviewed changes

errose28 merged commit d7f302e into apache:HDDS-10239-container-reconciliation Aug 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-10926. Block deletion should update container merkle tree.#6875

HDDS-10926. Block deletion should update container merkle tree.#6875
errose28 merged 28 commits intoapache:HDDS-10239-container-reconciliationfrom
errose28:HDDS-10926-mt-delete

errose28 commented Jun 27, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kerneltime commented Jun 28, 2024

Uh oh!

Uh oh!

errose28 commented Jul 31, 2024 •

edited

Loading

Uh oh!

errose28 commented Jul 31, 2024

Uh oh!

aswinshakil left a comment

Uh oh!

smengcl commented Aug 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

errose28 commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kerneltime commented Jun 28, 2024

Uh oh!

Uh oh!

errose28 commented Jul 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

errose28 commented Jul 31, 2024

Uh oh!

aswinshakil left a comment

Choose a reason for hiding this comment

Uh oh!

smengcl commented Aug 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

errose28 commented Jun 27, 2024 •

edited

Loading

errose28 commented Jul 31, 2024 •

edited

Loading