HDDS-13288. Container checksum file proto changes to account for deleted blocks.#8649
Merged
aswinshakil merged 5 commits intoapache:HDDS-10239-container-reconciliationfrom Jun 23, 2025
Conversation
Contributor
Author
errose28
changed the title
errose28
changed the title
aswinshakil
reviewed
Jun 18, 2025
Member
aswinshakil
left a comment
aswinshakil
left a comment
There was a problem hiding this comment.
Thanks @errose28 for working on this, The changes look good to me. I have a minor comment, we can merge this once that is addressed.
...vice/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java
Outdated
Show resolved
Hide resolved
.../src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerMerkleTreeWriter.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thanks Aswin, I've updated the java field names in both instances as well. |
aswinshakil
approved these changes
Jun 20, 2025
Member
aswinshakil
left a comment
aswinshakil
left a comment
There was a problem hiding this comment.
Thanks for the update @errose28 . LGTM+1. We can merge this after a clean CI.
aswinshakil
merged commit 3867346
into
apache:HDDS-10239-container-reconciliation
81 of 82 checks passed
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jun 26, 2025
* master: (90 commits) HDDS-12468. Check for space availability for all dns while container creation in pipeline (apache#8663) HDDS-12151. Fail write when volume is full considering min free space (apache#8642) HDDS-13251. Update Byteman usage README license (apache#8700) HDDS-13251. Support dynamic Byteman scripts via bmsubmit in ozonesecure-ha (apache#8654) HDDS-12070. Bump Ratis to 3.2.0 (apache#8689) HDDS-12984. Use InodeID to identify the SST files inside the tarball. (apache#8477) HDDS-13319. Simplify KeyPrefixFilter (apache#8692) HDDS-13295. Remove jackson1 exclusions for hadoop-common (apache#8687) HDDS-13314. Remove unused maven-pdf-plugin (apache#8686) HDDS-13324. Optimize memory footprint for Recon listKeys API (apache#8680) HDDS-13240. Add newly added metrics into grafana dashboard. (apache#8656) HDDS-13309. Add keyIterator/valueIterator methods to Table. (apache#8675) HDDS-13325. Introduce OZONE_SERVER_OPTS for common options for server processes (apache#8685) HDDS-13318. Simplify the getRangeKVs methods in Table (apache#8683) HDDS-13322. Remove module auto-detection from flaky-test-check (apache#8679) HDDS-13289. Remove usage of Jetty StringUtil (apache#8684) HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification (apache#8653) HDDS-13288. Container checksum file proto changes to account for deleted blocks. (apache#8649) HDDS-13306. Intermittent failure in testDirectoryDeletingServiceIntervalReconfiguration (apache#8682) HDDS-13317. Table should support empty array/String (apache#8676) ...
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jun 26, 2025
* master: (170 commits) HDDS-12468. Check for space availability for all dns while container creation in pipeline (apache#8663) HDDS-12151. Fail write when volume is full considering min free space (apache#8642) HDDS-13251. Update Byteman usage README license (apache#8700) HDDS-13251. Support dynamic Byteman scripts via bmsubmit in ozonesecure-ha (apache#8654) HDDS-12070. Bump Ratis to 3.2.0 (apache#8689) HDDS-12984. Use InodeID to identify the SST files inside the tarball. (apache#8477) HDDS-13319. Simplify KeyPrefixFilter (apache#8692) HDDS-13295. Remove jackson1 exclusions for hadoop-common (apache#8687) HDDS-13314. Remove unused maven-pdf-plugin (apache#8686) HDDS-13324. Optimize memory footprint for Recon listKeys API (apache#8680) HDDS-13240. Add newly added metrics into grafana dashboard. (apache#8656) HDDS-13309. Add keyIterator/valueIterator methods to Table. (apache#8675) HDDS-13325. Introduce OZONE_SERVER_OPTS for common options for server processes (apache#8685) HDDS-13318. Simplify the getRangeKVs methods in Table (apache#8683) HDDS-13322. Remove module auto-detection from flaky-test-check (apache#8679) HDDS-13289. Remove usage of Jetty StringUtil (apache#8684) HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification (apache#8653) HDDS-13288. Container checksum file proto changes to account for deleted blocks. (apache#8649) HDDS-13306. Intermittent failure in testDirectoryDeletingServiceIntervalReconfiguration (apache#8682) HDDS-13317. Table should support empty array/String (apache#8676) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Change the deleted block list in the checksum info file to store
BlockMerkleTreeinstead of just block ID. This allows us to use checksums as described in HDDS-13245. This PR just changes the proto and the code that interfaces with it. The translation that is currently happening is a placeholder until the functionality using this change is implemented in HDDS-13245.This PR also changes the
isHealthyproto field tochecksumMatchesas suggested in the mail thread for branch merge.What is the link to the Apache JIRA
HDDS-13288
How was this patch tested?
No functionality is expected to change with this patch. Tests that were directly checking the proto were updated as needed. Otherwise existing tests should pass.