Skip to content

HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test.#7021

Merged
jojochuang merged 1 commit intoapache:masterfrom
jojochuang:HDDS-11270
Aug 6, 2024
Merged

HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test.#7021
jojochuang merged 1 commit intoapache:masterfrom
jojochuang:HDDS-11270

Conversation

@jojochuang
Copy link
Copy Markdown
Contributor

@jojochuang jojochuang commented Aug 2, 2024 •
edited
Loading

What changes were proposed in this pull request?

HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test.

Please describe your PR in detail:
This is a test-only PR built on top of HDDS-11259.

Add a test class TestDatanodeUpgradeToHBaseSupport which has two test cases

  • PutBlock request with incremental chunk list
  • Block finalization request

These two requests are rejected before upgrade finalizes and accepted after upgrade.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11270

How was this patch tested?

Test only code

errose28
errose28 reviewed Aug 2, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this @jojochuang. The new test class looks good overall. Once #7012 is merged and the diff is reduced I can do a full review.

@smengcl smengcl added the hbase HBase on Ozone support label Aug 3, 2024
@jojochuang
Copy link
Copy Markdown
Contributor Author

jojochuang commented Aug 5, 2024

will rebase this PR against master after the HDDS-7593 feature branch gets merged.

errose28
errose28 approved these changes Aug 5, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this @jojochuang. Just one minor comment otherwise LGTM. If you want to wait until HBase branch is merged into master that's ok.

@errose28
Copy link
Copy Markdown
Contributor

errose28 commented Aug 5, 2024

BTW I filed HDDS-11282 to extract the common setup code for these types of upgrade tests.

@jojochuang
HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) u…
3b66b1b 
…pgrade test.

Change-Id: I9e3296ddbf5922d81d9754d80aac7e3618720565

Add tests covering incremental chunk list PutBlock requests during upgrade.

Change-Id: If473d1e5b5cc82adfac72fbe88a9ed71f7eaa169

Address review comments.

add additional check to make sure containers don't get marked unhealthy after DataNode rejects an unsupported operation.

Change-Id: I73dd83987d617bf5e17ff8e46d1d183b20c46b10

Move all reference of INCREMENTAL_CHUNK_LIST to OzoneConstants.

Change-Id: I00e7f2362b6a83d03c0eb3f615c1415c06cd668e
@jojochuang jojochuang changed the base branch from HDDS-7593 to master August 6, 2024 17:44
@jojochuang jojochuang merged commit 91b2b91 into apache:master Aug 6, 2024
@jojochuang
Copy link
Copy Markdown
Contributor Author

jojochuang commented Aug 6, 2024

Merged. Thanks @errose28 !

errose28 added a commit to errose28/ozone that referenced this pull request Aug 7, 2024
@errose28
Merge branch 'master' into HDDS-10239-container-reconciliation
98369a8 
* master: (181 commits)
  HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
  HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
  HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
  HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
  HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
  HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
  HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
  HDDS-11272. Statistics some node status information (apache#7025)
  HDDS-11278. Move code out of Hadoop util package (apache#7028)
  HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
  HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
  HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
  HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
  HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
  HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
  HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
  HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
  HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
  HDDS-11226. Make ExponentialBackoffPolicy maxRetries configurable (apache#6985)
  HDDS-11260. [hsync] Add Ozone Manager protocol version (apache#7015)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerReportHandler.java
errose28 added a commit to errose28/ozone that referenced this pull request Aug 13, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-11254-re…
6f68df6 
…p-supervisor

Merge conflicts are resolved but the change does not yet build.

* HDDS-10239-container-reconciliation: (183 commits)
  HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945)
  HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
  HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
  HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
  HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
  HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
  HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
  HDDS-10926. Block deletion should update container merkle tree. (apache#6875)
  HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
  HDDS-11272. Statistics some node status information (apache#7025)
  HDDS-11278. Move code out of Hadoop util package (apache#7028)
  HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
  HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
  HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
  HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
  HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
  HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
  HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
  HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
  HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
errose28 added a commit to errose28/ozone that referenced this pull request Aug 16, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-11253-co…
8601118 
…rrupt-files

* HDDS-10239-container-reconciliation: (183 commits)
  HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945)
  HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
  HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
  HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
  HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
  HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
  HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
  HDDS-10926. Block deletion should update container merkle tree. (apache#6875)
  HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
  HDDS-11272. Statistics some node status information (apache#7025)
  HDDS-11278. Move code out of Hadoop util package (apache#7028)
  HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
  HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
  HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
  HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
  HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
  HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
  HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
  HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
  HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@errose28 errose28 errose28 approved these changes

@ChenSammi ChenSammi Awaiting requested review from ChenSammi

@ashishkumar50 ashishkumar50 Awaiting requested review from ashishkumar50

Assignees

No one assigned

Labels

hbase HBase on Ozone support

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jojochuang @errose28 @smengcl