[DO NOT MERGE] Backport some fixes, performance optimizations from master to ozone-1.4 #6929#6964
Merged
xichen01 merged 53 commits intoapache:ozone-1.4from Jul 24, 2024
Merged
Conversation
…ng duplicates to Datanode (apache#4988) (cherry picked from commit 88e18e3)
(cherry picked from commit d71904d)
(cherry picked from commit 42ded03)
…can host new container (apache#5721) (cherry picked from commit 0e07225)
(cherry picked from commit 1b26dd3)
(cherry picked from commit ec2634d)
…pache#5859) (cherry picked from commit ddc19e9)
…close (apache#5827) (cherry picked from commit ac68072)
…ontainerReport from DN (apache#5793) (cherry picked from commit 6021582)
apache#5904) (cherry picked from commit 2066868)
…s nodes are not found (apache#5911) (cherry picked from commit 3f3deef)
apache#5930) (cherry picked from commit c39c554)
… command (apache#4939) (cherry picked from commit 43c9565)
…pache#6045) (cherry picked from commit 37213b7)
(cherry picked from commit d50776a)
(cherry picked from commit 1e6e6d1)
…apache#5888) (cherry picked from commit 41ea9fb)
… read performance (apache#6127) (cherry picked from commit 528fc92)
(cherry picked from commit c35e99f)
…_STORE buckets. (apache#4245) (cherry picked from commit 0a5fc69)
(cherry picked from commit 0763985)
(cherry picked from commit 89b700d)
…configurable (apache#6408) (cherry picked from commit c4dc6a0)
… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)
…anager. (apache#6138) (cherry picked from commit 301664e)
…ds based on size. (apache#6318) (cherry picked from commit 93a2489)
(cherry picked from commit 3e97d8f)
…ock data have checksum" after upgrade (apache#6520) (cherry picked from commit 99a5703)
(cherry picked from commit cc023e7)
(cherry picked from commit fff2b6c)
…ache#6660) (cherry picked from commit 673c112)
(cherry picked from commit 9074b8d)
…pache#6659) (cherry picked from commit 3f1a7ed)
…pache#6614) (cherry picked from commit a291682)
(cherry picked from commit 459feac)
…d before XceiverRatisServer#removeGroup (apache#6696) (cherry picked from commit 87c3945)
… replication (apache#6683) (cherry picked from commit c1bcdea)
…ache#6753) (cherry picked from commit 787c9d3)
…oto (apache#6812) (cherry picked from commit 81bc179)
…r in DN mismatches (apache#6779) (cherry picked from commit 769d09e)
(cherry picked from commit dc005f4)
(cherry picked from commit f2bb59b)
(cherry picked from commit f2671e7)
(cherry picked from commit e01a57d)
Contributor
Author
|
@adoroszlai @vtutrinov @ivandika3 @devmadhuu @whbing @Cyrill @hemantk-12 @ArafatKhan2198 @SaketaChalamchala @ceekay47 @swamirishi @siddhantsangwan @rohit-kb @errose28 @chungen0126 please check that your commits are correctly cherry-picked |
Contributor
|
/pending DO NOT MERGE |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
adoroszlai
approved these changes
Jul 18, 2024
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @xichen01 for working on this. Backport of my commits look OK.
Contributor
|
Thanks @xichen01 for working on this. Backport of my commits look OK. |
ArafatKhan2198
approved these changes
Jul 19, 2024
Contributor
ArafatKhan2198
left a comment
ArafatKhan2198
left a comment
There was a problem hiding this comment.
Thanks @xichen01 for working on this. BackPort of my commits look good.
Contributor
|
Thanks @xichen01 for the patch. Backport of my changes look good. |
Contributor
|
Thanks for the backport. Mine are good too |
Contributor
Author
|
Thank you for the reviews @adoroszlai and @ivandika3 @devmadhuu @ArafatKhan2198 @hemantk-12 @rohit-kb Merged using: git checkout ozone-1.4
git pull
git merge --ff-only ozone-1.4-backport
git push |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Backport the following fixes from master to ozone-1.4 for 1.4.1. Includes Ratis upgrade, some bug fixes, as well as performance optimizations, and necessary dependencies.
HDDS-11186. First container log missing from bundle (#6952)
HDDS-11172. Bump vite to 4.5.3 (#6918)
HDDS-11104. Bump maven-dependency-plugin to 3.7.1 (#6903)
HDDS-10910. Bump Ratis to 3.1.0 (#6872)
HDDS-10983. EC Key read corruption when the replica index of container in DN mismatches (#6779)
HDDS-11013. Ensure version is always set in ContainerCommandRequestProto (#6812)
HDDS-10999. Remove dependency on ratis-server from Ozone Client (#6800)
HDDS-10924. TestSCMHAManagerImpl#testAddSCM fails on ratis master (#6753)
HDDS-10832. Client should switch to streaming based on OpenKeySession replication (#6683)
HDDS-10875. XceiverRatisServer#getRaftPeersInPipeline should be called before XceiverRatisServer#removeGroup (#6696)
HDDS-10371. NPE in OzoneAclUtils.isOwner (#6676)
HDDS-10781. Do not use OFSPath in O3FS BasicOzoneClientAdapterImpl (#6614)
HDDS-10696. Fix test failure caused by empty snapshot installation (#6659)
HDDS-10608. Recon can't get full key when using Recon API. (#6492)
HDDS-10834. Revert snapshot diff output change added in HDDS-9360 (#6660)
HDDS-10815. Bump Spring Framework to 5.3.34 (#6643)
HDDS-10803. HttpServer fails to start with wildcard principal (#6631)
HDDS-10806. Bump Bouncy Castle to 1.78.1 (#6632)
HDDS-10787. Updated rocksdb-checkpoint-differ to use managed RocksDB objects (#6625)
HDDS-10720. Datanode volume DU reserved percent should have a non-zero default value. (#6561)
HDDS-10792. Bump Netty to 4.1.109.Final (#6622)
HDDS-10784. Multipart upload to encrypted bucket fails with ClassCastException (#6618)
HDDS-10783. Close SstFileReaderIterator in RocksDBCheckpointDiffer (#6616)
HDDS-10614. Avoid decreasing cached space usage below zero (#6508)
HDDS-10652. EC Reconstruction fails with "IOException: None of the block data have checksum" after upgrade (#6520)
HDDS-10156. Optimize Snapshot Cache get and eviction (#6024)
HDDS-10452. Improve Recon Disk Usage to fetch and display Top N records based on size. (#6318)
HDDS-10206. Expose jmx metrics for snapshot cache size on the ozone manager. (#6138)
HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled (#5517)
HDDS-5865. Make read retry interval and attempts in BlockInputStream configurable (#6408)
HDDS-10505. Move space reservation logic to VolumeUsage (#6370)
HDDS-10504. Remove unused VolumeInfo#configuredCapacity (#6363)
HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. (#4245)
HDDS-10250. Use SnapshotId as key in SnapshotCache (#6139)
HDDS-10262. Encapsulate SnapshotCache inside OmSnapshotManager (#6135)
HDDS-10246. Remove KeyValueHandler.checkContainerIsHealthy to improve read performance (#6127)
HDDS-10029. Improved logs for SCMDeletedBlockTransactionStatusManager (#5888)
HDDS-10225. Speed up TestSCMHAManagerImpl. (#6109)
HDDS-10219. Bump frontend-maven-plugin to 1.15.0 (#6104)
HDDS-10178. Shaded Jar build failure in case-insensitive filesystem (#6045)
HDDS-8888. Consider Datanode queue capacity when sending DeleteBlocks command (#4939)
HDDS-10070. Intermittent failure in TestWritableRatisContainerProvider (#5930)
HDDS-8982. Log flooded by WritableRatisContainerProvider if pipeline's nodes are not found (#5911)
HDDS-10046. Replace PrecomputedVolumeSpace with SpaceUsageSource.Fixed (#5904)
HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN (#5793)
HDDS-9959. Propagate group remove to other datanodes during pipeline close (#5827)
HDDS-10007. Rename ManagedSstFileReader in rocksdb-checkpoint-differ (#5859)
HDDS-10027. NPE in VolumeInfoMetrics.getCommitted() (#5885)
HDDS-9922. Migrate TestOzoneFileInterfaces to JUnit5 (#5838)
HDDS-9807. Consider volume committed space when checking if datanode can host new container (#5721)
HDDS-9966. Bump maven-shade-plugin to 3.5.1 (#5823)
HDDS-6152. Migrate TestOzoneFileSystem to JUnit5 (#5795)
HDDS-8882. Manage status of DeleteBlocksCommand in SCM to avoid sending duplicates to Datanode (#4988)
How was this patch tested?
https://github.com/xichen01/ozone/actions/runs/9990984927