Skip to content

HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification#8653

Merged
ivandika3 merged 12 commits intoapache:masterfrom
hevinhsu:HDDS-13270
Jun 24, 2025
Merged

HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification#8653
ivandika3 merged 12 commits intoapache:masterfrom
hevinhsu:HDDS-13270

Conversation

@hevinhsu
Copy link
Copy Markdown
Contributor

@hevinhsu hevinhsu commented Jun 18, 2025

What changes were proposed in this pull request?

Reduce getBucket API invocations in S3 bucket owner verification.

Please describe your PR in detail:

  • Replaced bucketName with OzoneBucket in affected APIs to perform bucket ownership verification directly without extra getBucket calls.
  • Introduced a new utility method hasBucketOwnershipVerificationConditions to determine whether verification is required based on request headers.
  • Used hasBucketOwnershipVerificationConditions to avoid invoking getBucket() for APIs that do not require bucket ownership validation, reducing overhead and unnecessary OM RPC calls.

What is the link to the Apache JIRA

HDDS-13270

How was this patch tested?

https://github.com/hevinhsu/ozone/actions/runs/15727321427

@adoroszlai adoroszlai added the s3 S3 Gateway label Jun 18, 2025
@ivandika3 ivandika3 self-requested a review June 19, 2025 01:19
@hevinhsu hevinhsu changed the title Hdds 13270. Reduce getBucket API invocations in S3 bucket owner verification HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification Jun 19, 2025
@peterxcli peterxcli self-requested a review June 19, 2025 02:05
@ivandika3
Copy link
Copy Markdown
Contributor

ivandika3 commented Jun 21, 2025

@hevinhsu Thanks for the patch. Could you help to merge from master one more time since the integration failures in master has stabilized? Thanks.

@hevinhsu
Copy link
Copy Markdown
Contributor Author

hevinhsu commented Jun 21, 2025

@ivandika3 Thanks for pointing this out! I've merged the main branch and fixed the test failures.
The tests also passed on my fork

Please let me know if there's anything else needed.

ivandika3
ivandika3 approved these changes Jun 23, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@ivandika3 ivandika3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hevinhsu Thanks for addressing this promptly and merging the conflicts. LGTM +1.

@ivandika3 ivandika3 merged commit ff7a20d into apache:master Jun 24, 2025
41 checks passed
@ivandika3
Copy link
Copy Markdown
Contributor

ivandika3 commented Jun 24, 2025

@hevinhsu Thanks for the improvement.

@hevinhsu
Copy link
Copy Markdown
Contributor Author

hevinhsu commented Jun 24, 2025

@ivandika3 Thanks for the review.

@hevinhsu hevinhsu deleted the HDDS-13270 branch June 24, 2025 01:18
errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2025
@errose28
Merge branch 'master' into diff-report-javadoc
8b9c416 
* master: (90 commits)
  HDDS-12468. Check for space availability for all dns while container creation in pipeline (apache#8663)
  HDDS-12151. Fail write when volume is full considering min free space (apache#8642)
  HDDS-13251. Update Byteman usage README license (apache#8700)
  HDDS-13251. Support dynamic Byteman scripts via bmsubmit in ozonesecure-ha (apache#8654)
  HDDS-12070. Bump Ratis to 3.2.0 (apache#8689)
  HDDS-12984. Use InodeID to identify the SST files inside the tarball. (apache#8477)
  HDDS-13319. Simplify KeyPrefixFilter (apache#8692)
  HDDS-13295. Remove jackson1 exclusions for hadoop-common (apache#8687)
  HDDS-13314. Remove unused maven-pdf-plugin (apache#8686)
  HDDS-13324. Optimize memory footprint for Recon listKeys API (apache#8680)
  HDDS-13240. Add newly added metrics into grafana dashboard. (apache#8656)
  HDDS-13309. Add keyIterator/valueIterator methods to Table. (apache#8675)
  HDDS-13325. Introduce OZONE_SERVER_OPTS for common options for server processes (apache#8685)
  HDDS-13318. Simplify the getRangeKVs methods in Table (apache#8683)
  HDDS-13322. Remove module auto-detection from flaky-test-check (apache#8679)
  HDDS-13289. Remove usage of Jetty StringUtil (apache#8684)
  HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification (apache#8653)
  HDDS-13288. Container checksum file proto changes to account for deleted blocks. (apache#8649)
  HDDS-13306. Intermittent failure in testDirectoryDeletingServiceIntervalReconfiguration (apache#8682)
  HDDS-13317. Table should support empty array/String (apache#8676)
  ...
errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2025
@errose28
Merge branch 'master' into HDDS-13083-handle-empty-mt
b248828 
* master: (170 commits)
  HDDS-12468. Check for space availability for all dns while container creation in pipeline (apache#8663)
  HDDS-12151. Fail write when volume is full considering min free space (apache#8642)
  HDDS-13251. Update Byteman usage README license (apache#8700)
  HDDS-13251. Support dynamic Byteman scripts via bmsubmit in ozonesecure-ha (apache#8654)
  HDDS-12070. Bump Ratis to 3.2.0 (apache#8689)
  HDDS-12984. Use InodeID to identify the SST files inside the tarball. (apache#8477)
  HDDS-13319. Simplify KeyPrefixFilter (apache#8692)
  HDDS-13295. Remove jackson1 exclusions for hadoop-common (apache#8687)
  HDDS-13314. Remove unused maven-pdf-plugin (apache#8686)
  HDDS-13324. Optimize memory footprint for Recon listKeys API (apache#8680)
  HDDS-13240. Add newly added metrics into grafana dashboard. (apache#8656)
  HDDS-13309. Add keyIterator/valueIterator methods to Table. (apache#8675)
  HDDS-13325. Introduce OZONE_SERVER_OPTS for common options for server processes (apache#8685)
  HDDS-13318. Simplify the getRangeKVs methods in Table (apache#8683)
  HDDS-13322. Remove module auto-detection from flaky-test-check (apache#8679)
  HDDS-13289. Remove usage of Jetty StringUtil (apache#8684)
  HDDS-13270. Reduce getBucket API invocations in S3 bucket owner verification (apache#8653)
  HDDS-13288. Container checksum file proto changes to account for deleted blocks. (apache#8649)
  HDDS-13306. Intermittent failure in testDirectoryDeletingServiceIntervalReconfiguration (apache#8682)
  HDDS-13317. Table should support empty array/String (apache#8676)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ivandika3 ivandika3 ivandika3 approved these changes

@peterxcli peterxcli Awaiting requested review from peterxcli

Assignees

No one assigned

Labels

performance s3 S3 Gateway

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hevinhsu @ivandika3 @adoroszlai