Eliminating aws and collaboratory profile and configuring s3 profile in application.yml and documentation.#479
Merged
Merged
Conversation
…and updating the application.yml along with documentation
Azher2Ali
requested review from
joneubank and
leoraba
and removed request for
leoraba
joneubank
requested changes
Aug 22, 2024
Contributor
joneubank
left a comment
joneubank
left a comment
There was a problem hiding this comment.
Code changes look good, need some markdown updates.
joneubank
approved these changes
Aug 23, 2024
Comment on lines
+54
to
+64
| | Property | Description | | ||
| | ------------- | ------------- | | ||
| |`s3.secured` | Determines whether the connection to the S3 service should use HTTPS (true) or HTTP (false). Set to true to secure the connection. | | ||
| |`s3.endpoint` |The URL of the S3-compatible service. This is the endpoint where the service is hosted. This property should be provided based on the service you're using. | ||
| |`s3.accessKey` |The access key for authenticating with the S3 service. It's required for secure access and should be kept confidential.| | ||
| |`s3.secretKey`|The secret key paired with the access key for secure authentication to the S3 service. It must also be kept confidential.| | ||
| |`s3.masterEncryptionKeyId`|The ID of the encryption key used for server-side encryption of files stored in S3. If provided, this key ensures that all data at rest is encrypted using the specified key.| | ||
| |`s3.customMd5Property`| A custom metadata property that stores an MD5 checksum of the uploaded files. This is useful for validating the integrity of the files.| | ||
| |`s3.connectionTimeout`| The maximum amount of time (in milliseconds) the client will wait to establish a connection to the S3 service before timing out. This helps manage delays in the network or service availability.| | ||
| |`s3.retryLimit`| The number of retries the client will attempt if an operation fails (e.g., an upload). This helps ensure robustness in case of transient issues. | ||
| |`s3.sigV4Enabled`| Enables AWS Signature Version 4 for request signing. This is required for certain regions or when using advanced features like server-side encryption with AWS KMS.| |
leoraba
added a commit
that referenced
this pull request
Aug 5, 2025
* Keycloak permission manager (#387) * enable Keycloak apiKeys * setup security config and update mvn dependencies * junit missing dependency * unit test mock Jwt Decoder * fix unit test - use a JWT decoder for testing - Remove unused clases - Remove JWT expired unit tests as validation is now implemented by Spring Security * code format * test profile * docker-compose update images * add keycloak to docker compose * fix merge conflict * fix typo curl command * update keycloak system client and apikeys * Renamed the constant ICGCFS to SCOREFS * change URI scheme from icgc:// to score:// * Remove outdated wiki reference comment in DownloadManifest.java * Updating the getName response * Updated bucket names from oicr.icgc.test to score.data and score.state * Remove unused scripts and static resources from score * Update token header key from ICGC to SCORe * Update READMEs to replace ICGC with SCORe * Deleting the settings folder belonging to specific IDEs * Removing out dated tests which are no longer in service * Renaming the icgc references to SCORe * Renaming the icgc reference to SCORe * Rephrasing the error handling messages * Updating the icgc buckets to overture.example.score * Deleting unused templated which are related to kf * Rename Default Buckets in Docker-Compose and Update References in SCORe (#471) * Consolidate Manifest Handling by Removing KF-Specific Implementation (#470) * Consolidate Manifest Handling by Removing KF-Specific Implementation * Removing unused classes and dependecies from the code * Eliminating Error messaging referencing ICGC (#454) * Eliminating Error messaging referencing ICGC * Updating the error message * Removing ICGC support contact info currently which are inside the SCO… (#453) * Removing ICGC support contact info currently which are inside the SCORe CLI * Removing the maintainter statement as per the feedback * Remove Hardcoded Profiles and Default URLs from SCORe Client (#469) * Remove Hardcoded Profiles and Default URLs from SCORe Client * Updating application.yml according to the description * Adding defaulturls in the test/application.yml * Changes related to urls in test/application.yml * Local docker-compose re-uses object bucket name for state bucket * Eliminating kf profile from the codebase (#476) * Eliminating kf profile from the codebase * Assigning default value to partSize * Remove @Profile Annotations from DownloadController and UploadController (#477) * Remove Hardcoded URL in Benchmark Profile (#478) * Eliminating aws and collaboratory profile and configuring s3 profile in application.yml and documentation. (#479) * Eliminating aws and collaboratory profile and configuring s3 profile and updating the application.yml along with documentation * Changes related to s3 description in README.md * Changes related to README file formatting * Rename COLLABORATORY Strorage Profile to S3 in SCORe * Eliminating amazong profile with s3 * Merge from `main` to `develop` (#483) * Versioning 5.10.1 * Rc/5.11.0 (#418) * Keycloak permission manager (#387) * enable Keycloak apiKeys * setup security config and update mvn dependencies * junit missing dependency * unit test mock Jwt Decoder * fix unit test - use a JWT decoder for testing - Remove unused clases - Remove JWT expired unit tests as validation is now implemented by Spring Security * code format * test profile * docker-compose update images * add keycloak to docker compose * fix merge conflict * fix typo curl command * update keycloak system client and apikeys * version 5.11.0 --------- Co-authored-by: Jon Eubank <joneubank@gmail.com> * Feature - Add `noSecurityDev` profile (#485) * Add noSecurityDev profile to require no auth tokens in dev * Include description for azure profile in server readme * Score Dev Documentation (#482) * basic folder structure * code of conduct * license * contributing * docs folder setup + ported existing docs * docs folder setup + ported existing docs * Fixed broken links * minor update * minor updates * updated symlink * updated symlinks * link update * updated application.yaml * updated application.yaml * simplified file naming * fixed links * cleaned up content * cleaned up content * missing variable download expiration time * updated application.yaml * updated application.yaml * updated setup documentation * temporarly sperated usage docs * updated overview page * updated docs + application.yaml * added empty line * minor fix * fixed broken link * added score client docker setup * updated user guides * updated readme * updating contributing & code of conduct * updated with object storage and keycloak docs * vault doc * links * minor update * score setup title * updated bat command * Add Key Features to main README * Update repository structure to mirror main README * Formatting and minor edits * netlify link * updated cross referenced urls to docs.overture.bio * removed code of conduct (.github covers this) removed repository structure in readme (it is in the overview) * fixed typos in images, reduced image size to ~1mb will try lower if needed * revert changes in application.yml --------- Co-authored-by: Leonardo Rivera <leorivera_88@hotmail.com> Co-authored-by: Jon Eubank <joneubank@gmail.com> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * docker version + naming (#489) Co-authored-by: Mitchell Shiell <mshiell@wl7053-spatel.ad.oicr.on.ca> * add JetBrains logo in ReadMe (#490) * Fix typo in README (#491) --------- Co-authored-by: Azher2Ali <121898125+Azher2Ali@users.noreply.github.com> Co-authored-by: Jon Eubank <joneubank@gmail.com> Co-authored-by: Mitchell Shiell <59712867+MitchellShiell@users.noreply.github.com> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Mitchell Shiell <mshiell@wl7053-spatel.ad.oicr.on.ca> Co-authored-by: bounlu <bounlu@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
In our ongoing efforts to streamline the SCORe server codebase and improve configuration management, we are consolidating multiple S3-compatible backend profiles into a single, unified profile. This change simplifies the configuration by removing outdated or redundant profiles and standardizing on a single s3 profile that works with any S3 API, including Amazon AWS, MinIO, and Ceph.
Changes Include:
Java Classes: Update @Profile annotations to replace collaboratory, amazon, and aws profiles with the new s3 profile. The affected classes are:
BackendHealth
ServerConfig
S3Config
S3DownloadService
S3ListingService
S3UploadService
application.yaml Configuration:
Removed : Sections for amazon and collaboratory.
Added: A new section for the s3 profile, including all configurable properties related to S3 connections, with commented-out default properties where appropriate.
Documentation Updates:
Updated score-server/README.md: Add a new section to document regarding the s3 profile, which includes required profile name and listing all configuration properties with their descriptions.