Skip to content

Retrieve MD5 from object metadata#414

Merged
joneubank merged 2 commits into
developfrom
fix/s3-download-md5
Jun 26, 2024
Merged

Retrieve MD5 from object metadata#414
joneubank merged 2 commits into
developfrom
fix/s3-download-md5

Conversation

@joneubank
Copy link
Copy Markdown
Contributor

For S3 downloads, when handling downloads in cases where the .meta file is not found, we need to retrieve the MD5 sum from the Object metadata. Our current implementation of this is reading the value of the eTag, but this is not guaranteed to contain an MD5 hash (and for multipart uploads is known to never contain the MD5 hash).

This change introduces a function to get the MD5 hash from the metadata. The function has two cases to check for where the MD5 hash might be.

  1. Priority is to use the built in S3 getContentMD5() method. If the hash is found here then this will be returned.
  2. Allow reading of a custom (user defined) metadata property. This allows a system administrator to assign the Base64 encoded MD5 hash in a custom location. This is given the default property name md5chksum which corresponds to the property name where the md5 is stored in Azure systems. This will match for cases where data is moved from Azure Blob storage into an S3 Object Storage system.

@joneubank joneubank requested a review from Azher2Ali June 26, 2024 19:58
Copy link
Copy Markdown
Contributor

@Azher2Ali Azher2Ali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@joneubank joneubank merged commit e5505bc into develop Jun 26, 2024
@joneubank joneubank deleted the fix/s3-download-md5 branch June 26, 2024 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants