Increase max metadata value size to 2KB#48
Open
diwu1989 wants to merge 1 commit intoNVIDIA:masterfrom
Open
Conversation
AWS S3 allows max metadata size of 2KB. It's computed as the sum of the key and value. Swift by default constrains the metadata value size to 256bytes, which is too low and not really compatible with what S3 provides.
timuralp
reviewed
Jan 14, 2020
| swift_hash_path_suffix = 0rBOk3EM17yId7Ke | ||
|
|
||
| [swift-constraints] | ||
| max_meta_value_length = 2048 |
Contributor
There was a problem hiding this comment.
If the goal is to make Swift constraints match S3, there are a few more relevant settings: https://github.com/openstack/swift/blob/master/swift/common/constraints.py
Specifically, I think you want to set max_meta_overall_size to 2048 and max_meta_header_size to 2048. It looks like S3 doesn't have a hard constraint on the length of the metadata key either (at least I can't find it here: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata), so maybe max_meta_name_length should also be 2048. Can you validate the various lengths against AWS S3? I'm also unclear if there is a maximum number of x-amz-meta- headers allowed (which Swift currently sets to 90).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AWS S3 allows max metadata size of 2KB. It's computed as the sum of the key and value.
Swift by default constrains the metadata value size to 256bytes, which is too low and not really compatible with what S3 provides.