Skip to content

Commit

Permalink
Allow turning off checksums in S3 Puts
Browse files Browse the repository at this point in the history
thanos-io/thanos#6746 bumped objstore,
which bumped minio, which made Thanos incompatible with otherwise
compliant S3 backends that do not support the x-amz-checksum header.

This adds in a `SendChecksum` (open to naming changes) config entry
to the S3 bucket config that allows turning off this header so
that Thanos supports these backends.

Signed-off-by: sinkingpoint <colin@quirl.co.nz>
  • Loading branch information
sinkingpoint committed Nov 8, 2023
1 parent ff7faac commit 66bb9ac
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions providers/s3/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ var DefaultConfig = Config{
},
PartSize: 1024 * 1024 * 64, // 64MB.
BucketLookupType: AutoLookup,
SendChecksum: true,
}

// HTTPConfig exists here only because Cortex depends on it, and we depend on Cortex.
Expand All @@ -136,6 +137,7 @@ type Config struct {
TraceConfig TraceConfig `yaml:"trace"`
ListObjectsVersion string `yaml:"list_objects_version"`
BucketLookupType BucketLookupType `yaml:"bucket_lookup_type"`
SendChecksum bool `yaml:"send_checksum"`
// PartSize used for multipart upload. Only used if uploaded object size is known and larger than configured PartSize.
// NOTE we need to make sure this number does not produce more parts than 10 000.
PartSize uint64 `yaml:"part_size"`
Expand Down Expand Up @@ -166,6 +168,7 @@ type Bucket struct {
storageClass string
partSize uint64
listObjectsV1 bool
sendChecksum bool
}

// parseConfig unmarshals a buffer into a Config with default values.
Expand Down Expand Up @@ -334,6 +337,7 @@ func NewBucketWithConfig(logger log.Logger, config Config, component string) (*B
storageClass: storageClass,
partSize: config.PartSize,
listObjectsV1: config.ListObjectsVersion == "v1",
sendChecksum: config.SendChecksum,
}
return bkt, nil
}
Expand Down Expand Up @@ -510,6 +514,7 @@ func (b *Bucket) Upload(ctx context.Context, name string, r io.Reader) error {
ServerSideEncryption: sse,
UserMetadata: userMetadata,
StorageClass: b.storageClass,
SendContentMd5: b.sendChecksum,
// 4 is what minio-go have as the default. To be certain we do micro benchmark before any changes we
// ensure we pin this number to four.
// TODO(bwplotka): Consider adjusting this number to GOMAXPROCS or to expose this in config if it becomes bottleneck.
Expand Down

0 comments on commit 66bb9ac

Please sign in to comment.