Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

objectstore/cos: support multi-part upload (#5137) #5139

Merged

Conversation

hanjm
Copy link
Member

@hanjm hanjm commented Feb 10, 2022

Signed-off-by: Jimmie Han hanjinming@outlook.com

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Support multi-part upload. fix issue #5137 can not upload file more than 5GB.

Verification

e2e test.

export COS_APP_ID=*** COS_SECRET_ID=*** COS_SECRET_KEY=*** COS_REGION=ap-guangzhou  
THANOS_TEST_OBJSTORE_SKIP=AZURE,ALIYUNOSS,BOS,GCS,S3,SWIFT go test -cover -coverpkg "github.com/thanos-io/thanos/pkg/objstore/cos" -coverprofile=/tmp/test_cos.out -test.run ^TestObjStore_AcceptanceTest_e2e$ ./pkg/objstore/objtesting/...
go tool cover -html=/tmp/test_cos.out

result:

ok      github.com/thanos-io/thanos/pkg/objstore/objtesting     50.698s coverage: 68.2% of statements in github.com/thanos-io/thanos/pkg/objstore/cos

image

Signed-off-by: Jimmie Han <hanjinming@outlook.com>
@hanjm hanjm force-pushed the feature/cos-store-support-multi-part-upload branch from ed58455 to 3ee1714 Compare February 10, 2022 03:35
Signed-off-by: Jimmie Han <hanjinming@outlook.com>
@hanjm hanjm force-pushed the feature/cos-store-support-multi-part-upload branch from 3ee1714 to 0546e08 Compare February 10, 2022 03:35
Signed-off-by: Jimmie Han <hanjinming@outlook.com>
@hanjm hanjm requested review from yeya24, kakkoyun and jojohappy and removed request for kakkoyun February 11, 2022 00:17
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hanjm
Copy link
Member Author

hanjm commented Feb 11, 2022

Thank you for reviewing 🚀

@yeya24 yeya24 merged commit cb1f1d3 into thanos-io:main Feb 11, 2022
@wu3396
Copy link

wu3396 commented Feb 15, 2022

@hanjm The merge commit cb1f1d3 unexpectedly raised a Bucket replicate error
Full logs :
level=info ts=2022-02-15T08:22:59.732635287Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=info ts=2022-02-15T08:22:59.767260437Z caller=scheme.go:206 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="adding block to be replicated" block_uuid=01FVXQ4WZVQQGT675KXSNQCQR2 level=info ts=2022-02-15T08:22:59.815890088Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=info ts=2022-02-15T08:22:59.858718234Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=info ts=2022-02-15T08:22:59.897825296Z caller=scheme.go:206 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="adding block to be replicated" block_uuid=01FVXQ713XKT2YEFQBRTPG0AYX level=info ts=2022-02-15T08:22:59.931214473Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=debug ts=2022-02-15T08:22:59.931235437Z caller=scheme.go:234 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="ensuring block is replicated" block_uuid=01FVXQAE20VJA91GQ3WDYVNQNH level=debug ts=2022-02-15T08:23:00.113699966Z caller=scheme.go:305 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="ensuring object is replicated" object=01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 level=debug ts=2022-02-15T08:23:00.140302813Z caller=scheme.go:318 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="object not present in target bucket, replicating" object=01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 level=warn ts=2022-02-15T08:23:00.287307799Z caller=intrumentation.go:67 component=replicate msg="changing probe status" status=not-ready reason="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=info ts=2022-02-15T08:23:00.287350972Z caller=http.go:84 component=replicate service=http/server component=replicate msg="internal server is shutting down" err="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=info ts=2022-02-15T08:23:00.287465777Z caller=http.go:103 component=replicate service=http/server component=replicate msg="internal server is shutdown gracefully" err="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=info ts=2022-02-15T08:23:00.287511374Z caller=intrumentation.go:81 component=replicate msg="changing probe status" status=not-healthy reason="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=error ts=2022-02-15T08:23:00.287770659Z caller=main.go:158 err="unsupported type of io.Reader: *objstore.timingReadCloser\ngithub.com/thanos-io/thanos/pkg/objstore.TryToGetSize\n\t/app/pkg/objstore/objstore.go:151\ngithub.com/thanos-io/thanos/pkg/objstore.newTracingReadCloser\n\t/app/pkg/objstore/tracing.go:128\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Get\n\t/app/pkg/objstore/tracing.go:43\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureObjectReplicated\n\t/app/pkg/replicate/scheme.go:320\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:277\ngithub.com/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.com/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.com/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\ngetting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001\ngithub.com/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Upload\n\t/app/pkg/objstore/cos/cos.go:223\ngithub.com/thanos-io/thanos/pkg/objstore.(*metricBucket).Upload\n\t/app/pkg/objstore/objstore.go:454\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Upload.func1\n\t/app/pkg/objstore/tracing.go:79\ngithub.com/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Upload\n\t/app/pkg/objstore/tracing.go:77\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureObjectReplicated\n\t/app/pkg/replicate/scheme.go:327\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:277\ngithub.com/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.com/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.com/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nupload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureObjectReplicated\n\t/app/pkg/replicate/scheme.go:328\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:277\ngithub.com/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.com/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.com/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nreplicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:279\ngithub.com/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.com/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.com/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.com/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated\ngithub.com/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:219\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nreplication execute\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:207\ngithub.com/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\ntools bucket replicate command failed\nmain.main\n\t/app/cmd/thanos/main.go:158\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

@hanjm hanjm mentioned this pull request Feb 17, 2022
2 tasks
@hanjm
Copy link
Member Author

hanjm commented Feb 17, 2022

@wu3396 I fix it by #5166. could you help verify?

@wu3396
Copy link

wu3396 commented Feb 18, 2022

@hanjm Yes! Thanks for your help! That one works!

@hanjm hanjm mentioned this pull request Feb 25, 2022
Nicholaswang pushed a commit to Nicholaswang/thanos that referenced this pull request Mar 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants