Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

objstore/swift: use ncw/swift and support large files for OpenStack Swift #2732

Merged
merged 20 commits into from
Dec 9, 2020
Merged

objstore/swift: use ncw/swift and support large files for OpenStack Swift #2732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
bb31f19
feat swift: use ncs/swift and support large files for OpenStack Swift
FUSAKLA Jun 7, 2020
fceec97
Update pkg/objstore/swift/swift.go
FUSAKLA Jun 8, 2020
4412228
fix swift: CR minor fixes
FUSAKLA Jun 8, 2020
d95f13b
fix swift: fix broken syntax
FUSAKLA Jun 8, 2020
9c7c970
Update docs/storage.md
FUSAKLA Jul 2, 2020
f412066
swift: ad default autVersion
FUSAKLA Jul 2, 2020
dda255c
fix: fixed trailing comma of comment
FUSAKLA Jul 2, 2020
13c3a5d
fix: drop unused return string
FUSAKLA Jul 14, 2020
6cf34f5
fix: renme unused params
FUSAKLA Jul 14, 2020
d1ca388
fix: fixed tests
FUSAKLA Jul 14, 2020
06bbd06
rebase
FUSAKLA Aug 12, 2020
5922d18
cr: minor CR fixes
FUSAKLA Sep 7, 2020
a322aa5
fix: use io.copy for writing swift uploaded files
FUSAKLA Sep 7, 2020
09d59cb
fix: drop unneeded error returned by swift connectionFromConfig
FUSAKLA Sep 7, 2020
6876b7c
feat swift: add support for DLO
FUSAKLA Nov 4, 2020
8716555
fix swift: drop un-needed sleep
FUSAKLA Nov 4, 2020
33d6af2
fix swift: removed swift reference from errror wrapping
FUSAKLA Nov 12, 2020
5d8c6ba
fix swift: make create file errors more descriptive
FUSAKLA Nov 12, 2020
cb577fe
fix: fix go deps
FUSAKLA Nov 23, 2020
2695266
docs: fix changelog
FUSAKLA Dec 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re
### Changed

- [#3496](https://github.com/thanos-io/thanos/pull/3496) s3: Respect SignatureV2 flag for all credential providers.
- [#2732](https://github.com/thanos-io/thanos/pull/2732) Swift: Switched to a new library [ncw/swift](https://github.com/ncw/swift) providing large objects support.
By default, segments will be uploaded to the same container directory `segments/` if the file is bigger than `1GB`.
To change the defaults see [the docs](./docs/storage.md#openstack-swift).

## [v0.17.0](https://github.com/thanos-io/thanos/releases/tag/v0.17.0) - 2020.11.18

Expand Down
21 changes: 19 additions & 2 deletions docs/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Current object storage client implementations:
| [Google Cloud Storage](./storage.md#gcs) | Stable | Production Usage | yes | @bwplotka |
| [AWS/S3](./storage.md#s3) (and all S3-compatible storages e.g disk-based [Minio](https://min.io/)) | Stable | Production Usage | yes | @bwplotka |
| [Azure Storage Account](./storage.md#azure) | Stable | Production Usage | no | @vglafirov |
| [OpenStack Swift](./storage.md#openstack-swift) | Beta (working PoC) | Production Usage | yes | @sudhi-vm |
| [OpenStack Swift](./storage.md#openstack-swift) | Beta (working PoC) | Production Usage | yes | @FUSAKLA |
| [Tencent COS](./storage.md#tencent-cos) | Beta | Production Usage | no | @jojohappy |
| [AliYun OSS](./storage.md#aliyun-oss) | Beta | Production Usage | no | @shaulboozhiao,@wujinhu |
| [Local Filesystem](./storage.md#filesystem) | Stable | Testing and Demo only | yes | @bwplotka |
Expand Down Expand Up @@ -331,16 +331,27 @@ config:

#### OpenStack Swift

Thanos uses [gophercloud](http://gophercloud.io/) client to upload Prometheus data into [OpenStack Swift](https://docs.openstack.org/swift/latest/).
Thanos uses [ncw/swift](https://github.com/ncw/swift) client to upload Prometheus data into [OpenStack Swift](https://docs.openstack.org/swift/latest/).

Below is an example configuration file for thanos to use OpenStack swift container as an object store.
Note that if the `name` of a user, project or tenant is used one must also specify its domain by ID or name.
Various examples for OpenStack authentication can be found in the [official documentation](https://developer.openstack.org/api-ref/identity/v3/index.html?expanded=password-authentication-with-scoped-authorization-detail#password-authentication-with-unscoped-authorization).

By default, OpenStack Swift has a limit for maximum file size of 5 GiB. Thanos index files are often larger than that.
To resolve this issue, Thanos uses [Static Large Objects (SLO)](https://docs.openstack.org/swift/latest/overview_large_objects.html)
FUSAKLA marked this conversation as resolved.
Show resolved Hide resolved
which are uploaded as segments. These are by default put into the `segments` directory of the same container.
The default limit for using SLO is 1 GiB which is also the maximum size of the segment.
If you don't want to use the same container for the segments
(best practise is to use `<container_name>_segments` to avoid polluting listing of the container objects)
you can use the `large_file_segments_container_name` option to override the default and put the segments to other container.
_In rare cases you can switch to [Dynamic Large Objects (DLO)](https://docs.openstack.org/swift/latest/overview_large_objects.html)
by setting the `use_dynamic_large_objects` to true, but use it with caution since it even more relies on eventual consistency._

[embedmd]:# (flags/config_bucket_swift.txt yaml)
```yaml
type: SWIFT
config:
auth_version: 0
auth_url: ""
username: ""
user_domain_name: ""
Expand All @@ -355,6 +366,12 @@ config:
project_domain_name: ""
region_name: ""
container_name: ""
large_object_chunk_size: 1073741824
large_object_segments_container_name: ""
retries: 3
connect_timeout: 10s
timeout: 5m
use_dynamic_large_objects: false
```

#### Tencent COS
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ require (
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
github.com/golang/snappy v0.0.2
github.com/googleapis/gax-go v2.0.2+incompatible
github.com/gophercloud/gophercloud v0.14.0
github.com/grpc-ecosystem/go-grpc-middleware v1.1.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/hashicorp/golang-lru v0.5.4
Expand All @@ -37,6 +36,7 @@ require (
github.com/minio/minio-go/v7 v7.0.2
github.com/mozillazg/go-cos v0.13.0
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/ncw/swift v1.0.52
github.com/oklog/run v1.1.0
github.com/oklog/ulid v1.3.1
github.com/olekukonko/tablewriter v0.0.2
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -893,6 +893,8 @@ github.com/nats-io/nkeys v0.1.0/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxzi
github.com/nats-io/nkeys v0.1.3/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxziKVo7w=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/ncw/swift v1.0.50/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
github.com/ncw/swift v1.0.52 h1:ACF3JufDGgeKp/9mrDgQlEgS8kRYC4XKcuzj/8EJjQU=
github.com/ncw/swift v1.0.52/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
Expand Down
Loading