Skip to content

Commit

Permalink
* Implement Redis configuration values splitting support
Browse files Browse the repository at this point in the history
* Actualize Redis main docs
* Actualize configuration docs
* Use both old & new Redis API for tests
  • Loading branch information
AJIOB committed Mar 3, 2024
1 parent 1d4578d commit 9424a81
Show file tree
Hide file tree
Showing 4 changed files with 109 additions and 26 deletions.
50 changes: 49 additions & 1 deletion .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
name: integration-tests
path: ./target/debug/sccache

redis:
redis-deprecated:
runs-on: ubuntu-latest
needs: build

Expand Down Expand Up @@ -75,6 +75,54 @@ jobs:
${SCCACHE_PATH} --show-stats | grep -e "Cache hits\s*[1-9]"
redis:
runs-on: ubuntu-latest
needs: build

services:
redis:
image: redis
ports:
- 6379:6379

env:
SCCACHE_REDIS_ENDPOINT: tcp://127.0.0.1
RUSTC_WRAPPER: /home/runner/.cargo/bin/sccache

steps:
- name: Clone repository
uses: actions/checkout@v4

- name: Install rust
uses: ./.github/actions/rust-toolchain
with:
toolchain: "stable"

- uses: actions/download-artifact@v4
with:
name: integration-tests
path: /home/runner/.cargo/bin/
- name: Chmod for binary
run: chmod +x ${SCCACHE_PATH}

- name: Test
run: cargo clean && cargo build

- name: Output
run: |
${SCCACHE_PATH} --show-stats
${SCCACHE_PATH} --show-stats | grep redis
- name: Test Twice for Cache Read
run: cargo clean && cargo build

- name: Output
run: |
${SCCACHE_PATH} --show-stats
${SCCACHE_PATH} --show-stats | grep -e "Cache hits\s*[1-9]"
s3_minio:
runs-on: ubuntu-latest
needs: build
Expand Down
17 changes: 11 additions & 6 deletions docs/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,12 @@ will default to `us-east-1`.

#### redis

* `SCCACHE_REDIS` full redis url, including auth and access token/passwd.
* `SCCACHE_REDIS` full redis url, including auth and access token/passwd (deprecated).
* `SCCACHE_REDIS_ENDPOINT` redis url without auth and access token/passwd - single node configuration.
* `SCCACHE_REDIS_CLUSTER_ENDPOINTS` redis cluster urls, separated by comma - shared cluster configuration.
* `SCCACHE_REDIS_USERNAME` redis username (optional).
* `SCCACHE_REDIS_PASSWORD` redis password (optional).
* `SCCACHE_REDIS_DB` redis database (optional, default is 0).
* `SCCACHE_REDIS_EXPIRATION` / `SCCACHE_REDIS_TTL` ttl for redis cache, don't set for default behavior.
* `SCCACHE_REDIS_KEY_PREFIX` key prefix (optional).

Expand Down Expand Up @@ -188,11 +193,11 @@ The full url appears then as `redis://user:passwd@1.2.3.4:6379/?db=1`.

#### webdav

* `SCCACHE_WEBDAV_ENDPOINT`
* `SCCACHE_WEBDAV_KEY_PREFIX`
* `SCCACHE_WEBDAV_USERNAME`
* `SCCACHE_WEBDAV_PASSWORD`
* `SCCACHE_WEBDAV_TOKEN`
* `SCCACHE_WEBDAV_ENDPOINT` a webdav service endpoint to store cache, such as `http://127.0.0.1:8080/my/webdav.php`.
* `SCCACHE_WEBDAV_KEY_PREFIX` specify the key prefix (subfolder) of cache (optional).
* `SCCACHE_WEBDAV_USERNAME` a username to authenticate with webdav service (optional).
* `SCCACHE_WEBDAV_PASSWORD` a password to authenticate with webdav service (optional).
* `SCCACHE_WEBDAV_TOKEN` a token to authenticate with webdav service (optional) - may be used instead of login & password.

#### OSS

Expand Down
19 changes: 17 additions & 2 deletions docs/Redis.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,32 @@
# Redis

Set `SCCACHE_REDIS` to a [Redis](https://redis.io/) url in format `redis://[[<username>]:<passwd>@]<hostname>[:port][/?db=<db>]` to store the cache in a Redis instance. Redis can be configured as a LRU (least recently used) cache with a fixed maximum cache size. Set `maxmemory` and `maxmemory-policy` according to the [Redis documentation](https://redis.io/topics/lru-cache). The `allkeys-lru` policy which discards the *least recently accessed or modified* key fits well for the sccache use case.
If you want to use [Redis](https://redis.io/) storage for the sccache cache, you need to set the `SCCACHE_REDIS_ENDPOINT` with the single-node redis URL.
If you want to use a Redis cluster, set `SCCACHE_REDIS_CLUSTER_ENDPOINTS` instead of `SCCACHE_REDIS_ENDPOINT` with the comma-separated list of redis node URLs.

Redis endpoint URL format can be found in the [OpenDAL source code](https://github.com/apache/opendal/blob/5f1d5d1d61ed28f63d4955538b33a4d582feebef/core/src/services/redis/backend.rs#L268-L307). Some valid examples:
* `redis://127.0.0.1:6379` or `tcp://127.0.0.1:6379` or `127.0.0.1:6379` - TCP-based Redis connection (non-secure)
* `rediss://@1.2.3.4:6379` - TLS-based Redis connection over TCP (secure)
* `unix:///tmp/redis.sock` or `redis+unix:///tmp/redis.sock` - Unix socket-based Redis connection

Redis can be configured as a LRU (least recently used) cache with a fixed maximum cache size. Set `maxmemory` and `maxmemory-policy` according to the [Redis documentation](https://redis.io/topics/lru-cache). The `allkeys-lru` policy which discards the *least recently accessed or modified* key fits well for the sccache use case.

Redis over TLS is supported. Use the [`rediss://`](https://www.iana.org/assignments/uri-schemes/prov/rediss) url scheme (note `rediss` vs `redis`). Append `#insecure` the the url to disable hostname verification and accept self-signed certificates (dangerous!). Note that this also disables [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication).

If you want to authenticate to Redis, set `SCCACHE_REDIS_USERNAME` and `SCCACHE_REDIS_PASSWORD` to the username and password accordingly.

`SCCACHE_REDIS_DB` is the database number to use. Default is 0.

Set `SCCACHE_REDIS_EXPIRATION` in seconds if you don't want your cache to live forever. This will override the default behavior of redis.

`SCCACHE_REDIS_TTL` is a deprecated synonym for `SCCACHE_REDIS_EXPIRATION`.

Set `SCCACHE_REDIS_KEY_PREFIX` if you want to prefix all cache keys. This can be
useful when sharing a Redis instance with another application or cache.

## Examples
`SCCACHE_REDIS` is deprecated, use `SCCACHE_REDIS_ENDPOINT` instead.
If you really want to use `SCCACHE_REDIS`, you should URL in format `redis://[[<username>]:<passwd>@]<hostname>[:port][/?db=<db>]`.

## Deprecated API Examples

Use the local Redis instance with no password:
```sh
Expand Down
49 changes: 32 additions & 17 deletions src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -657,23 +657,38 @@ fn config_from_env() -> Result<EnvConfig> {
}

// ======= redis =======
let redis = if let Ok(url) = env::var("SCCACHE_REDIS") {
let ttl = number_from_env_var("SCCACHE_REDIS_EXPIRATION")
.or_else(|| number_from_env_var("SCCACHE_REDIS_TTL"))
.transpose()?
.unwrap_or(DEFAULT_REDIS_CACHE_TTL);

let key_prefix = key_prefix_from_env_var("SCCACHE_REDIS_KEY_PREFIX");

// TODO: add possibility to load new values from env
Some(RedisCacheConfig {
url: Some(url),
ttl,
key_prefix,
..Default::default()
})
} else {
None
let redis = match (
env::var("SCCACHE_REDIS").ok(),
env::var("SCCACHE_REDIS_ENDPOINT").ok(),
env::var("SCCACHE_REDIS_CLUSTER_ENDPOINTS").ok(),
) {
(None, None, None) => None,
(url, endpoint, cluster_endpoints) => {
let db = number_from_env_var("SCCACHE_REDIS_DB")
.transpose()?
.unwrap_or(DEFAULT_REDIS_DB);

let username = env::var("SCCACHE_REDIS_USERNAME").ok();
let password = env::var("SCCACHE_REDIS_PASSWORD").ok();

let ttl = number_from_env_var("SCCACHE_REDIS_EXPIRATION")
.or_else(|| number_from_env_var("SCCACHE_REDIS_TTL"))
.transpose()?
.unwrap_or(DEFAULT_REDIS_CACHE_TTL);

let key_prefix = key_prefix_from_env_var("SCCACHE_REDIS_KEY_PREFIX");

Some(RedisCacheConfig {
url,
endpoint,
cluster_endpoints,
username,
password,
db,
ttl,
key_prefix,
})
}
};

if env::var_os("SCCACHE_REDIS_EXPIRATION").is_some()
Expand Down

0 comments on commit 9424a81

Please sign in to comment.