Skip to content

Commit

Permalink
Dynamic config update via xDS Management Server (envoyproxy#373)
Browse files Browse the repository at this point in the history
Signed-off-by: Renuka Fernando <renukapiyumal@gmail.com>
  • Loading branch information
renuka-fernando authored and barroca committed Sep 1, 2023
1 parent 6894407 commit 9457f59
Show file tree
Hide file tree
Showing 32 changed files with 2,005 additions and 249 deletions.
75 changes: 73 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
- [API Deprecation History](#api-deprecation-history)
- [Building and Testing](#building-and-testing)
- [Docker-compose setup](#docker-compose-setup)
- [Full test environment](#full-test-environment)
- [Full test environment - Configure rate limits through files](#full-test-environment---configure-rate-limits-through-files)
- [Full test environment - Configure rate limits through an xDS Management Server](#full-test-environment---configure-rate-limits-through-an-xds-management-server)
- [Self-contained end-to-end integration test](#self-contained-end-to-end-integration-test)
- [Configuration](#configuration)
- [The configuration format](#the-configuration-format)
Expand All @@ -27,6 +28,8 @@
- [Example 7](#example-7)
- [Example 8](#example-8)
- [Loading Configuration](#loading-configuration)
- [File Based Configuration Loading](#file-based-configuration-loading)
- [xDS Management Server Based Configuration Loading](#xds-management-server-based-configuration-loading)
- [Log Format](#log-format)
- [GRPC Keepalive](#grpc-keepalive)
- [Request Fields](#request-fields)
Expand Down Expand Up @@ -131,11 +134,12 @@ If you want to run with [two redis instances](#two-redis-instances), you will ne
the docker-compose.yml file to run a second redis container, and change the environment variables
as explained in the [two redis instances](#two-redis-instances) section.

## Full test environment
## Full test environment - Configure rate limits through files

To run a fully configured environment to demo Envoy based rate limiting, run:

```bash
export CONFIG_TYPE=FILE
docker-compose -f docker-compose-example.yml up --build --remove-orphans
```

Expand All @@ -162,6 +166,36 @@ To see the metrics in the example
curl http://localhost:9102/metrics | grep -i shadow
```

## Full test environment - Configure rate limits through an xDS Management Server

To run a fully configured environment to demo Envoy based rate limiting, run:

```bash
export CONFIG_TYPE=GRPC_XDS_SOTW
docker-compose -f docker-compose-example.yml --profile xds-config up --build --remove-orphans
```

This will run in `xds-config` docker-compose profile which will run example xDS-Server, ratelimit, redis, prom-statsd-exporter and two Envoy containers such that you can demo rate limiting by hitting the below endpoints.

```bash
curl localhost:8888/test
curl localhost:8888/header -H "foo: foo" # Header based
curl localhost:8888/twoheader -H "foo: foo" -H "bar: bar" # Two headers
curl localhost:8888/twoheader -H "foo: foo" -H "baz: baz" # This will be rate limited
curl localhost:8888/twoheader -H "foo: foo" -H "bar: banned" # Ban a particular header value
curl localhost:8888/twoheader -H "foo: foo" -H "baz: shady" # This will never be ratelimited since "baz" with value "shady" is in shadow_mode
curl localhost:8888/twoheader -H "foo: foo" -H "baz: not-so-shady" # This is subject to rate-limiting because the it's now in shadow_mode
```

Edit[`examples/xds-sotw-config-server/resource.go`](examples/xds-sotw-config-server/resource.go) to test different rate limit configs.

To see the metrics in the example

```bash
# The metrics for the shadow_mode keys
curl http://localhost:9102/metrics | grep -i shadow
```

## Self-contained end-to-end integration test

Integration tests are coded as bash-scripts in `integration-test/scripts`.
Expand Down Expand Up @@ -538,6 +572,18 @@ rather than the normal
## Loading Configuration
Rate limit service supports following configuration loading methods. You can define which methods to use by configuring environment variable `CONFIG_TYPE`.
| Config Loading Method | Value for Environment Variable `CONFIG_TYPE` |
| --------------------------------------------------------------------------------- | -------------------------------------------- |
| [File Based Configuration Loading](#file-based-configuration-loading) | `FILE` (Default) |
| [xDS Server Based Configuration Loading](#xds-server-based-configuration-loading) | `GRPC_XDS_SOTW` |
When the environment variable `FORCE_START_WITHOUT_INITIAL_CONFIG` set to `false`, the Rate limit service will wait for initial rate limit configuration before
starting the server (gRPC, Rest server endpoints). When set to `true` the server will start even without initial configuration.
### File Based Configuration Loading
The Ratelimit service uses a library written by Lyft called [goruntime](https://github.com/lyft/goruntime) to do configuration loading. Goruntime monitors
a designated path, and watches for symlink swaps to files in the directory tree to reload configuration files.
Expand Down Expand Up @@ -571,6 +617,31 @@ For more information on how runtime works you can read its [README](https://gith
By default it is not possible to define multiple configuration files within `RUNTIME_SUBDIRECTORY` referencing the same domain.
To enable this behavior set `MERGE_DOMAIN_CONFIG` to `true`.
### xDS Management Server Based Configuration Loading
xDS Management Server is a gRPC server which implements the [Aggregated Discovery Service (ADS)](https://github.com/envoyproxy/data-plane-api/blob/97b6dae39046f7da1331a4dc57830d20e842fc26/envoy/service/discovery/v3/ads.proto).
The xDS Management server serves [Discovery Response](https://github.com/envoyproxy/data-plane-api/blob/97b6dae39046f7da1331a4dc57830d20e842fc26/envoy/service/discovery/v3/discovery.proto#L69) with [Ratelimit Configuration Resources](api/ratelimit/config/ratelimit/v3/rls_conf.proto)
and with Type URL `"type.googleapis.com/ratelimit.config.ratelimit.v3.RateLimitConfig"`.
The xDS client in the Rate limit service configure Rate limit service with the provided configuration.
For more information on xDS protocol please refer to the [envoy proxy documentation](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol).
You can refer to [the sample xDS configuration management server](examples/xds-sotw-config-server/README.md).
The xDS server for listening for configuration can be set via [settings](https://github.com/envoyproxy/ratelimit/blob/master/src/settings/settings.go)
package with the following environment variables:
```
CONFIG_GRPC_XDS_NODE_ID default:"default"
CONFIG_GRPC_XDS_SERVER_URL default:"localhost:18000"
CONFIG_GRPC_XDS_SERVER_CONNECT_RETRY_INTERVAL default:"3s"
```
As well Ratelimit supports TLS connections, these can be configured using the following environment variables:
1. `CONFIG_GRPC_XDS_SERVER_USE_TLS`: set to `"true"` to enable a TLS connection with the xDS configuration management server.
2. `CONFIG_GRPC_XDS_CLIENT_TLS_CERT`, `CONFIG_GRPC_XDS_CLIENT_TLS_KEY`, and `CONFIG_GRPC_XDS_SERVER_TLS_CACERT` to provides files to specify a TLS connection configuration to the xDS configuration management server.
3. `CONFIG_GRPC_XDS_SERVER_TLS_SAN`: (Optional) Override the SAN value to validate from the server certificate.
## Log Format
A centralized log collection system works better with logs in json format. JSON format avoids the need for custom parsing rules.
Expand Down
104 changes: 104 additions & 0 deletions api/ratelimit/config/ratelimit/v3/rls_conf.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
syntax = "proto3";

package ratelimit.config.ratelimit.v3;

option java_package = "io.envoyproxy.ratelimit.config.ratelimit.v3";
option java_outer_classname = "RlsConfigProto";
option java_multiple_files = true;
option go_package = "github.com/envoyproxy/go-control-plane/ratelimit/config/ratelimit/v3;ratelimitv3";

// [#protodoc-title: Rate limit service configuration]
// A management server which supports ADS (Aggregated Discovery Service - SotW or delta protocol) can apply
// rate limit service configuration using the message type RateLimitConfig. The ADS client within the rate limit service
// will stream Discovery Request with the resource type URL "type.googleapis.com/ratelimit.config.ratelimit.v3.RateLimitConfig".
// The ADS management server should respond stream of Discovery Response with the same type URL and array of RateLimitConfigs
// within resources of the Discovery Response.

// Rate limit configuration for a single domain.
message RateLimitConfig {
// Name of the rate limit configuration. This should be unique for each configuration.
string name = 1;

// Domain name for the rate limit configuration.
string domain = 2;

// List of rate limit configuration descriptors.
repeated RateLimitDescriptor descriptors = 3;
}

// Rate limit configuration descriptor.
message RateLimitDescriptor {
// Key of the descriptor.
string key = 1;

// Optional value of the descriptor.
string value = 2;

// Rate limit policy of the descriptor.
RateLimitPolicy rate_limit = 3;

// List of sub rate limit descriptors.
repeated RateLimitDescriptor descriptors = 4;

// Mark the descriptor as shadow. When the values is true, rate limit service allow requests to the backend.
bool shadow_mode = 5;
}

// Rate-limit policy.
message RateLimitPolicy {
// Unit of time for the rate limit.
RateLimitUnit unit = 1;

// Number of requests allowed in the policy within `unit` time.
uint32 requests_per_unit = 2;

// Mark the rate limit policy as unlimited. All requests are allowed to the backend.
bool unlimited = 3;

// Optional name for the rate limit policy. Name the policy, if it should be replaced (dropped evaluation) by
// another policy.
string name = 4;

// List of rate limit policies, this rate limit policy will replace (drop evaluation)
// For more information: https://github.com/envoyproxy/ratelimit/tree/0b2f4d5fb04bf55e1873e2c5e2bb28da67c0643f#replaces
// Example: https://github.com/envoyproxy/ratelimit/tree/0b2f4d5fb04bf55e1873e2c5e2bb28da67c0643f#example-7
repeated RateLimitReplace replaces = 5;
}

// Replace specifies the rate limit policy that should be replaced (dropped evaluation).
// For more information: https://github.com/envoyproxy/ratelimit/tree/0b2f4d5fb04bf55e1873e2c5e2bb28da67c0643f#replaces
message RateLimitReplace {
// Name of the rate limit policy, that is being replaced (dropped evaluation).
string name = 1;
}

// Identifies the unit of of time for rate limit.
enum RateLimitUnit {
// The time unit is not known.
UNKNOWN = 0;

// The time unit representing a second.
SECOND = 1;

// The time unit representing a minute.
MINUTE = 2;

// The time unit representing an hour.
HOUR = 3;

// The time unit representing a day.
DAY = 4;
}

// [#protodoc-title: Rate Limit Config Discovery Service (RLS Conf DS)]

// Return list of all rate limit configs that rate limit service should be configured with.
service RateLimitConfigDiscoveryService {
rpc StreamRlsConfigs(stream envoy.service.discovery.v3.DiscoveryRequest)
returns (stream envoy.service.discovery.v3.DiscoveryResponse) {
}

rpc FetchRlsConfigs(envoy.service.discovery.v3.DiscoveryRequest)
returns (envoy.service.discovery.v3.DiscoveryResponse) {
}
}
16 changes: 16 additions & 0 deletions docker-compose-example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,22 @@ services:
- RUNTIME_ROOT=/data
- RUNTIME_SUBDIRECTORY=ratelimit
- RUNTIME_WATCH_ROOT=false
- CONFIG_TYPE=${CONFIG_TYPE:-FILE}
- CONFIG_GRPC_XDS_NODE_ID=test-node-id
- CONFIG_GRPC_XDS_SERVER_URL=ratelimit-xds-config-server:18000

ratelimit-xds-config-server:
image: ratelimit-xds-config-server:latest
build:
context: examples/xds-sotw-config-server
dockerfile: Dockerfile
command: ["-nodeID", "test-node-id", "-port", "18000", "-debug", "true"]
expose:
- 18000
networks:
- ratelimit-network
profiles:
- xds-config

envoy-proxy:
image: envoyproxy/envoy-dev:latest
Expand Down
11 changes: 11 additions & 0 deletions examples/xds-sotw-config-server/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
FROM golang:1.18 AS build
WORKDIR /xds-server

COPY . .

RUN CGO_ENABLED=0 GOOS=linux go build -o /go/bin/xds-server -v main/main.go

FROM alpine:3.16 AS final
RUN apk --no-cache add ca-certificates && apk --no-cache update
COPY --from=build /go/bin/xds-server /bin/xds-server
ENTRYPOINT [ "/bin/xds-server" ]
17 changes: 17 additions & 0 deletions examples/xds-sotw-config-server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Example Rate-limit Configuration SotW xDS Server

This is an example of a trivial xDS V3 control plane server similar to the example server in [go-control-plane](https://github.com/envoyproxy/go-control-plane/tree/main/internal/example). It serves sample Rate limit configuration. You can run the example using the project top-level docker-compose-example.yml, e.g.:

```bash
export CONFIG_TYPE=GRPC_XDS_SOTW
docker-compose -f docker-compose-example.yml --profile xds-config up --build --remove-orphans
```

The docker-compose builds and runs the example server along with Rate limit server. The example server serves a configuration defined in [`resource.go`](resource.go). If everything works correctly, you can follow the [examples in project top-level README.md file](../../README.md#examples).

## Files

- [main/main.go](main/main.go) is the example program entrypoint. It instantiates the cache and xDS server and runs the xDS server process.
- [resource.go](resource.go) generates a `Snapshot` structure which describes the configuration that the xDS server serves to Envoy.
- [server.go](server.go) runs the xDS control plane server.
- [logger.go](logger.go) is the logger.
20 changes: 20 additions & 0 deletions examples/xds-sotw-config-server/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
module github.com/envoyproxy/ratelimit/examples/xds-sotw-config-server

go 1.18

require (
github.com/envoyproxy/go-control-plane v0.10.3-0.20230127155013-72157d335c8f
google.golang.org/grpc v1.52.0
)

require (
github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
github.com/cncf/xds/go v0.0.0-20220314180256-7f1daf1720fc // indirect
github.com/envoyproxy/protoc-gen-validate v0.9.1 // indirect
github.com/golang/protobuf v1.5.2 // indirect
golang.org/x/net v0.4.0 // indirect
golang.org/x/sys v0.3.0 // indirect
golang.org/x/text v0.5.0 // indirect
google.golang.org/genproto v0.0.0-20221118155620-16455021b5e6 // indirect
google.golang.org/protobuf v1.28.1 // indirect
)
Loading

0 comments on commit 9457f59

Please sign in to comment.