Dynamic config update via xDS Management Server

README.md

@@ -7,7 +7,8 @@
   - [API Deprecation History](#api-deprecation-history)
 - [Building and Testing](#building-and-testing)
   - [Docker-compose setup](#docker-compose-setup)
-  - [Full test environment](#full-test-environment)
+  - [Full test environment - Configure rate limits through files](#full-test-environment---configure-rate-limits-through-files)
+  - [Full test environment - Configure rate limits through an xDS Management Server](#full-test-environment---configure-rate-limits-through-an-xds-management-server)
   - [Self-contained end-to-end integration test](#self-contained-end-to-end-integration-test)
 - [Configuration](#configuration)
   - [The configuration format](#the-configuration-format)
@@ -25,6 +26,8 @@
       - [Example 6](#example-6)
       - [Example 7](#example-7)
   - [Loading Configuration](#loading-configuration)
+    - [File Based Configuration Loading](#file-based-configuration-loading)
+    - [xDS Management Server Based Configuration Loading](#xds-management-server-based-configuration-loading)
   - [Log Format](#log-format)
   - [GRPC Keepalive](#grpc-keepalive)
 - [Request Fields](#request-fields)
@@ -129,11 +132,12 @@ If you want to run with [two redis instances](#two-redis-instances), you will ne
 the docker-compose.yml file to run a second redis container, and change the environment variables
 as explained in the [two redis instances](#two-redis-instances) section.
 
-## Full test environment
+## Full test environment - Configure rate limits through files
 
 To run a fully configured environment to demo Envoy based rate limiting, run:
 
 ```bash
+export CONFIG_TYPE=FILE
 docker-compose -f docker-compose-example.yml up --build --remove-orphans
 ```
 
@@ -160,6 +164,36 @@ To see the metrics in the example
 curl http://localhost:9102/metrics | grep -i shadow
 ```
 
+## Full test environment - Configure rate limits through an xDS Management Server
+
+To run a fully configured environment to demo Envoy based rate limiting, run:
+
+```bash
+export CONFIG_TYPE=GRPC_XDS_SOTW
+docker-compose -f docker-compose-example.yml --profile xds-config up --build --remove-orphans
+```
+
+This will run in `xds-config` docker-compose profile which will run example xDS-Server, ratelimit, redis, prom-statsd-exporter and two Envoy containers such that you can demo rate limiting by hitting the below endpoints.
+
+```bash
+curl localhost:8888/test
+curl localhost:8888/header -H "foo: foo" # Header based
+curl localhost:8888/twoheader -H "foo: foo" -H "bar: bar" # Two headers
+curl localhost:8888/twoheader -H "foo: foo" -H "baz: baz"  # This will be rate limited
+curl localhost:8888/twoheader -H "foo: foo" -H "bar: banned" # Ban a particular header value
+curl localhost:8888/twoheader -H "foo: foo" -H "baz: shady" # This will never be ratelimited since "baz" with value "shady" is in shadow_mode
+curl localhost:8888/twoheader -H "foo: foo" -H "baz: not-so-shady" # This is subject to rate-limiting because the it's now in shadow_mode
+```
+
+Edit[`examples/xds-sotw-config-server/resource.go`](examples/xds-sotw-config-server/resource.go) to test different rate limit configs.
+
+To see the metrics in the example
+
+```bash
+# The metrics for the shadow_mode keys
+curl http://localhost:9102/metrics | grep -i shadow
+```
+
 ## Self-contained end-to-end integration test
 
 Integration tests are coded as bash-scripts in `integration-test/scripts`.
@@ -492,6 +526,18 @@ descriptors:
 
 ## Loading Configuration
 
+Rate limit service supports following configuration loading methods. You can define which methods to use by configuring environment variable `CONFIG_TYPE`.
+
+| Config Loading Method                                                             | Value for Environment Variable `CONFIG_TYPE` |
+| --------------------------------------------------------------------------------- | -------------------------------------------- |
+| [File Based Configuration Loading](#file-based-configuration-loading)             | `FILE` (Default)                             |
+| [xDS Server Based Configuration Loading](#xds-server-based-configuration-loading) | `GRPC_XDS_SOTW`                              |
+
+When the environment variable `FORCE_START_WITHOUT_INITIAL_CONFIG` set to `false`, the Rate limit service will wait for initial rate limit configuration before
+starting the server (gRPC, Rest server endpoints). When set to `true` the server will start even without initial configuration.
+
+### File Based Configuration Loading
+
 The Ratelimit service uses a library written by Lyft called [goruntime](https://github.com/lyft/goruntime) to do configuration loading. Goruntime monitors
 a designated path, and watches for symlink swaps to files in the directory tree to reload configuration files.
 
@@ -525,6 +571,31 @@ For more information on how runtime works you can read its [README](https://gith
 By default it is not possible to define multiple configuration files within `RUNTIME_SUBDIRECTORY` referencing the same domain.
 To enable this behavior set `MERGE_DOMAIN_CONFIG` to `true`.
 
+### xDS Management Server Based Configuration Loading
+
+xDS Management Server is a gRPC server which implements the [Aggregated Discovery Service (ADS)](https://github.com/envoyproxy/data-plane-api/blob/97b6dae39046f7da1331a4dc57830d20e842fc26/envoy/service/discovery/v3/ads.proto).
+The xDS Management server serves [Discovery Response](https://github.com/envoyproxy/data-plane-api/blob/97b6dae39046f7da1331a4dc57830d20e842fc26/envoy/service/discovery/v3/discovery.proto#L69) with [Ratelimit Configuration Resources](api/ratelimit/config/ratelimit/v3/rls_conf.proto)
+and with Type URL `"type.googleapis.com/ratelimit.config.ratelimit.v3.RateLimitConfig"`.
+The xDS client in the Rate limit service configure Rate limit service with the provided configuration.
+For more information on xDS protocol please refer to the [envoy proxy documentation](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol).
+
+You can refer to [the sample xDS configuration management server](examples/xds-sotw-config-server/README.md).
+
+The xDS server for listening for configuration can be set via [settings](https://github.com/envoyproxy/ratelimit/blob/master/src/settings/settings.go)
+package with the following environment variables:
+
+```
+CONFIG_GRPC_XDS_NODE_ID default:"default"
+CONFIG_GRPC_XDS_SERVER_URL default:"localhost:18000"
+CONFIG_GRPC_XDS_SERVER_CONNECT_RETRY_INTERVAL default:"3s"
+```
+
+As well Ratelimit supports TLS connections, these can be configured using the following environment variables:
+
+1. `CONFIG_GRPC_XDS_SERVER_USE_TLS`: set to `"true"` to enable a TLS connection with the xDS configuration management server.
+2. `CONFIG_GRPC_XDS_CLIENT_TLS_CERT`, `CONFIG_GRPC_XDS_CLIENT_TLS_KEY`, and `CONFIG_GRPC_XDS_SERVER_TLS_CACERT` to provides files to specify a TLS connection configuration to the xDS configuration management server.
+3. `CONFIG_GRPC_XDS_SERVER_TLS_SAN`: (Optional) Override the SAN value to validate from the server certificate.
+
 ## Log Format
 
 A centralized log collection system works better with logs in json format. JSON format avoids the need for custom parsing rules.

api/ratelimit/config/ratelimit/v3/rls_conf.proto

@@ -0,0 +1,104 @@
+syntax = "proto3";
+
+package ratelimit.config.ratelimit.v3;
+
+option java_package = "io.envoyproxy.ratelimit.config.ratelimit.v3";
+option java_outer_classname = "RlsConfigProto";
+option java_multiple_files = true;
+option go_package = "github.com/envoyproxy/go-control-plane/ratelimit/config/ratelimit/v3;ratelimitv3";
+
+// [#protodoc-title: Rate limit service configuration]
+// A management server which supports ADS (Aggregated Discovery Service - SotW or delta protocol) can apply
+// rate limit service configuration using the message type RateLimitConfig. The ADS client within the rate limit service
+// will stream Discovery Request with the resource type URL "type.googleapis.com/ratelimit.config.ratelimit.v3.RateLimitConfig".
+// The ADS management server should respond stream of Discovery Response with the same type URL and array of RateLimitConfigs
+// within resources of the Discovery Response.
+
+// Rate limit configuration for a single domain.
+message RateLimitConfig {
+  // Name of the rate limit configuration. This should be unique for each configuration.
+  string name = 1;
+
+  // Domain name for the rate limit configuration.
+  string domain = 2;
+
+  // List of rate limit configuration descriptors.
+  repeated RateLimitDescriptor descriptors = 3;
+}
+
+// Rate limit configuration descriptor.
+message RateLimitDescriptor {
+  // Key of the descriptor.
+  string key = 1;
+
+  // Optional value of the descriptor.
+  string value = 2;
+
+  // Rate limit policy of the descriptor.
+  RateLimitPolicy rate_limit = 3;
+
+  // List of sub rate limit descriptors.
+  repeated RateLimitDescriptor descriptors = 4;
+
+  // Mark the descriptor as shadow. When the values is true, rate limit service allow requests to the backend.
+  bool shadow_mode = 5;
+}
+
+// Rate-limit policy.
+message RateLimitPolicy {
+  // Unit of time for the rate limit.
+  RateLimitUnit unit = 1;
+
+  // Number of requests allowed in the policy within `unit` time.
+  uint32 requests_per_unit = 2;
+
+  // Mark the rate limit policy as unlimited. All requests are allowed to the backend.
+  bool unlimited = 3;
+
+  // Optional name for the rate limit policy. Name the policy, if it should be replaced (dropped evaluation) by
+  // another policy.
+  string name = 4;
+
+  // List of rate limit policies, this rate limit policy will replace (drop evaluation)
+  // For more information: https://github.com/envoyproxy/ratelimit/tree/0b2f4d5fb04bf55e1873e2c5e2bb28da67c0643f#replaces
+  // Example: https://github.com/envoyproxy/ratelimit/tree/0b2f4d5fb04bf55e1873e2c5e2bb28da67c0643f#example-7
+  repeated RateLimitReplace replaces = 5;
+}
+
+// Replace specifies the rate limit policy that should be replaced (dropped evaluation).
+// For more information: https://github.com/envoyproxy/ratelimit/tree/0b2f4d5fb04bf55e1873e2c5e2bb28da67c0643f#replaces
+message RateLimitReplace {
+  // Name of the rate limit policy, that is being replaced (dropped evaluation).
+  string name = 1;
+}
+
+// Identifies the unit of of time for rate limit.
+enum RateLimitUnit {
+  // The time unit is not known.
+  UNKNOWN = 0;
+
+  // The time unit representing a second.
+  SECOND = 1;
+
+  // The time unit representing a minute.
+  MINUTE = 2;
+
+  // The time unit representing an hour.
+  HOUR = 3;
+
+  // The time unit representing a day.
+  DAY = 4;
+}
+
+// [#protodoc-title: Rate Limit Config Discovery Service (RLS Conf DS)]
+
+// Return list of all rate limit configs that rate limit service should be configured with.
+service RateLimitConfigDiscoveryService {
+  rpc StreamRlsConfigs(stream envoy.service.discovery.v3.DiscoveryRequest)
+      returns (stream envoy.service.discovery.v3.DiscoveryResponse) {
+  }
+
+  rpc FetchRlsConfigs(envoy.service.discovery.v3.DiscoveryRequest)
+      returns (envoy.service.discovery.v3.DiscoveryResponse) {
+  }
+}

docker-compose-example.yml

@@ -49,6 +49,22 @@ services:
       - RUNTIME_ROOT=/data
       - RUNTIME_SUBDIRECTORY=ratelimit
       - RUNTIME_WATCH_ROOT=false
+      - CONFIG_TYPE=${CONFIG_TYPE:-FILE}
+      - CONFIG_GRPC_XDS_NODE_ID=test-node-id
+      - CONFIG_GRPC_XDS_SERVER_URL=ratelimit-xds-config-server:18000
+
+  ratelimit-xds-config-server:
+    image: ratelimit-xds-config-server:latest
+    build:
+      context: examples/xds-sotw-config-server
+      dockerfile: Dockerfile
+    command: ["-nodeID", "test-node-id", "-port", "18000", "-debug", "true"]
+    expose:
+      - 18000
+    networks:
+      - ratelimit-network
+    profiles:
+      - xds-config
 
   envoy-proxy:
     image: envoyproxy/envoy-dev:latest

examples/xds-sotw-config-server/Dockerfile

@@ -0,0 +1,11 @@
+FROM golang:1.18 AS build
+WORKDIR /xds-server
+
+COPY . .
+
+RUN CGO_ENABLED=0 GOOS=linux go build -o /go/bin/xds-server -v main/main.go
+
+FROM alpine:3.16 AS final
+RUN apk --no-cache add ca-certificates && apk --no-cache update
+COPY --from=build /go/bin/xds-server /bin/xds-server
+ENTRYPOINT [ "/bin/xds-server" ]

examples/xds-sotw-config-server/README.md

@@ -0,0 +1,17 @@
+# Example Rate-limit Configuration SotW xDS Server
+
+This is an example of a trivial xDS V3 control plane server similar to the example server in [go-control-plane](https://github.com/envoyproxy/go-control-plane/tree/main/internal/example). It serves sample Rate limit configuration. You can run the example using the project top-level docker-compose-example.yml, e.g.:
+
+```bash
+export CONFIG_TYPE=GRPC_XDS_SOTW
+docker-compose -f docker-compose-example.yml --profile xds-config up --build --remove-orphans
+```
+
+The docker-compose builds and runs the example server along with Rate limit server. The example server serves a configuration defined in [`resource.go`](resource.go). If everything works correctly, you can follow the [examples in project top-level README.md file](../../README.md#examples).
+
+## Files
+
+- [main/main.go](main/main.go) is the example program entrypoint. It instantiates the cache and xDS server and runs the xDS server process.
+- [resource.go](resource.go) generates a `Snapshot` structure which describes the configuration that the xDS server serves to Envoy.
+- [server.go](server.go) runs the xDS control plane server.
+- [logger.go](logger.go) is the logger.

examples/xds-sotw-config-server/go.mod

@@ -0,0 +1,20 @@
+module github.com/envoyproxy/ratelimit/examples/xds-sotw-config-server
+
+go 1.18
+
+require (
+	github.com/envoyproxy/go-control-plane v0.10.3-0.20230127155013-72157d335c8f
+	google.golang.org/grpc v1.52.0
+)
+
+require (
+	github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
+	github.com/cncf/xds/go v0.0.0-20220314180256-7f1daf1720fc // indirect
+	github.com/envoyproxy/protoc-gen-validate v0.9.1 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
+	golang.org/x/net v0.4.0 // indirect
+	golang.org/x/sys v0.3.0 // indirect
+	golang.org/x/text v0.5.0 // indirect
+	google.golang.org/genproto v0.0.0-20221118155620-16455021b5e6 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
+)