From 789326ba892891dcc92d05cae6b9487e63817aca Mon Sep 17 00:00:00 2001 From: Dave Protasowski Date: Wed, 27 Sep 2023 18:14:37 -0400 Subject: [PATCH] GEP-1911 - Backend Protocol Selection (#1979) * Initial draft of GEP-1911 allowing end-users to specify backend protocols * fix tab vs spaces * indicate that per-protocol structs can be extended in the future * Add small intro * Address Candace's feedback * add an option to consider re-using Route resources * expand appProtocol based on findings * how would we configure certain protocols via appProtocol * rewrite doc to focus on supporting Kubernetes Standard Application Protocols * fix link * add back multiple protocols on the same port * change multiple protocol example to TCP/UDP which is what K8s supports * fix links * move alternate API under a detail block so people don't confuse that as the proposed API * add a compatability table * fix links * Drop raw support Upstream K8s folks did not want to add this constant. See: https://github.com/kubernetes/enhancements/pull/4106 * tweak default protocol clause * tweak tables so columns are the same * limit multiplex protocols to just those supported by K8s (UDP/TCP for now) * tweak new appProtocol section - include an example of using a gateway api special prefix * label open questions explicitly * three kubernetes.io appProtocols * incompatible backend protocols make the backend invalid * add godoc to HTTPBackendRef explaining and invalid backend * drop alternative example because people keep commenting on it * Drop alternative mechanic that's used in GAMMA * address Nicks comments * add references to Google and AWS's approach to the problem * TLSConnectionPolicy => BackendTLSPolicy * address Rob's feedback * drop SCTP multiplexing question * absence of appProtocol doesn't mean implementations disable features * multiplexing UDP/TCP is optional * TLSRoute paired to non-TLS listener doesn't make sense * fix casing on some section titles * specific protocol support is optional * Update geps/gep-1911.md Co-authored-by: Rob Scott --------- Co-authored-by: Rob Scott --- apis/v1beta1/httproute_types.go | 5 + geps/gep-1911.md | 183 ++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 189 insertions(+) create mode 100644 geps/gep-1911.md diff --git a/apis/v1beta1/httproute_types.go b/apis/v1beta1/httproute_types.go index fb6809ddb3..665758a735 100644 --- a/apis/v1beta1/httproute_types.go +++ b/apis/v1beta1/httproute_types.go @@ -1134,12 +1134,17 @@ type HTTPBackendRef struct { // case, the Reason must be set to `RefNotPermitted` and the Message of the // Condition must explain which cross-namespace reference is not allowed. // + // * It refers to a Kubernetes Service that has an incompatible appProtocol + // for the given Route type + // // Support: Core for Kubernetes Service // // Support: Implementation-specific for any other resource // // Support for weight: Core // + // Support for Kubernetes Service appProtocol: Extended + // // +optional BackendRef `json:",inline"` diff --git a/geps/gep-1911.md b/geps/gep-1911.md new file mode 100644 index 0000000000..817b376b82 --- /dev/null +++ b/geps/gep-1911.md @@ -0,0 +1,183 @@ +# GEP-1911: Backend Protocol Selection + +* Issue: [#1911](https://github.com/kubernetes-sigs/gateway-api/issues/1911) +* Status: Provisional + +(See status definitions [here](overview.md#status).) + +## TLDR + +Not all implementations support automatic protocol selection. Even in some cases protocols are disabled without an explicit opt-in (eg. websockets with Contour & NGINX). Thus application developers need the ability to specify the protocol(s) that their application supports. + +## Goals + +- Support protocols that can have a Gateway `*Route` resource as a frontend +- Standardize Gateway API implementations on the protocols & constants defined by the Kubernetes [Standard Application Protocols (KEP-3726)][kep-3726] +- Support backends with multiple protocols on the same port (ie. tcp/udp) + +## Non-Goals + +- Backend TLS (covered in [GEP-1897](https://github.com/kubernetes-sigs/gateway-api/issues/1897)) +- Additional protocol specific configuration +- Disabling Protocols + +## Introduction + +Since Kubernetes 1.20 the [`core/v1.Service`][k8s-service] and [`core/v1.EndpointSlice`][k8s-endpointslices] resource has a stable `appProtocol` field. It's purpose is to allow end-users to specify an application protocol (L7) for each service port. + +Originally the use of this field in the Gateway API was rejected in [GEP-1282](geps/gep-1282#non-goals): +> v1.Service’s appProtocol field is not fit for purpose, because it is defined as accepting values either from the IANA Service Name registry, or domain-prefixed values and we need more flexibility than that. + +Since then a Kubernetes enhancement proposal was created [KEP-3726][kep-3726] to repurpose `appProtocol` to include a convention for protocols that are not IANA service names. This would involve prefixing protocol names with `kubernetes.io/*`. + +Note: Kubernetes will automatically create `EndpointSlices` for `Services` that have a selector. [Custom `EndpointSlices`](https://kubernetes.io/docs/concepts/services-networking/service/#custom-endpointslices) can manually be created. + +## API Semantics + +A Gateway implementation MUST recognize the Kubernetes Standard Application Protocols ([KEP-3726][kep-3726]) for specifying the protocol for a backend reference in a Gateway API `*Route` resource + +Thus when a `*Route` points to a Kubernetes Service resource the backend protocol for each port can be specified by + +- setting the `appProtocol` field on the Kubernetes `Service` +- setting the `appProtocol` field on an `Endpoint`/`EndpointSlice` object associated with a Kubernetes `Service` + +At the moment there exists three defined constants: + +- `kubernetes.io/h2c` - HTTP/2 over cleartext as described in [RFC7540](https://www.rfc-editor.org/rfc/rfc7540) +- `kubernetes.io/ws` - WebSocket over cleartext as described in [RFC6445](https://www.rfc-editor.org/rfc/rfc6455) +- `kubernetes.io/wss` - WebSocket over TLS as described in [RFC6455](https://www.rfc-editor.org/rfc/rfc6455) + +### New Protocols & Reserved Prefix + +To add support for a new protocol it should first become a Kubernetes Standard Application Protocol by updating the [KEP-3726][kep-3726]. [KEP-3726][kep-3726] also states the `appProtocol` field accepts a domain-prefixed implementation specific value. Thus, if the suggested protocol is not suited to have a `kubernetes.io/*` prefix, then the Gateway API MAY support the new protocol using it's own prefix `gateway.networking.k8s.io/*`. Please make a PR to this GEP. + +For example we may want to add a sentinel `appProtocol` value that prevents Gateway implementations from discovering the protocol of the application. Instead they should just refer to the Service's `protocol` field. Such a constant was rejected upstream (https://github.com/kubernetes/enhancements/pull/4106) but as an example it could be defined in a future addition to this GEP as `gateway.networking.k8s.io/no-sniff`. + +### Default Protocols + +If a Service `appProtocol` isn't specified an implementation MAY infer the backend protocol through its own means. Implementations MAY infer the protocol from the `Route` type referring to the backend Service. + +Absence of the `appProtocol` field does not imply the implementation should disable any features (eg. websocket upgrades). + +### Multiple Protocols on the Same Port + +Only the Kubernetes `Service` `protocol` field supports multiple protocols on the same port. See the details in [KEP-1435][kep-1435]. + +Implementations MAY support multiplexing `TCP` and `UDP` on the same port. Otherwise implementations MUST set `ResolvedRefs` condition to `False` with the Reason `UnsupportedProtocol` with a clear message that multiplexing is not supported. + +Currently Kubernetes `Service` API only allows different `appProtocol` values for the same port when `protocol` fields differs. At this time there seems to be interest in changing `appProtocol` to be a list in order to faciliate this use-case. + +### Supporting Protocols + +If a Route attached to a Gateway is not able to send traffic to the backend using the specified protocol then the backend is considered invalid. Implementations MUST set `ResolvedRefs` condition to `False` with the Reason `UnsupportedProtocol`. + +Implementations MAY support the following combinations below: + +ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported +-|-|-|- +`TCP` | `kubernetes.io/h2c` | `GRPCRoute` | Yes [1] +`TCP` | `kubernetes.io/h2c` | `HTTPRoute` | Yes +`TCP` | `kubernetes.io/ws` | `HTTPRoute` | Yes +`TCP` | `kubernetes.io/wss` | `TLSRoute` | Yes + +1. GRPC works over h2c - so a GRPCRoute should be able to connect to an h2c backend + +Implementations MAY support the following combinations below: + +ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported +-|-|-|- +ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported +`TCP` | `kubernetes.io/wss` | `HTTPRoute` | Conditional [1] + +1. Only if there is a corresponding `BackendTLSPolicy` - see [GEP-1897](geps/gep-1897) + +## Open Questions + +1. TLSRoute & UDP protocol + +TLS over UDP seems to be a thing via QUIC/HTTP3 [ref](https://www.smashingmagazine.com/2021/08/http3-core-concepts-part1/). +Likewise there's also [DTLS](https://en.wikipedia.org/wiki/Datagram_Transport_Layer_Security). But it's unclear if Gateway's TLSRoute +applies to an underlying UDP protocol. + +2. Websockets & HTTP/2/3 + +Should we upstream new constants for websocket over [HTTP/2](https://www.rfc-editor.org/rfc/rfc8441.html) & [HTTP/3](https://www.rfc-editor.org/rfc/rfc9220.html) ? HTTP/3 makes things more complicated since its supports UDP as the underlying protocol. + +## Alternatives + +### Single Meta-resource + +The first pass of this GEP proposed a new meta-resource [GEP-713](geps/gep-713) called `BackendProtocol`. + +This allows end-users to specify a list of ports and a list of corresponding protocols that that single +port supports. + +This was dropped in favour of supporting Kubernetes Standard Application Protocols. + +### Multiple Protocol Meta-resources + +Rather than bundle protocol details into a single resource an alternative would be to create distinct meta resources. +ie. `HTTP2Backend`, `GPRCBackend`, `WebsocketBackend`. + +The advantages of this approach are: + +- Easy to introduce new protocols +- Definitions/types would be simpler + +The disadvantages of this approach are: + +- N resources for N protocols need to be created to describe a single backend +- No easy mechanic to specify priority of protocols + +### Adding Properties on Gateway Route Objects + +From [GEP-1282](geps/gep-1282#tldr): +> some types of configuration requested by users are more about defining functionality that describes capabilities of the backend more than the route you take to get to the backend. + +Backend protocol is specifying capabilities. This configuration is less about routing. + +### Kubernetes Service - Expanding Protocol field + +The `protocol` field on a Kubernetes service is used to specify a L4 protocol over IP. This field isn't appropriate to describe protocols +that operate at a higher 'application' level (eg. HTTP/GRPC etc.) + +### Extending Kubernetes Service + +This is considered untenable due to the 'the turnaround time for those changes can be years.' ([ref-1282](geps/gep-1282#non-goals)) + +### Unstructured Data/Special Values + +Unstructured data refers to using labels and annotations. + +From [GEP-1282](geps/gep-1282#non-goals): +> these are very sticky and hard to get rid of once you start using them. + +Special values refers to using special strings in existing Kubernetes Resources. +For example Istio allows for protocol to be specified by prefixing the Kubernetes +Service's port name with the protocol (ie. `http-`, `grpc-`). This approach is +limiting as it doesn't allow for multiple protocols on the same port and future +configuration per protocol. One protocol per port may be relaxed in the future see +[KEP 1435][kep-1435] + +Additionally, annotations are not self-documenting unlike CRD fields which can display +documentation via `kubectl explain` + +## References + +- [GitHub Discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1244) +- GEP-1282 - Describing Backend Properties + - [GEP](geps/gep-1282) + - [Issue](https://github.com/kubernetes-sigs/gateway-api/issues/1911) +- [GEP-713 - Metaresources](geps/gep-713) +- [Linkerd Protocol Detection](https://linkerd.io/2.12/features/protocol-detection/) +- [Istio Protocol Selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/) +- Contour Protocol Selection + - [Websockets](https://projectcontour.io/docs/1.24/config/websockets/) + - [GRPC](https://projectcontour.io/docs/1.24/guides/grpc/#httpproxy-configuration) +- [AWS Gateway Protocol Selection](https://github.com/aws/aws-application-networking-k8s/blob/a277fb39449383f53cd7d1e5576b4fa190a1a853/config/crds/bases/application-networking.k8s.aws_targetgrouppolicies.yaml#L109) +- [Google GKE AppProtocol Selection](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#https_tls_between_load_balancer_and_your_application) + +[k8s-service]: https://kubernetes.io/docs/concepts/services-networking/service/ +[k8s-endpointslices]: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/ +[kep-3726]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3726-standard-application-protocols +[kep-1435]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1435-mixed-protocol-lb diff --git a/mkdocs.yml b/mkdocs.yml index 4c06a17712..3e2cbc0c86 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -94,6 +94,7 @@ nav: - geps/gep-1619.md - geps/gep-1867.md - geps/gep-1897.md + - geps/gep-1911.md - geps/gep-2162.md # - Implementable: # -