Canary deployments with Gateway API between namespaces

[arapulido]

The Gateway API spec seems to expect each team to work on a single namespace, which feels a bit too rigid for certain use cases.

In my case, I would like to be able to do canary deployments that involve services that are not directly getting external traffic. This is a lot easier to do if you can do a different deployment in a second namespace, something like this:

kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: httproute1
  namespace: ns1
  labels:
    app: ecommerce
spec:
  gateways:
    allow: "All"
  hostnames:
    - "www.example.org"
  rules:
    - matches:
        - path:
            type: Prefix
            value: /
      forwardTo:
        - serviceName: service1
          port: 80
          weight: 50

kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: httproute2
  namespace: ns2
  labels:
    app: ecommerce
spec:
  gateways:
    allow: "All"
  hostnames:
    - "www.example.org"
  rules:
    - matches:
        - path:
            type: Prefix
            value: /
      forwardTo:
        - serviceName: service1
          port: 80
          weight: 50

This is not currently possible with the current spec and something that will be useful to have (to put an example, the NGINX Ingress controller allows to do this).

Similar, I am wondering if this other way to implement the same thing is allowed by the spec:

Basically, having a single HTTPRoute object, but that points to an ExternalService, pointing to the service in the second namespace

kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: httproute-external
  namespace: ns1
  labels:
    app: ecommerce
spec:
  gateways:
    allow: "All"
  hostnames:
    - "www.example.org"
  rules:
    - matches:
        - path:
            type: Prefix
            value: /
      forwardTo:
        - serviceName: service1
          port: 80
          weight: 50
        - serviceName: service1-external
          port: 80
          weight: 50

With service1-external looking like this:

kind: Service
apiVersion: v1
metadata:
  name: service1-external
spec:
  type: ExternalName
  externalName: service1.ns2.svc.cluster.local

[bowei]

We are currently discussion the addition of "inclusion" or "delegation" of HTTPRoute which may meet your criteria.

In your example, your HTTPRoute objects are the same -- how is this different than a simple traffic split? Can you post the nginx config you are thinking of?

    +-----------------+           +----------------+
    |                 |           |                |
    |  HTTPRoute   50%-------------> Service 1     |
    |              50%---------+  |                |
    +-----------------+        |  +----------------+
                               |  +----------------+
                               |  |                |
                               +-->> Service 2     |
                                  |                |
                                  +----------------+

[arapulido]

Sure, the difference would be that service 2 is on a different namespace, to make it easier to do traffic splits on whole applications, when the service that changes is not the first one.

These would be the nginx configs:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  namespace: ns1
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: www.example.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service1
            port:
              number: 80

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  namespace: ns2
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
spec:
  rules:
  - host: www.example.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service1
            port:
              number: 80

[mark-church]

There are actually two active discussions about making this possible as a 1st class capability of the API. #711 is the proposal for cross-Namespace Route->Service (by @robscott) and this is the proposal for cross-Namespace Route->Route (by @youngnick). Both are a bit early proposals but making steady progress.

Today there can be a cross-Namespace relationship between Gateways and Routes but not between any other resources. There are several other cross-Namespace relationships that are very useful though.

• Gateway-Route is useful when merging different Routes to the same Gateway, different teams that want full control of their routing config but want to share the same load balancer
• Route-Route is useful for Route delegation, or when a portion of the Route should be defined in one Namespace and sub-portion of it defined in another Namespace
• Route-Service is useful for having routing configured in different Namespace where the app is deployed (your use-case)
• Gateway-Secret is useful for when secrets shouldn't be accessed directly by the Gateway-owner. They can be referenced but not directly accessed.

The syntax could likely change and this is not in the API today, but your use-case could look something like the following. Here I've put the HTTPRoute in ns1 while the Services are in ns2 and ns3.

kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: cross-ns-route
  namespace: ns1
  labels:
    gateway: external-http
spec:
  hostnames:
    - "www.example.org"
  rules:
  - forwardTo:
    - serviceName: service1
      namespace: ns2
      port: 8080
      weight: 50
    - serviceName: service1
      namespace: ns3
      port: 8080
      weight: 90

@arapulido can I ask why you would like to do canaries between different Namespaces? Is this because it's simple to canary an entire stack because every service is using the same name within a Namespace?

[howardjohn]

Another good use cases is having my mail.ns Service and mail.google.com external service (ExternalName or similar), and all external services are defined in some external-services namespace or similar.

[arapulido]

@arapulido can I ask why you would like to do canaries between different Namespaces? Is this because it's simple to canary an entire stack because every service is using the same name within a Namespace?

Yes, it makes things clearer and if you happen to have some service hostnames hardcoded in the code, it would still work

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[robscott]

I think this has mostly been solved by cross-namespace forwarding from Routes in v1alpha2 (#741). Although we discussed using a round robin approach when multiple routes match a request, we ultimately determined that that would be too confusing, so instead we have pretty detailed conflict resolution guidelines on which Route needs to be chosen when multiple match:

gateway-api/apis/v1alpha2/httproute_types.go

Lines 139 to 155 in 165ecc3

	// Proxy or Load Balancer routing configuration generated from HTTPRoutes
	// MUST prioritize rules based on the following criteria, continuing on
	// ties. Precedence must be given to the the Rule with the largest number
	// of:
	//
	// * Characters in a matching non-wildcard hostname.
	// * Characters in a matching hostname.
	// * Characters in a matching path.
	// * Header matches.
	// * Query param matches.
	//
	// If ties still exist across multiple Routes, matching precedence MUST be
	// determined in order of the following criteria, continuing on ties:
	//
	// * The oldest Route based on creation timestamp.
	// * The Route appearing first in alphabetical order by
	// "<namespace>/<name>".

I'm going to go ahead and close this since I think we've addressed it now, but feel free to reopen if not.