Gloo Edge not starting correctly with an incorrect Upstream

[djannot]

Describe the bug
When an Upstream is incorrect (for example, it references a Kubernetes Service that doesn't exist), then Gloo Edge can't start

To Reproduce
Create an Upstream which references a Kubernetes Service that doesn't exist:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: bad-upstream
  namespace: gloo-system
spec:
  discoveryMetadata:
    labels:
      app: nginx
  kube:
    selector:
      app: nginx
    serviceName: doesntexist
    serviceNamespace: default
    servicePort: 80

Restart the gloo deployment, then the gateway-proxy deployment:

kubectl -n gloo-system rollout restart deploy/gloo
kubectl -n gloo-system rollout status deploy/gloo
kubectl -n gloo-system rollout restart deploy/gateway-proxy
kubectl -n gloo-system rollout status deploy/gateway-proxy 

No snapshot will be created and no service will be available

Additional context

• Gloo Edge version: reproduced with 1.6.8 EE and the latest 1.7 beta

[djannot]

Seems related to this issue as well: #4438

[kdorosh]

I was able to reproduce, and then fix by enabling route replacement.

This is by design. You can have the invalid route replaced by enabling route replacement in the Gloo Settings:

apiVersion: gloo.solo.io/v1
kind: Settings
metadata:
  annotations:
    meta.helm.sh/release-name: gloo
    meta.helm.sh/release-namespace: gloo-system
  labels:
    app: gloo
    app.kubernetes.io/managed-by: Helm
    gloo: settings
  name: default
  namespace: gloo-system
spec:
  discovery:
    fdsMode: WHITELIST
  discoveryNamespace: gloo-system
  gateway:
    readGatewaysFromAllNamespaces: false
    validation:
      allowWarnings: true
      alwaysAccept: true
      proxyValidationServerAddr: gloo:9988
  gloo:
    disableKubernetesDestinations: false
    disableProxyGarbageCollection: false
    enableRestEds: true
    invalidConfigPolicy:
      invalidRouteResponseBody: Gloo Gateway has invalid configuration. Administrators
        should run `glooctl check` to find and fix config errors.
      invalidRouteResponseCode: 404
      replaceInvalidRoutes: true # add this