Missing kubernetes deployment/service cause gloo to silently fail - break routing

[kevin-shelaga]

Describe the bug
Missing kubernetes deployment/service causes gloo to silently fail - breaks routing. It seems like all routing for all VSs fails when this error occurs, even if they do not reference the RT or US.

To Reproduce
Steps to reproduce the behavior:

helm install gloo glooe/gloo-ee --namespace gloo-system --create-namespace --set-string license_key=$KEY --version v1.8.0

cat <<EOF | kubectl apply -f -
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: default-petstore-8080
  namespace: gloo-system
spec:
  discoveryMetadata:
    labels:
      service: petstore
  kube:
    selector:
      app: petstore
    serviceName: petstore
    serviceNamespace: default
    servicePort: 8080
---  
apiVersion: gateway.solo.io/v1
kind: RouteTable
metadata:
  name: "my-rt"
  namespace: "gloo-system"
spec:
  routes:
    - matchers:
        - prefix: /
      routeAction:
        single:
          upstream:
            name: default-petstore-8080
            namespace: gloo-system
---
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: my-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - "*"  
    routes:
      - matchers:
        - prefix: /
        options:
          extauth:
            disable: true
        delegateAction:
          ref:
            name: "my-rt"
            namespace: "gloo-system"
EOF

 k logs deploy/gloo -n gloo-system

{"level":"warn","ts":1626295142.0915785,"logger":"gloo-ee.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:140","msg":"proxy gloo-system.gateway-proxy was rejected due to invalid config: 2 errors occurred:\n\t* invalid resource gloo-system.default-petstore-8080\n\t* Upstream name:\"default-petstore-8080\"  namespace:\"gloo-system\" references the service \"petstore\" which does not exist in namespace \"default\"\n\n\nAttempting to update only EDS information","version":"1.8.0"}

glooctl check

Checking deployments... OK
Checking pods... OK
Checking upstreams... OK
Checking upstream groups... OK
Checking auth configs... OK
Checking rate limit configs... OK
Checking VirtualHostOptions... OK
Checking RouteOptions... OK
Checking secrets... OK
Checking virtual services... OK
Checking gateways... OK
Checking proxies... OK
No problems detected.
I07 60206 request.go:645] Throttling request took 1.019111389s, request: GET:https://1.1.1.1/apis/acme.cert-manager.io/v1alpha3?timeout=32s
Detected Gloo Federation!

glooctl get us

...
| default-petstore-8080                                | Kubernetes | Accepted | svc name:      petstore             |
|                                                      |            |          | svc namespace: default              |
|                                                      |            |          | port:          8080                 |
|                                                      |            |          |                                     |
...

Expected behavior

1. Warn reported in the gloo deployment should be an error.
2. Glooctl should report an error
3. The upstream with the missing K8s deployment should NOT be accepted

Additional context
Add any other context about the problem here, e.g.

• Gloo Edge version - v1.8.0
• Kubernetes version - 1.20

[sam-heilbron]

Kube plugin flags the upstream with an error: https://github.com/solo-io/gloo/blob/master/projects/gloo/pkg/plugins/kubernetes/plugin.go#L83

which should report that error: https://github.com/solo-io/gloo/blob/master/projects/gloo/pkg/translator/clusters.go#L73

we fail to sanitize the snapshot: https://github.com/solo-io/gloo/blob/master/projects/gloo/pkg/syncer/envoy_translator_syncer.go#L139

are we successfully calling the upstream_removing_sanitizer: https://github.com/solo-io/gloo/blob/master/projects/gloo/pkg/syncer/sanitizer/upstream_removing_sanitizer.go#L28?