[1.17] Ensure endpoints for upstreams are listed within watchNamespaces

[davidjumani]

Backport of #9872

Description

When following the production recommendations, the following error is seen in the gloo pod

E1005 20:41:30.275565       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
W1005 20:42:21.655454       1 reflector.go:533] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
E1005 20:42:21.655512       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope

This is caused by the combination of the following settings :

discovery:
  enabled: false
global:
  glooRbac:
    namespaced: true
settings:
  disableKubernetesDestinations: true

• Since discovery is disabled, no upstream are created
• Since the rbac is namespaced, all the roles and bindings are namespace scoped
• Since disableKubernetesDestinations, no special kube client is created to fetch the endpoints

In this scenario, when the Kubernetes upstream plugin tries to watch upstreams and no upstream exists, the list of namespaces to watch that contain upstreams is empty. An empty namespace list by convention means to watch all namespaces. However the roles are only namespace scoped and not cluster wide. This leads to the watcher trying to list all upstreams in all namespaces that leads to the following error and the gloo pod unable to come up.

This PR fixess it by ensuring that if no upstreams exist, that we fallback to only watching upstreams within the watchNamespaces

Steps to reproduce :

• Deploy edge with the following settings :

helm upgrade --install -n gloo-system --create-namespace gloo ./_test/gloo-1.0.0-ci1.tgz  --values test/kube2e/helm/artifacts/helm.yaml --set settings.disableKubernetesDestinations=true --set global.glooRbac.namespaced=true

• The gloo pod has the following error

{"level":"info","ts":"2024-08-07T15:59:11.583Z","logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:193","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.0.0-ci1"}
W0807 15:59:11.583958       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
E0807 15:59:11.583984       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
W0807 15:59:13.093690       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
E0807 15:59:13.093758       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
W0807 15:59:14.882132       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
E0807 15:59:14.882210       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope

With this fix, the gloo pod comes up without any issues

Context

#5885
#8801

Testing steps

• Added unit tests
• Updated the helm kubernetes tests

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[solo-changelog-bot]

Issues linked to changelog:
#5885