Setting settings.disableKubernetesDestinations && global.glooRbac.namespaced leads to panic in gloo due to RBAC issues

[davidjumani]

Gloo Edge Product

Open Source

Gloo Edge Version

v1.15.x

Kubernetes Version

All

Describe the bug

When setting settings.disableKubernetesDestinations && global.glooRbac.namespaced as true as part of the production recommendations mentioned here, the gloo pod errors with

E1005 20:41:30.275565       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
W1005 20:42:21.655454       1 reflector.go:533] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
E1005 20:42:21.655512       1 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:gloo-system:gloo" cannot list resource "endpoints" in API group "" at the cluster scope
{"level":"panic","ts":"2023-10-05T20:42:37.677Z","logger":"gloo.v1.event_loop.setup","caller":"setup/setup_syncer.go:648","msg":"failed warming up endpoints - consider adjusting endpointsWarmingTimeout","version":"1.0.0-ci","warmTimeoutDuration":300,"stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGlooWithExtensions\n\t/Users/djumani/lab/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:648\ngithub.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGloo\n\t/Users/djumani/lab/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:450\ngithub.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.(*setupSyncer).Setup\n\t/Users/djumani/lab/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:424\ngithub.com/solo-io/gloo/pkg/utils/setuputils.(*SetupSyncer).Sync\n\t/Users/djumani/lab/gloo/pkg/utils/setuputils/setup_syncer.go:60\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*setupEventLoop).Run.func1\n\t/Users/djumani/lab/gloo/projects/gloo/pkg/api/v1/setup_event_loop.sk.go:107"}
panic: failed warming up endpoints - consider adjusting endpointsWarmingTimeout

goroutine 139 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2, 0x4000b84ea0, {0x0, 0x4000945c00, 0x3f0})
	/Users/djumani/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:198 +0xc0
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x4000b84ea0, {0x40006d0b80, 0x1, 0x2})
	/Users/djumani/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:264 +0x4e0
go.uber.org/zap.(*SugaredLogger).log(0x4000812460, 0x4, {0x37ca41a, 0x48}, {0x0, 0x0, 0x0}, {0x4000a7e260, 0x2, 0x2})
	/Users/djumani/go/pkg/mod/go.uber.org/zap@v1.24.0/sugar.go:295 +0x140
go.uber.org/zap.(*SugaredLogger).Panicw(0x4000812460, {0x37ca41a, 0x48}, {0x4000a7e260, 0x2, 0x2})
	/Users/djumani/go/pkg/mod/go.uber.org/zap@v1.24.0/sugar.go:235 +0x60
github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGlooWithExtensions({{0x4000938e10, 0xb}, {0x400007408e, 0xb}, {0x400043fdb0, 0x1, 0x1}, {0x3ae4140, 0x40009b4b00}, {0x3b31080, ...}, ...}, ...)
	/Users/djumani/lab/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:648 +0x17f0
github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGloo({{0x4000938e10, 0xb}, {0x400007408e, 0xb}, {0x400043fdb0, 0x1, 0x1}, {0x3ae4140, 0x40009b4b00}, {0x3b31080, ...}, ...})
	/Users/djumani/lab/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:450 +0x170
github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.(*setupSyncer).Setup(0x40004c2380, {0x3b2a1b0, 0x40009f3290}, {0x3b2bcc8, 0x400045cb60}, {0x3b2e6e0, 0x4000b00580}, 0x40007e4b40, {0x3b05628, 0x40000a29e0})
	/Users/djumani/lab/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:424 +0x1574
github.com/solo-io/gloo/pkg/utils/setuputils.(*SetupSyncer).Sync(0x4000b0b380, {0x3b2a1b0, 0x40009f3290}, 0x400090af90)
	/Users/djumani/lab/gloo/pkg/utils/setuputils/setup_syncer.go:60 +0x340
github.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*setupEventLoop).Run.func1()
	/Users/djumani/lab/gloo/projects/gloo/pkg/api/v1/setup_event_loop.sk.go:107 +0x2a0
created by github.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*setupEventLoop).Run in goroutine 1
	/Users/djumani/lab/gloo/projects/gloo/pkg/api/v1/setup_event_loop.sk.go:88 +0x478
[time: 4m36s] [16:42:38] ~/lab/gloo/projects/gateway/pkg/translator ➦ fix-readiness-check $

Setting settings.endpointsWarmingTimeout to 0s is a workaround but the error still occurs.

Expected Behavior

It should not error out

Steps to reproduce the bug

Follow the recommendations here
If settings.disableKubernetesDestinations && global.glooRbac.namespaced is not set, the deployment is successful

Additional Environment Detail

No response

Additional Context

No response

[davidjumani]

Testing a bit more, it appears that setting global.glooRbac.namespaced as true along with this setting causes the issue

[github-actions]

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.

[davidjumani]

Closing as this will be fixed as part of #5885