Daskhub Controller error: Error in cluster informer

[kyprifog]

Suddenly receiving:

controller-data-ops-dask-gateway-5956b46957-fxfxc controller [E 2020-11-09 21:47:15.213 KubeController] Error in cluster informer, retrying...
controller-data-ops-dask-gateway-5956b46957-fxfxc controller Traceback (most recent call last):
controller-data-ops-dask-gateway-5956b46957-fxfxc controller   File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/utils.py", line 161, in run
controller-data-ops-dask-gateway-5956b46957-fxfxc controller     initial = await method(**self.method_kwargs)
controller-data-ops-dask-gateway-5956b46957-fxfxc controller   File "/usr/local/lib/python3.8/site-packages/dask_gateway_server/backends/kubernetes/utils.py", line 59, in func
controller-data-ops-dask-gateway-5956b46957-fxfxc controller     return await method(*args, **kwargs)
controller-data-ops-dask-gateway-5956b46957-fxfxc controller   File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/api_client.py", line 180, in __call_api
controller-data-ops-dask-gateway-5956b46957-fxfxc controller     response_data = await self.request(
controller-data-ops-dask-gateway-5956b46957-fxfxc controller   File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/rest.py", line 192, in GET
controller-data-ops-dask-gateway-5956b46957-fxfxc controller     return (await self.request("GET", url,
controller-data-ops-dask-gateway-5956b46957-fxfxc controller   File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/rest.py", line 186, in request
controller-data-ops-dask-gateway-5956b46957-fxfxc controller     raise ApiException(http_resp=r)
controller-data-ops-dask-gateway-5956b46957-fxfxc controller kubernetes_asyncio.client.exceptions.ApiException: (401)
controller-data-ops-dask-gateway-5956b46957-fxfxc controller Reason: Unauthorized
controller-data-ops-dask-gateway-5956b46957-fxfxc controller HTTP response headers: <CIMultiDictProxy('Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 09 Nov 2020 21:47:15 GMT', 'Content-Length': '129')>
controller-data-ops-dask-gateway-5956b46957-fxfxc controller HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Seems related to upgrading from 4.5.1 to 4.5.4 daskhub version, but still not sure.

[kyprifog]

Also receiving:

traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik panic: Timeout while stopping traefik, killing instance ✝
traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik
traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik goroutine 1694 [running]:
traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik github.com/containous/traefik/v2/pkg/server.(*Server).Close.func1(0x29c8280, 0xc000133320)
traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik  /go/src/github.com/containous/traefik/pkg/server/server.go:87 +0x10c
traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik created by github.com/containous/traefik/v2/pkg/server.(*Server).Close
traefik-data-ops-dask-gateway-564dd4b857-f5v5f traefik  /go/src/github.com/containous/traefik/pkg/server/server.go:82 +0x82

[kyprifog]

After alot of digging it turns out that this was because the version of pangeo base-notebook I was using was in kubespawner_override not compatible wit the new version of jupyterhub.

[consideRatio]

❤️ for the followup @kyprifog!