Connection error from keda-operator-metrics-apiserver logs

[prashathsenthil]

Report

After updating KEDA to version 2.9.2 (via chart version 2.9.3) from version 2.8.1 the metrics apiserver logs shows connection issue from the keda operator service at port 9666

Expected Behavior

The keda metrics apiserver to be able to use gRPC connection to successfully get metrics from operator

Actual Behavior

When I deployed the latest KEDA version 2.9.2 started seeing the below error in the metrics apiserver logs,

I0201 19:41:19.967541 1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.mynamespace.svc.cluster.local:9666"

Err: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: "HTTP/1.1 502 Bad Gateway\r\n

Steps to Reproduce the Problem

Update KEDA version from 2.8.1 to 2.9.2

Logs from KEDA operator

I0201 19:41:19.967541       1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.<MYNAMESPACE>.svc.cluster.local:9666"

Err: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 502 Bad Gateway\\r\\n

KEDA Version

2.9.2

Kubernetes Version

1.23

Platform

Amazon Web Services

Scaler Details

No response

Anything else?

The keda service (13-keda-service.yaml) was introduced with the latest version referring to this issue #3920 as this service wasn't there in the older version

I can try KEDA_USE_METRICS_SERVICE_GRPC=false but as it will be deprecated, wanted to see how to resolve the issue

KEDA runs behind enterprise proxy and the values are set via HTTP_PROXY HTTPS_PROXY and NO_PROXY

[JorTurFer]

Hello,
I can see a typo in the url, the message says keda-operator..svc.cluster.local:9666 but it should be keda-operator.KEDA_NAMESPACE.svc.cluster.local:9666.
The value is set here https://github.com/kedacore/charts/blob/v2.9.3/keda/templates/22-metrics-deployment.yaml#L109
Could you check if that value is present? I think that the message you see is because there are missing configs after the update. If that line isn't provided, KEDA tries to generate it in fallback but KEDA namespace is required, and that variables is also missing

[prashathsenthil]

Hello, I can see a typo in the url, the message says keda-operator..svc.cluster.local:9666 but it should be keda-operator.KEDA_NAMESPACE.svc.cluster.local:9666. The value is set here https://github.com/kedacore/charts/blob/v2.9.3/keda/templates/22-metrics-deployment.yaml#L109 Could you check if that value is present? I think that the message you see is because there are missing configs after the update. If that line isn't provided, KEDA tries to generate it in fallback but KEDA namespace is required, and that variables is also missing

@JorTurFer its just a typo while I wrote it, it actually has the namespace name there ""address"="keda-operator.mynamespace.svc.cluster.local:9666""

[JorTurFer]

@zroubalik , Maybe we should exclude this communication from the proxied traffic, as it's internal traffic. I guess that we could explicitly set the values without the proxy to skip env vars configs
WDYT?

[prashathsenthil]

After adding cluster.local to NO_PROXY list, the 502 bad gateway error is resolved, however still seeing the below error in the metric server logs,

grpc: addrConn.createTransport failed to connect to "Addr": "keda-operator.mynamespace.svc.cluster.local:9666"

[JorTurFer]

After adding cluster.local to NO_PROXY list, the 502 bad gateway error is resolved, however still seeing the below error in the metric server logs,

grpc: addrConn.createTransport failed to connect to "Addr": "keda-operator.mynamespace.svc.cluster.local:9666"

Sorry, I missed your reply 😢

Is the error persistent? I ask because we print the failure message but we don't print the established connection message, if you see the error only once or twice, it's not a problem and it's normal.

[jstefankowski]

Has this been resolved ?

After upgrading keda from 2.6.1 to 2.10.1 autoscaling is not working and seeing this error in apiserver log:

Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup keda-sa.keda.svc.cluster.local on 10.106.0.2:53: no such host"
1 client.go:93] keda_metrics_adapter/provider "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server"
1 provider.go:110] keda_metrics_adapter/provider "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "server"="keda-sa.keda.svc.cluster.local:9666"

[JorTurFer]

Has this been resolved ?
It is, yep.

Seeing that error during the startup could be normal if the operator is down, but it should be transitory.
Is your operator working?

[jstefankowski]

I'm new to Keda. How can I confirm the operator is working correctly ?

[zroubalik]

Also, did you change anything in the configuration? keda-sa in the part with the url doesn't seem right to me.

[jstefankowski]

No configuration changes whatsoever.

I think the url configured in metrics apiserver deployment should be the operator internal endpoint: keda-sa.keda:9666 instead of keda-sa.keda.svc.cluster.local:9666.

keda-sa is the operator service name.

[jstefankowski]

After changing the apiserver deployment metrics-service-address image arg sill getting the same error:
lookup keda-sa.keda.svc on 10.108.0.2:53: no such host

Wondering if the --metrics-service-address argument format is wrong. It is: --metrics-service-address=keda-sa.keda.svc:9666'
which translates to: ..svc:9666

[jstefankowski]

From operator logs and from testing keda by sending SQS messages keda is scaling out successfully.

2023-06-15T17:10:16Z INFO scaleexecutor Successfully updated ScaleTarget {"scaledobject.Name": "aws-sqs-queue-scaledobject-ds", "scaledObject.Namespace": "test-namespace", "scaleTarget.Name": "ds", "Original Replicas Count": 0, "New Replicas Count": 1}

Just trying to understand the root cause of the timeout errors and its potential impact on production deployments.

[zroubalik]

Imho there should be keda-operator instead of keda-sa - basically the name of K8s Service that fronts the operator Deployment.

[zroubalik]

keda-sa is the operator service name

I missed this note.

So you did some changes because the original name is keda-operator

[zroubalik]

From operator logs and from testing keda by sending SQS messages keda is scaling out successfully.

2023-06-15T17:10:16Z INFO scaleexecutor Successfully updated ScaleTarget {"scaledobject.Name": "aws-sqs-queue-scaledobject-ds", "scaledObject.Namespace": "test-namespace", "scaleTarget.Name": "ds", "Original Replicas Count": 0, "New Replicas Count": 1}

Just trying to understand the root cause of the timeout errors and its potential impact on production deployments.

It won't scale to more than 1 replica

[jstefankowski]

@zroubalik
Spot on. I just realized, that's exactly the behavior, keda won't scale to more than 1 replica.
Why is that ?

[jstefankowski]

keda-sa is the operator service name

I missed this note.

So you did some changes because the original name is keda-operator

I tried patching metrics-apiserver deployment by changing metric-service-address to the operator service endpoint which is
keda-sa.keda:9666 but to no avail.

[zroubalik]

Well, in first place we should ask why is the operator service name changed?

[jstefankowski]

That I'm not sure (inherited cluster). The operator name is set to equal service account hence keda-sa.
--set operator.name=${service_account_name} \

Do you think non default operator service name is the culprit here ?

[jstefankowski]

@zroubalik Also, can you expand on why keda "won't scale to more than 1 replica" ?

[Gershon-A]

Does someone have a solution?
I facing the same issue ....

[JorTurFer]

@Gershon-A , this issue was closed 5 months ago, I suggest opening another issue to track your case. There are multiple reasons that can produce this issue