Kafka scaler | Scale to 0 during Kafka broker replacement even if the fallback is configured #4098

sdomme · 2023-01-12T12:58:27Z

Report

Hello,

we periodically encounter an issue with some of the ScaledObjects with the Kafka trigger during Kafka broker replacement. Even if we configure the fallback the deployment is scaled to 0. In the case, where I will provide shrinked operator logs for it took approx. 2h for the fallback mechanism to kick in.

Expected Behavior

During faulty Kafka responses, deployment goes to the amount of PODs configured in fallback.replicas

Actual Behavior

Deployment goes to 0 during Kafka broker replacement and recovers after a long period to the number set in fallback.replicas

Steps to Reproduce the Problem

Here the ScaledObject we use

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-kafka-scaling
  namespace: zzz
spec:
  cooldownPeriod: 300
  fallback:
    failureThreshold: 3
    replicas: 1
  idleReplicaCount: 0
  maxReplicaCount: 2
  minReplicaCount: 1
  pollingInterval: 30
  scaleTargetRef:
    name: foo-deployment
  triggers:
    - authenticationRef:
        name: keda-trigger-kafka-auth
      metadata:
        activationLagThreshold: '1'
        allowIdleConsumers: 'false'
        bootstrapServers: bootstrapserver:28751
        consumerGroup: consumergroup
        lagThreshold: '50000'
        offsetResetPolicy: earliest
        scaleToZeroOnInvalidOffset: 'false'
        topic: FOO
      type: kafka

Logs from KEDA operator

2023-01-10T18:42:05Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error listing consumer group offsets: kafka server: Request was for a consumer group that is not coordinated by this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getConsumerAndProducerOffsets.func1
	/workspace/pkg/scalers/kafka_scaler.go:461
2023-01-10T18:42:05Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 0"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T18:42:05Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 1"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
.
.
.
2023-01-10T20:06:37Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error listing consumer group offsets: kafka server: Request was for a consumer group that is not coordinated by this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getConsumerAndProducerOffsets.func1
	/workspace/pkg/scalers/kafka_scaler.go:461
2023-01-10T20:06:37Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 2"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T20:06:37Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 0"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
.
.
.
2023-01-10T20:10:07Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error listing consumer group offsets: kafka server: Request was for a consumer group that is not coordinated by this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getConsumerAndProducerOffsets.func1
	/workspace/pkg/scalers/kafka_scaler.go:461
2023-01-10T20:10:08Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "keda-clickhouse-ingester-scaling", "scaledObject.Namespace": "inventory-platform-services", "scaleTarget.Name": "clickhouse-ingester", "error": "kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T20:10:09Z	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "keda-kafka-scaling", "scaledObject.Namespace": "zzz", "scaleTarget.Name": "foo-deployment", "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T20:10:09Z	INFO	scaleexecutor	Successfully set ScaleTarget replicas count to ScaledObject fallback.replicas	{"scaledobject.Name": "keda-kafka-scaling", "scaledObject.Namespace": "zzz", "scaleTarget.Name": "foo-deployment", "Original Replicas Count": 0, "New Replicas Count": 1}

KEDA Version

2.8.1

Kubernetes Version

1.23

Platform

Amazon Web Services

Scaler Details

Kafka

Anything else?

From this Graph (kube_deployment_status_replicas) you can see the scaling of the Deployment going to 0. The time fits the log I provided.

The text was updated successfully, but these errors were encountered:

JorTurFer · 2023-01-12T21:40:18Z

Hi
Could you share a bigger part of the log please? I'd like to see more than 2 failures in a row because the fallback is configured for 3 or more and some logs could be there

sdomme · 2023-01-13T11:08:58Z

Hi, Thanks for looking into it. Sure, I can provide more, but there is actually nothing else specific. The log is full of information of processing other ScaledObjects of the same kind but for different kafka topics. But it is always the same schema.

After the error "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes" immediately comes the "scaleexecutor Successfully set ScaleTarget replicas count to ScaledObject fallback.replicas"

If the error is "error": "error finding offset block for topic ... it is repeating a lot of times without recovering.

Here is a bigger log snippet. I had to replace the clear names:

2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error listing consumer group offsets: kafka server: Request was for a consumer group that is not coordinated by this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getConsumerAndProducerOffsets.func1
	/workspace/pkg/scalers/kafka_scaler.go:461
2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 0"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 2"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 4"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 3"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 5"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:00:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 1"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:00:40Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "7040e42d-fbc0-40d8-bbbe-74ed3d440874"}
2023-01-10T19:00:40Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "1e8f54a8-7eb4-44a2-a978-054e1ddfcb29"}
2023-01-10T19:00:41Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "d2d22dbd-3cdd-47f6-9d84-7b346e283ea1"}
2023-01-10T19:00:41Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "9da0a779-54a5-461b-929a-028a3d3c251b"}
2023-01-10T19:00:41Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "5b17e3c4-e78f-414c-be55-43c500c662fa"}
2023-01-10T19:00:55Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "29987902-7631-406e-8834-5cfe890f79d2"}
2023-01-10T19:00:56Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "b3e020ec-6ca9-4882-80d9-3432f6178d64"}
2023-01-10T19:00:56Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "e247f2d1-a6da-4551-9440-32b46af0fff1"}
2023-01-10T19:00:56Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--: "24c49e2f-c202-4024-b751-b14ade59a344"}
2023-01-10T19:00:57Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": --DIFFERENT-OBJECT--, "reconcileID": "a2f8e800-8481-4717-a8b6-b3bdbdb26586"}
2023-01-10T19:01:01Z	ERROR	scalehandler	Error getting scale decision	{--DIFFERENT-OBJECT--, "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:01Z	INFO	scaleexecutor	Successfully set ScaleTarget replicas count to ScaledObject fallback.replicas	{--DIFFERENT-OBJECT--, "Original Replicas Count": 4, "New Replicas Count": 4}
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error listing consumer group offsets: kafka server: Request was for a consumer group that is not coordinated by this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getConsumerAndProducerOffsets.func1
	/workspace/pkg/scalers/kafka_scaler.go:461
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 3"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 4"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 5"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 1"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 0"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:06Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 2"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:06Z	ERROR	scalehandler	Error getting scale decision	{--DIFFERENT-OBJECT--, "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "0c78f251-c5c8-4377-86a3-338b3c52ccb4"}
2023-01-10T19:01:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "fc775a0e-7367-495e-8158-d0beda677166"}
2023-01-10T19:01:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "5c929eb3-feb5-4851-b790-f35f1c13c749"}
2023-01-10T19:01:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "429efe40-254f-4b6c-afcb-6f2686bc95f3"}
2023-01-10T19:01:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "c7b813e8-77e7-4a65-92df-13b63c63c7f3"}
2023-01-10T19:01:25Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "934b802c-baae-465a-9433-dc51234f57e0"}
2023-01-10T19:01:26Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {--DIFFERENT-OBJECT--, "reconcileID": "37554aad-bc97-49c0-8964-f903cfaeeeff"}
2023-01-10T19:01:29Z	ERROR	scalehandler	Error getting scale decision	{--DIFFERENT-OBJECT--, "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:29Z	INFO	scaleexecutor	Successfully set ScaleTarget replicas count to ScaledObject fallback.replicas	{--DIFFERENT-OBJECT--, "Original Replicas Count": 4, "New Replicas Count": 4}
2023-01-10T19:01:31Z	ERROR	scalehandler	Error getting scale decision	{--DIFFERENT-OBJECT--, "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:31Z	INFO	scaleexecutor	Successfully set ScaleTarget replicas count to ScaledObject fallback.replicas	{--DIFFERENT-OBJECT--, "Original Replicas Count": 4, "New Replicas Count": 4}
2023-01-10T19:01:31Z	INFO	Reconciling ScaledObject	{--DIFFERENT-OBJECT--, "reconcileID": "f9a92b15-d75e-49ce-b222-168aae404ff4"}
2023-01-10T19:01:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error listing consumer group offsets: kafka server: Request was for a consumer group that is not coordinated by this broker"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getConsumerAndProducerOffsets.func1
	/workspace/pkg/scalers/kafka_scaler.go:461
2023-01-10T19:01:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 2"}
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).getTotalLag
	/workspace/pkg/scalers/kafka_scaler.go:511
github.com/kedacore/keda/v2/pkg/scalers.(*kafkaScaler).IsActive
	/workspace/pkg/scalers/kafka_scaler.go:268
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
	/workspace/pkg/scaling/cache/scalers_cache.go:89
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
2023-01-10T19:01:36Z	ERROR	kafka_scaler		{"type": "ScaledObject", "namespace": "zzz", "name": "keda-kafka-scaling", "error": "error finding offset block for topic FOO and partition 1"}

JorTurFer · 2023-01-13T11:41:59Z

Thanks for the info!
I have discovered that we are ignoring the error during the partition iteration , I think this isn't correct.
IDK if we should return an error if a single partition fails, but if all of them fails we definitively should return error, WDYT @zroubalik ?

zroubalik · 2023-01-16T10:53:16Z

That might be the problematic part, @sdomme are you willing to fix that part of code and test if it solves your problem?

sdomme · 2023-01-17T08:22:44Z

I don't have the right skills to contribute in this regards. Sorry.
On the Kafka side we are relying on an external partner and can't afaik influence the broker exchange.

theurichde · 2023-01-18T16:22:31Z

Hi there! @sdomme and me are going to have a look at the code and try to fix the problem. Let's see where it goes 😄

JorTurFer · 2023-01-18T16:37:51Z

I have assigned the issue to you to track the work in progress, if you want I can unassign it if you have troubles with the fix

sdomme · 2023-01-20T15:01:08Z

@theurichde Thx for helping out.

sdomme added the bug Something isn't working label Jan 12, 2023

JorTurFer assigned sdomme and theurichde Jan 18, 2023

sdomme mentioned this issue Jan 20, 2023

fix(kafka-scaler): return error if partition processing fails #4149

Merged

3 tasks

zroubalik closed this as completed in #4149 Feb 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka scaler | Scale to 0 during Kafka broker replacement even if the fallback is configured #4098

Kafka scaler | Scale to 0 during Kafka broker replacement even if the fallback is configured #4098

sdomme commented Jan 12, 2023

JorTurFer commented Jan 12, 2023

sdomme commented Jan 13, 2023

JorTurFer commented Jan 13, 2023

zroubalik commented Jan 16, 2023

sdomme commented Jan 17, 2023

theurichde commented Jan 18, 2023

JorTurFer commented Jan 18, 2023

sdomme commented Jan 20, 2023

Kafka scaler | Scale to 0 during Kafka broker replacement even if the fallback is configured #4098

Kafka scaler | Scale to 0 during Kafka broker replacement even if the fallback is configured #4098

Comments

sdomme commented Jan 12, 2023

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

Anything else?

JorTurFer commented Jan 12, 2023

sdomme commented Jan 13, 2023

JorTurFer commented Jan 13, 2023

zroubalik commented Jan 16, 2023

sdomme commented Jan 17, 2023

theurichde commented Jan 18, 2023

JorTurFer commented Jan 18, 2023

sdomme commented Jan 20, 2023