Connection to and operations against Redis cluster take too long and timeout

[giuseppe-sirigu]

I have been trying to connect to a local Redis cluster built with the following docker-compose:

version: '2'
services:
  redis-node-0:
    image: docker.io/bitnami/redis-cluster:6.2
    restart: always
    volumes:
      - redis-cluster_data-0:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    ports:
      - "7000:6379"

  redis-node-1:
    image: docker.io/bitnami/redis-cluster:6.2
    restart: always
    volumes:
      - redis-cluster_data-1:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    ports:
      - "7001:6379"

  redis-node-2:
    image: docker.io/bitnami/redis-cluster:6.2
    restart: always
    volumes:
      - redis-cluster_data-2:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    ports:
      - "7002:6379"

  redis-node-3:
    image: docker.io/bitnami/redis-cluster:6.2
    restart: always
    volumes:
      - redis-cluster_data-3:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    ports:
      - "7003:6379"

  redis-node-4:
    image: docker.io/bitnami/redis-cluster:6.2
    restart: always
    volumes:
      - redis-cluster_data-4:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    ports:
      - "7004:6379"

  redis-node-5:
    image: docker.io/bitnami/redis-cluster:6.2
    restart: always
    volumes:
      - redis-cluster_data-5:/bitnami/redis/data
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
    environment:
      - 'REDIS_PASSWORD=bitnami'
      - 'REDISCLI_AUTH=bitnami'
      - 'REDIS_CLUSTER_REPLICAS=1'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
      - 'REDIS_CLUSTER_CREATOR=yes'
    ports:
      - "7005:6379"

volumes:
  redis-cluster_data-0:
    driver: local
  redis-cluster_data-1:
    driver: local
  redis-cluster_data-2:
    driver: local
  redis-cluster_data-3:
    driver: local
  redis-cluster_data-4:
    driver: local
  redis-cluster_data-5:
    driver: local

Then I installed redis-py-cluster==2.1.3 on my Python 3.9 environment and tried to run the following script:

import rediscluster
startup_nodes = [{"host": "127.0.0.1", "port": "7000"},
                             {"host": "127.0.0.1", "port": "7001"},
                             {"host": "127.0.0.1", "port": "7002"},
                             {"host": "127.0.0.1", "port": "7003"},
                             {"host": "127.0.0.1", "port": "7004"},
                             {"host": "127.0.0.1", "port": "7005"}]
client = rediscluster.RedisCluster(
    startup_nodes=startup_nodes,
    decode_responses=True,
    password=self.redis_psswd,
    skip_full_coverage_check=False
)
ping = client.ping()

But the commands timeout and I get the following error: Error 10060 connecting to 192.168.32.4:6379. A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected

Why is this happening?

[Grokzen]

@giuseppe-sirigu So this is a very common case of a misunderstanding between the internal node to node Redis cluster configuration and the client to cluster configuration.

When you setup a cluster the redis nodes will figure out what internal IP:PORT that each Redis node needs to use in order to connect to any other redis node in order to sync and do any operations it needs to do to maintain and have a working cluster. This however is not the same as what IP:PORT that a client needs to use in order to connect to a cluster. The client can in the case of docker use the externally available ip:port but internally redis needs to have another set of configuration in order to function and this is the internal 192.168.x.y:zzzz address that you are seeing. startup_nodes in the client case is just the routes and addresses that a client can take to make the initial connection into one of the cluster nodes, but once it is inside it will ask for the clusters view of the cluster and how each node is reachable in the cluster. This is where the error occurs as the internal cluster can be different from what the external clients needs and this shines through in the case of docker really well.

What is the solution? the simplest one is to use host network and to not have each node run on 6379 but the externally mapped ports 7000-7005 instead. Other solutions you will have to tinker out yourself. A tip is that you should look at the commands CLUSTER INFO CLUSTER NODES CLUSTER SLOTS as they are the internal commands that is used by this client to figure out the current cluster configuration once it has connected to one node.