Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Health-check process of keepalived service is not working properly while reloading keepalived #751

Open
cagatay23 opened this issue Aug 9, 2021 · 0 comments
Open

Health-check process of keepalived service is not working properly while reloading keepalived #751

cagatay23 opened this issue Aug 9, 2021 · 0 comments

Comments

@cagatay23
Copy link

cagatay23 commented Aug 9, 2021 •
edited
Loading

Health-check process of keepalived service is not working properly while reloading keepalived.

Aug  9 11:48:46 lb-stage-1 systemd: Reloading LVS and VRRP High Availability Monitor.
Aug  9 11:48:46 lb-stage-1 Keepalived[26324]: Reloading ...
Aug  9 11:48:46 lb-stage-1 Keepalived[26324]: Opening file '/etc/keepalived/keepalived.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived[26324]: Opening file '/etc/keepalived/keepalived_progressive.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived[26324]: Opening file '/etc/keepalived/keepalived_stream.conf'.
Aug  9 11:48:46 lb-stage-1 systemd: Reloaded LVS and VRRP High Availability Monitor.
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Reloading
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Got SIGHUP, reloading checker configuration
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Reloading
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Opening file '/etc/keepalived/keepalived.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Opening file '/etc/keepalived/keepalived.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: find dpdk interface -- dpdk0
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Opening file '/etc/keepalived/keepalived_progressive.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Opening file '/etc/keepalived/keepalived_progressive.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Opening file '/etc/keepalived/keepalived_stream.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Opening file '/etc/keepalived/keepalived_stream.conf'.
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Assigned address 192.168.100.2 for interface dpdk0.kni
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Assigned address fe80::6eb3:11ff:fe07:6552 for interface dpdk0.kni
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: Registering DPVS gratuitous ARP.
Aug  9 11:48:46 lb-stage-1 Keepalived_vrrp[26326]: VRRP sockpool: [ifindex(18), family(IPv4), proto(112), unicast(0), fd(10,11)]
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.4]:tcp:80 for VS [x.y.0.170]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.7]:tcp:80 for VS [x.y.0.170]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.4]:tcp:443 for VS [x.y.0.170]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.7]:tcp:443 for VS [x.y.0.170]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.10]:tcp:80 for VS [x.y.0.171]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.13]:tcp:80 for VS [x.y.0.171]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.10]:tcp:443 for VS [x.y.0.171]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.13]:tcp:443 for VS [x.y.0.171]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.5]:tcp:80 for VS [x.y.0.172]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.8]:tcp:80 for VS [x.y.0.172]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.5]:tcp:443 for VS [x.y.0.172]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.8]:tcp:443 for VS [x.y.0.172]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.11]:tcp:80 for VS [x.y.0.173]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.14]:tcp:80 for VS [x.y.0.173]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.11]:tcp:443 for VS [x.y.0.173]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.14]:tcp:443 for VS [x.y.0.173]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.1]:tcp:80 for VS [x.y.0.175]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.17]:tcp:80 for VS [x.y.0.175]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.1]:tcp:443 for VS [x.y.0.175]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.17]:tcp:443 for VS [x.y.0.175]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.1]:tcp:80 for VS [x.y.0.176]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.1]:tcp:443 for VS [x.y.0.176]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.2]:tcp:80 for VS [x.y.0.177]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.2]:tcp:443 for VS [x.y.0.177]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.31]:tcp:80 for VS [x.y.0.178]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.31]:tcp:443 for VS [x.y.0.178]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.18]:tcp:80 for VS [x.y.0.179]:tcp:80
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.5.18]:tcp:443 for VS [x.y.0.179]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.4]:tcp:443 for VS [x.y.0.210]:tcp:443
Aug  9 11:48:46 lb-stage-1 Keepalived_healthcheckers[26325]: Activating healthchecker for service [192.168.100.7]:tcp:443 for VS [x.y.0.210]:tcp:443

As can be seen from the /var/log/messages logs related to reloading keepalived above:

Reloading keepalived service activates health-check subprocess and health-checks are not return any info about success or failure. So, virtual server ips could not be added via ipvsadm.

(telnet to these servers are ok and these servers are reachable but could not be added with reload)

I tried kill -HUP $(cat /run/keepalived.pid), kill -HUP $(cat /run/checkers.pid) and pkill -HUP -F /run/checkers.pid but health-checks are not returned any info about success or failure , so same situation is repeated exactly.

If i pkill to health-check process ( pkill -f /run/checkers.pid ), health-checks return with success and added via ipvsadm. But, the pid of the health-check process is changed and reloading only adds new IP addresses.

For instance:
"x" and "y" addresses exist inside keepalived.conf at initial state. I added z and remove y. The result is "x", "y" and "z" addresses exist but "y" is not removed. (Expected result must be only "x" and "z" remains on the list, so "y" must be removed)

On the other hand,

As can be seen from the /var/log/messages logs related to restarting keepalived below:

Restarting is works like a charm. While restarting keepalived ( systemctl restart keepalived ), health-checks result in success and IP addresses inside keepalived.conf are added via ipvsadm.

Aug  9 12:20:01 lb-stage-1 systemd: Started Session 708 of user root.
Aug  9 12:20:01 lb-stage-1 systemd: Removed slice User Slice of root.
Aug  9 12:20:09 lb-stage-1 rsyslogd: -- MARK --
Aug  9 12:30:01 lb-stage-1 systemd: Created slice User Slice of root.
Aug  9 12:30:01 lb-stage-1 systemd: Started Session 709 of user root.
Aug  9 12:30:01 lb-stage-1 systemd: Removed slice User Slice of root.
Aug  9 12:40:01 lb-stage-1 systemd: Created slice User Slice of root.
Aug  9 12:40:01 lb-stage-1 systemd: Started Session 710 of user root.
Aug  9 12:40:01 lb-stage-1 systemd: Removed slice User Slice of root.
Aug  9 12:40:09 lb-stage-1 rsyslogd: -- MARK --
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]: Stopping
Aug  9 12:41:41 lb-stage-1 Keepalived_vrrp[26326]: (VI_1) sent 0 priority
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]: Keepalived_healthcheckers exited due to segmentation fault (SIGSEGV).
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Please report a bug at https://github.com/acassen/keepalived/issues
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   and include this log from when keepalived started, a description
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   of what happened before the crash, your configuration file and the details below.
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Also provide the output of keepalived -v, what Linux distro and version
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   you are running on, and whether keepalived is being run in a container or VM.
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   A failure to provide all this information may mean the crash cannot be investigated.
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   If you are able to provide a stack backtrace with gdb that would really help.
Aug  9 12:41:41 lb-stage-1 systemd: Stopping LVS and VRRP High Availability Monitor...
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Source version 2.0.19 
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Built with kernel headers for Linux 3.10.0
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Running on Linux 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Command line: '/etc/keepalived/sbin/keepalived' '-D'
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   Config options: LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:   System options: PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV6_ADVANCED_API RTA_ENCAP
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:                   RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:                   RTA_VIA FRA_OIFNAME IFA_FLAGS IP_MULTICAST_ALL NET_LINUX_IF_H_COLLISION
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:                   LIBIPTC_LINUX_NET_IF_H_COLLISION VRRP_VMAC IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:                   SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE SO_MARK SCHED_RT
Aug  9 12:41:41 lb-stage-1 Keepalived[26324]:                   SCHED_RESET_ON_FORK
Aug  9 12:41:41 lb-stage-1 kernel: [351731.964531] keepalived[26325]: segfault at 8 ip 00005654dffea118 sp 00007ffc80fe5940 error 4 in keepalived[5654dffc8000+7e000]
Aug  9 12:41:41 lb-stage-1 kernel: keepalived[26325]: segfault at 8 ip 00005654dffea118 sp 00007ffc80fe5940 error 4 in keepalived[5654dffc8000+7e000]
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[26326]: Unregistering DPVS gratuitous ARP.
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[26326]: Stopped - used 0.037816 user time, 0.099793 system time
Aug  9 12:41:42 lb-stage-1 Keepalived[26324]: Stopped Keepalived v2.0.19 (04/28,2021)
Aug  9 12:41:42 lb-stage-1 systemd: Stopped LVS and VRRP High Availability Monitor.
Aug  9 12:41:42 lb-stage-1 systemd: Starting LVS and VRRP High Availability Monitor...
Aug  9 12:41:42 lb-stage-1 Keepalived[12070]: Starting Keepalived v2.0.19 (04/28,2021)
Aug  9 12:41:42 lb-stage-1 Keepalived[12070]: Running on Linux 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 (built for Linux 3.10.0)
Aug  9 12:41:42 lb-stage-1 Keepalived[12070]: Command line: '/etc/keepalived/sbin/keepalived' '-D'
Aug  9 12:41:42 lb-stage-1 Keepalived[12070]: Opening file '/etc/keepalived/keepalived.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived[12070]: Opening file '/etc/keepalived/keepalived_progressive.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived[12070]: Opening file '/etc/keepalived/keepalived_stream.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived[12071]: Starting Healthcheck child process, pid=12072
Aug  9 12:41:42 lb-stage-1 Keepalived[12071]: Starting VRRP child process, pid=12073
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Opening file '/etc/keepalived/keepalived.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Opening file '/etc/keepalived/keepalived_progressive.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Registering Kernel netlink reflector
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Registering Kernel netlink command channel
Aug  9 12:41:42 lb-stage-1 systemd: Started LVS and VRRP High Availability Monitor.
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Opening file '/etc/keepalived/keepalived.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: find dpdk interface -- dpdk0
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Opening file '/etc/keepalived/keepalived_progressive.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Opening file '/etc/keepalived/keepalived_stream.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Opening file '/etc/keepalived/keepalived_stream.conf'.
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.170]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Assigned address 192.168.100.2 for interface dpdk0.kni
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Assigned address fe80::6eb3:11ff:fe07:6552 for interface dpdk0.kni
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: Registering DPVS gratuitous ARP.
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: VRRP sockpool: [ifindex(18), family(IPv4), proto(112), unicast(0), fd(10,11)]
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: (VI_1) Receive advertisement timeout
Aug  9 12:41:42 lb-stage-1 Keepalived_vrrp[12073]: (VI_1) Entering MASTER STATE
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.170]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.171]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.171]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.172]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.172]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.173]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.173]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.175]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.175]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.176]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.176]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.177]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.177]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.178]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.178]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.179]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 200 for VS [x.y.0.179]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Gained quorum 1+0=1 <= 400 for VS [x.y.0.210]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.4]:tcp:80 for VS [x.y.0.170]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.7]:tcp:80 for VS [x.y.0.170]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.4]:tcp:443 for VS [x.y.0.170]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.7]:tcp:443 for VS [x.y.0.170]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.10]:tcp:80 for VS [x.y.0.171]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.13]:tcp:80 for VS [x.y.0.171]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.10]:tcp:443 for VS [x.y.0.171]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.13]:tcp:443 for VS [x.y.0.171]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.5]:tcp:80 for VS [x.y.0.172]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.8]:tcp:80 for VS [x.y.0.172]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.5]:tcp:443 for VS [x.y.0.172]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.8]:tcp:443 for VS [x.y.0.172]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.11]:tcp:80 for VS [x.y.0.173]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.14]:tcp:80 for VS [x.y.0.173]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.11]:tcp:443 for VS [x.y.0.173]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.14]:tcp:443 for VS [x.y.0.173]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.1]:tcp:80 for VS [x.y.0.175]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.17]:tcp:80 for VS [x.y.0.175]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.1]:tcp:443 for VS [x.y.0.175]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.17]:tcp:443 for VS [x.y.0.175]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.1]:tcp:80 for VS [x.y.0.176]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.1]:tcp:443 for VS [x.y.0.176]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.2]:tcp:80 for VS [x.y.0.177]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.2]:tcp:443 for VS [x.y.0.177]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.31]:tcp:80 for VS [x.y.0.178]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.31]:tcp:443 for VS [x.y.0.178]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.18]:tcp:80 for VS [x.y.0.179]:tcp:80
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.5.18]:tcp:443 for VS [x.y.0.179]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.4]:tcp:443 for VS [x.y.0.210]:tcp:443
Aug  9 12:41:42 lb-stage-1 Keepalived_healthcheckers[12072]: Activating healthchecker for service [192.168.100.7]:tcp:443 for VS [x.y.0.210]:tcp:443

I am sharing my keepalived configurations and basic settings including dpip below.

keepalived.conf

vrrp_instance VI_1 {
    state MASTER                  # master
    interface dpdk0.kni           # should be kni interface
    dpdk_interface dpdk0          # should be DPDK interface
    virtual_router_id 150
}
include /etc/keepalived/keepalived_progressive.conf
include /etc/keepalived/keepalived_stream.conf

keepalived_progressive.conf

global_defs {
    notification_email_from root
    smtp_server localhost
    smtp_connect_timeout 30
    router_id x.y.0.153
}
virtual_server x.y.0.176 80 {
    persistence_timeout  70
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP
    virtualhost abc.com
    real_server 192.168.5.1 80 {
        weight 200
        HTTP_GET {
            url {
                path /
                digest a05f3456ead2f722ff2fb663aefa44e2
                status_code 200
            }
            connect_port 80
            connect_timeout 15
            retry 5
            delay_before_retry 5
        }
    }
}
virtual_server x.y.0.176 443 {
    persistence_timeout  70
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP
    virtualhost  abc.com
    real_server 192.168.5.1 443 {
        weight 200
        HTTP_GET {
            url {
                path /
                digest a05f3456ead2f722ff2fb663aefa44e2
                status_code 200
            }
            connect_port 80
            connect_timeout 15
            retry 5
            delay_before_retry 5
        }
    }
}
..
.

keepalived_stream.conf

global_defs {
    notification_email_from root
    smtp_server localhost
    smtp_connect_timeout 30
    router_id x.y.0.153
}
virtual_server x.y.0.170 80 {
    persistence_timeout  70
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP
    virtualhost abc.com
    real_server 192.168.5.1 80 {
        weight 200
        HTTP_GET {
            url {
                path /
                digest a05f3456ead2f722ff2fb663aefa44e2
                status_code 200
            }
            connect_port 80
            connect_timeout 15
            retry 5
            delay_before_retry 5
        }
    }
}
virtual_server x.y.0.170 443 {
    persistence_timeout  70
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP
    virtualhost abc.com
    real_server 192.168.5.1 443 {
        weight 200
        HTTP_GET {
            url {
                path /
                digest a05f3456ead2f722ff2fb663aefa44e2
                status_code 200
            }
            connect_port 80
            connect_timeout 15
            retry 5
            delay_before_retry 5
        }
    }
}
..
.

keepalived version

Keepalived v2.0.19 (04/28,2021)

Copyright(C) 2001-2021 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 3.10.0
Running on Linux 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021

configure options: 

Config options:  LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING

System options:  PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV6_ADVANCED_API RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA FRA_OIFNAME IFA_FLAGS IP_MULTICAST_ALL NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION VRRP_VMAC IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE SO_MARK SCHED_RT SCHED_RESET_ON_FORK

dpip addr show

inet x.y.0.176/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.173/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.177/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.170/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.178/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet 192.168.5.11/16 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.171/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.175/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.179/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet x.y.0.172/24 scope global dpdk0
     valid_lft forever preferred_lft forever
inet 192.168.100.2/16 scope global dpdk0
     valid_lft forever preferred_lft forever

dpip route show

inet 192.168.100.2/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet 192.168.5.11/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.170/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.171/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.172/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.173/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.175/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.176/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.177/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.178/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.179/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet x.y.0.0/24 via 0.0.0.0 src x.y.0.170 dev dpdk0 mtu 1500 tos 0 scope link metric 0 proto auto 
inet 192.168.0.0/16 via 0.0.0.0 src 192.168.100.2 dev dpdk0 mtu 1500 tos 0 scope link metric 0 proto auto

keepalived.service

[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target 
Wants=network-online.target 

[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/keepalived/etc/sysconfig/keepalived
ExecStart=/etc/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

I have also searched Keepalived_healthcheckers exited due to segmentation fault (SIGSEGV). message and I confirmed that that issue is solved by a commit. ( https://github.com/acassen/keepalived/issues/1323 )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cagatay23