Setting up a High Availability Cluster with Keepalived and Failover IPv4

Introduction

High availability (HA) refers to the ability of a system to continue operating with a high probability despite the failure of one of its components. There are several applications that help us to achieve high availability. In this tutorial we will use Keepalived, an application for load balancing and high availability, which uses the Virtual Router Redundancy Protocol (VRRP; RFC5798).

Keepalived is used as a daemon that monitors the system state or certain conditions to automatically fail over to a standby system if an error occurs. We will additionally use a netcup Failover IPv4 and a Cloud vLAN Free.

Please note that this is not a tutorial for absolute beginners. I use references to other tutorials or the official documentation of the applications used. If instructions already exist, I see no reason to write them again. So the main point here is how to implement a high availability setup with the mentioned products in the netcup environment.

This tutorial can also be used as a starting point for other distributions, but some modifications will be required.

Requirements

• 2 servers: Either vServers or root servers with Debian installed
• Failover IPv4
• Cloud vLAN Free (or higher)
• Access to your Customer Control Panel (CCP) & Server Control Panel (SCP)
• Access to a user with root privileges.

I assume that you have already ordered the mentioned products and fulfill the stated requirements. I will simply refer to the two servers as server 1 (s1) and server 2 (s2). If I do not specify a particular server, it means that the commands must be executed on both servers.

Step 1 - Set up Cloud vLAN

Step 1.1 - Preparation

Step 1.1.1 - Log in to your SCP, either directly or via the autologin feature in your CCP.

Step 1.1.2 - Choose a server (from the list on the top left side).

Step 1.1.3 - Navigate to the Network section (in the left sidebar).

Step 1.1.4 - Choose add Ethernet Interface (on the top right side).

Step 1.1.5 - Select your Cloud vLAN and choose virtio as the driver.

Step 1.1.6 - Select the second server and repeat steps 1.1.3 - 1.1.5; then continue with 1.2.

Step 1.2 - Create interface

Log in to your servers and create a vLAN configuration for the network interface using your favorite text editor, e.g. sudo vim /etc/network/interfaces.d/vlan.cfg:

Example /etc/network/interfaces.d/vlan.cfg for s1

auto eth1
iface eth1 inet static
    address 10.132.0.10
    netmask 255.255.255.0

Example /etc/network/interfaces.d/vlan.cfg for s2

auto eth1
iface eth1 inet static
    address 10.132.0.20
    netmask 255.255.255.0

Step 1.3 - Restart networking

sudo systemctl restart networking

Step 1.4 - Verify vLAN setup

There are various methods to verify that your vLAN is set up correctly. For example, you could start with a simple check using ip addr show:

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e6:1c:ef:12:66:ae brd ff:ff:ff:ff:ff:ff
    inet 10.132.0.10/24 brd 10.132.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::e41c:efff:fe12:66ae/64 scope link
       valid_lft forever preferred_lft forever

Next to your other network interfaces, you can already see our newly configured interface eth1 on s1 with IP 10.132.0.10, which we previously set in the vlan.cfg.

Continue with a simple ping between s1 and s2 to ensure that you actually have a working connection:

operator@s1:~$ ping 10.132.0.20 -c 5

PING 10.132.0.20 (10.132.0.20) 56(84) bytes of data.
64 bytes from 10.132.0.20: icmp_seq=1 ttl=64 time=0.450 ms
64 bytes from 10.132.0.20: icmp_seq=2 ttl=64 time=0.510 ms
64 bytes from 10.132.0.20: icmp_seq=3 ttl=64 time=0.403 ms
64 bytes from 10.132.0.20: icmp_seq=4 ttl=64 time=0.379 ms
64 bytes from 10.132.0.20: icmp_seq=5 ttl=64 time=0.402 ms

--- 10.132.0.20 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 104ms
rtt min/avg/max/mdev = 0.379/0.428/0.510/0.053 ms

Then also check if the connection works in the reverse direction from s2 to s1. Now that we have confirmed our vLAN configuration, we can continue with the rest of our setup.

Step 2 - Setup Failover IPv4

Step 2.1 - Preparation

Step 2.1.1 - Log in to your SCP either directly or via the CCP.

Step 2.1.2 - Navigate to Options (on the top right side).

Step 2.1.3 - Navigate to Webservice (in the left sidebar).

Step 2.1.4 - Activate the Webservice and set a secure Webservice Password.

Step 2.1.5 - Assign Failover IP.

We determine that s1 should be our master server and therefore we want to assign the Failover IP to s1: Open the SCP, make sure to choose s1 and then navigate to Network in the left sidebar. You should see your Failover IPv4 in the IPv4 section. Assign it to your server by clicking on Route IP to Server.

Step 2.2 - Call netcup Webservice to change routing

We now create a script on both servers that will make a call to the netcup Webservice to change the routing of our Failover IP in case of an error situation. This script is needed in our Keepalived configuration, which you will see in a moment.

to_master.sh

##!/usr/bin/bash

CURL_REROUTE='
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://enduser.service.web.vcp.netcup.de/">
    <SOAP-ENV:Body>
        <ns1:changeIPRouting>
            <loginName>YOUR_LOGIN_NAME</loginName>
            <password>YOUR_WEBSERVICE_PASSWORD</password>
            <routedIP>YOUR_FAILOVER_IP</routedIP>
            <routedMask>32</routedMask>
            <destinationVserverName>YOUR_VSERVER_NAME</destinationVserverName>
            <destinationInterfaceMAC>YOUR_VSERVER_MAC</destinationInterfaceMAC>
        </ns1:changeIPRouting>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>'

CURL_CHECK='
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://enduser.service.web.vcp.netcup.de/">
    <SOAP-ENV:Body>
        <ns1:getVServerIPs>
            <loginName>YOUR_LOGIN_NAME</loginName>
            <password>YOUR_WEBSERVICE_PASSWORD</password>
            <vserverName>YOUR_VSERVER_NAME</vserverName>
        </ns1:getVServerIPs>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>'

REROUTE_RESULT=$(curl -s -H "Content-Type: text/xml; charset=utf-8" -H "SOAPAction:" -d "$CURL_REROUTE" -X POST https://www.servercontrolpanel.de:443/WSEndUser?xsd=1)
CHECK_RESULT=$(curl -s -H "Content-Type: text/xml; charset=utf-8" -H "SOAPAction:" -d "$CURL_CHECK" -X POST https://www.servercontrolpanel.de:443/WSEndUser?xsd=1)
FAILOVER_IP_ACTIVE=$(echo $CHECK_RESULT | grep YOUR_FAILOVER_IP | wc -l)

n=0
while [ $n -lt 10 ] && [ $FAILOVER_IP_ACTIVE -eq 0 ]
do
    REROUTE_RESULT=$(curl -s -H "Content-Type: text/xml; charset=utf-8" -H "SOAPAction:" -d "$CURL_REROUTE" -X POST https://www.servercontrolpanel.de:443/WSEndUser?xsd=1)
    sleep 3
    CHECK_RESULT=$(curl -s -H "Content-Type: text/xml; charset=utf-8" -H "SOAPAction:" -d "$CURL_CHECK" -X POST https://www.servercontrolpanel.de:443/WSEndUser?xsd=1)
    FAILOVER_IP_ACTIVE=$(echo $CHECK_RESULT | grep YOUR_FAILOVER_IP | wc -l)
    n=$((n+1))
done

The above script changes the routing of the Failover IPv4 to the specified server. If the rerouting does not work for any reason, the script tries again until the maximum tries (in this case 10) are reached. Make sure to use your actual data wherever needed and to make the script executable with sudo chmod +x to_master.sh.

Hints:

• YOUR_LOGIN_NAME is your customer id.
• YOUR_WEBSERVICE_PASSWORD is the one we just set in step 2.1.4.
• YOUR_FAILOVER_IP can be found in your CCP in the Products section or in the SCP in the Network section.
• YOUR_VSERVER_NAME can be found in your SCP in the General section of the corresponding server.
• YOUR_VSERVER_MAC can also be found in your SCP in the General section of the corresponding server (note that you need the MAC of your primary interface not the MAC of the vLAN).

Step 3 - Set up Keepalived

Step 3.1 - Install Keepalived

Navigate to either the website https://keepalived.org/download.html or the respective GitHub repository https://github.com/acassen/keepalived and install the application. The instructions are available at https://github.com/acassen/keepalived/blob/master/INSTALL.

Step 3.2 - Turn on packet forwarding

In order for the Keepalived service to forward packets, you need to turn on forwarding in the kernel. Edit /etc/sysctl.conf and make sure to include the following line using your favorite text editor, e.g. sudo vim /etc/sysctl.conf:

net.ipv4.ip_forward = 1

For the changes to take effect execute sudo sysctl -p.

Step 3.3 - Configure Keepalived

I will provide a basic configuration here that is a good starting point for further tweaks if needed. In this configuration, s1 starts as master and s2 starts as backup. If for some reason s1 stops sending adverts / s2 stops receiving adverts, s2 will automatically jump in as failover.

An interesting parameter to start working on is the advert_int parameter. On the one hand, this parameter defines how often an advert should be sent (in this case every 2 seconds) and, on the other hand, that if no advert has been received after 3 * advert_int seconds, the failover is triggered (so in this case; after 3 * 2 = 6 seconds).

If a server changes from BACKUP to MASTER state, the notify_master script is triggered, which is our to_master.sh (the one that reroutes our Failover IPv4).

However, there are many more methods besides adverts that you can use to trigger a failover. For example, you could also track files, scripts, processes, interfaces, etc. to increase/decrease priority and trigger the failover based on that. Scripts in particular are very flexible and offer the possibility for more sophisticated checks. If you are interested in more configuration methods, see this Red Hat Article for some configurations.

Example /etc/keepalived/keepalived.conf for s1

vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 101
    priority 255
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass CUSTOM_PASSWORD
    }
    virtual_ipaddress {
        10.132.0.100
    }
    notify_master /PATH/TO/YOUR/SCRIPT/to_master.sh
}

Example /etc/keepalived/keepalived.conf for s2

vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 101
    priority 254
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass CUSTOM_PASSWORD
    }
    virtual_ipaddress {
        10.132.0.100
    }
    notify_master /PATH/TO/YOUR/SCRIPT/to_master.sh
}

Note: The CUSTOM_PASSWORD should be a secure password set by you and must be the same for s1 and s2.

Step 3.4 - Restart Keepalived

sudo systemctl restart keepalived

Conclusion

You should now have a running configuration of Keepalived that automatically triggers a rerouting of your Failover IPv4 to a backup server if the master server is down. You can verify that your system is set up correctly by turning off s1 and observing whether s2 detects the failure situation and reroutes the Failover IPv4.

Some additional remarks:

• If you have any problems with Keepalived, you may want to review your firewall settings and make sure that VRRP (protocol 112) is allowed.
• You can check sudo tcpdump proto 112 -i eth1 to verify that the servers are receiving each other's adverts.
• There are many Keepalived configuration options to play around with. I highly recommend digging deeper if you need something that is not covered in this tutorial.
• There is always a sweet spot between short downtime in case of a failover and the prevention of false or frequent failovers due to overly strict constraints (flapping).
• It should be noted that the call to the netcup Webservice and the rerouting of the Failover IPv4 may take several seconds.

License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicence, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contributor's Certificate of Origin

By making a contribution to this project, I certify that:

1. The contribution was created in whole or in part by me and I have the right to submit it under the license indicated in the file; or

2. The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same license (unless I am permitted to submit under a different license), as indicated in the file; or

3. The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

4. I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the license(s) involved.