Use Keepalived Health Checks with BGP-based Failover

Keepalived is one of the most commonly used applications that implements VRRP, a networking protocol that manages IP address assignment and ARP-based failover. It can be configured with additional health checks, such as checking the status of a service or running a custom script. When one of these health checks detects an issue, the instance changes to a fault state and failover is triggered. During these state transitions, additional task can be performed through custom scripts.

The Linode platform is currently undergoing network infrastructure upgrades, which affects IP address assignment and failover. Once this upgrade occurs for the data center and hardware that your Compute Instances reside on, VRRP software like Keepalived can no longer directly manage failover. However, other features of Keepalived can still be used. For instance, Keepalived can continue to run health checks or VRRP scripts. It can then be configured to interact with whichever BGP daemon your system is using to manage IP address assignment and failover.

This guide covers how to configure Keepalived with a simple health check and enable it to control lelastic, a BGP daemon created for the Linode platform.

Note
If you are migrating to BGP-based failover and currently have health checks configured with Keepalived, you can modify the steps in this guide to include your own settings.

Before continuing, IP Sharing and BGP failover must be properly configured on both Compute Instances. To do this, follow the Configuring Failover on a Compute Instance guide, which walks you through the process of configuring failover with lelastic. If you decide to use a tool other than lelastic, you will need to make modifications to some of the commands or code examples provided in some of the following sections.

Install and Configure Keepalived

This section covers installing the keepalived software from your distribution’s repository. See Installing Keepalived on the official documentation if you prefer to install it from source.

Log in to your Compute Instance over SSH. See Connecting to a Remote Server Over SSH for assistance.
Install keepalived by following the instructions for your system’s distribution.
Ubuntu and Debian:
sudo apt update && sudo apt upgrade sudo apt install keepalived
CentOS 8 Stream, CentOS/RHL 8 (including derivatives such as AlmaLinux 8 and Rocky Linux 8), Fedora:
sudo dnf upgrade sudo dnf install keepalived
CentOS 7:
sudo yum update sudo yum install keepalived
Create and edit a new keepalived configuration file.
sudo nano /etc/keepalived/keepalived.conf

Enter the following settings for your configuration into this file. Use the example below as a starting point, replacing each item below with the appropriate values for your Compute Instance. For more configuration options, see Configuration Options.

$password: A secure password to use for this keepalived configuration instance. The same password must be used for each Compute Instance you configure.
$ip-a: The IP address of this Compute Instance.
$ip-b: The IP address of the other Compute Instance.
$ip-shared: The Shared IP address.

File: /etc/keepalived/keepalived.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
vrrp_instance example_instance {
    state BACKUP
    nopreempt
    interface eth0
    virtual_router_id 10
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass $password
    }
    unicast_src_ip $ip-a
    unicast_peer {
        $ip-b
    }
    virtual_ipaddress {
        $ip-shared/32
    }
}

In the above configuration file, the state is set to BACKUP and the parameter nopreempt is included. When each Compute Instance uses these settings, failover is sticky. This means the Shared IP address remains routed to a Compute Instance until it enters a FAULT state, even if it is lower priority than the other Compute Instance. If you wish to prioritize one instance over the other, remove the nopreempt parameter, set one of the Compute Instances to a MASTER state, and adjust the PRIORITY parameter as desired.

Enable and start the keepalived service.

sudo systemctl enable keepalived
sudo systemctl start keepalived

Perform these steps again on the other Compute Instance you would like to configure.

Create the Notify Script

Keepalived can be configured to run notification scripts when the instance changes state (such as when entering a MASTER, BACKUP ,or FAULT state). These scripts can perform any action and are commonly used to interact with a service or modify network configuration files. For this guide, the scripts are used to update a log file and start or stop the BGP daemon that controls BGP failover on your Compute Instance.

Create and edit the notify script.
sudo nano /etc/keepalived/notify.sh

Copy and paste the following bash script into the newly created file. If you wish to control a BGP daemon other than lelastic, replace sudo systemctl restart lelastic and sudo systemctl stop lelastic with the appropriate commands for your service.

File: /etc/keepalived/notify.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/bin/bash

keepalived_log='/tmp/keepalived.state'
function check_state {
        local state=$1
        cat << EOF >> $keepalived_log
===================================
Date:  $(date +'%d-%b-%Y %H:%M:%S')
[INFO] Now $state

EOF
        if [[ "$state" == "Master" ]]; then
                sudo systemctl restart lelastic
        else
                sudo systemctl stop lelastic
        fi
}

function main {
        local state=$1
        case $state in
        Master)
                check_state Master;;
        Backup)
                check_state Backup;;
        Fault)
                check_state Fault;;
        *)
                echo "[ERR] Provided arguement is invalid"
        esac
}
main $1

Make the file executable.
sudo chmod +x /etc/keepalived/notify.sh

Modify the keepalived configuration files so that the notify script is used for each state change.

File: /etc/keepalived/keepalived.conf

1
2
3
4
5
6
vrrp_instance example_instance {
    ...
    notify_master "/etc/keepalived/notify.sh Master"
    notify_backup "/etc/keepalived/notify.sh Backup"
    notify_fault "/etc/keepalived/notify.sh Fault"
}

Restart your BGP daemon and keepalived.

sudo systemctl restart lelastic
sudo systemctl restart keepalived

View the log file to see if it was properly created and updated. If the notification script was successfully used, this log file should have an accurate timestamp and the current state of the instance.
cat /tmp/keepalived.state
```
===================================
Date:  14-Oct-2022 14:30:54
[INFO] Now Master
```

Configure the Health Check (VRRP Script)

The next step is to configure Keepalived with a health check so that it can failover if it ever detects an issue. This is the primary reason you may want to use Keepalived alongside a BGP daemon. Keepalived can be configured to track a file (track_file), track a process (track_process), or run a custom script so that you can preform more complex health checks. When using a script, like is shown in this example, the script should return a 0 to indicate success and return any other value to indicate a failure. When a failure is detected, the state is changed to FAULT and the notify script runs.

This guide helps you configure a custom script that detects if a file is present or not. If the file is present, the script returns a 1 to indicate a failure.

Create and edit the health check script.
sudo nano /etc/keepalived/check.sh

Copy the following script and paste it into the file.

File: /etc/keepalived/check.sh

1
2
3
4
5
6
7
8
#!/bin/bash

trigger='/etc/keepalived/trigger.file'
if [ -f $trigger ]; then
  exit 1
else
  exit 0
fi

Make the file executable.

sudo chmod +x /etc/keepalived/failover.sh

Update the keepalived configuration file to define the VRRP script and enable your VRRP instance to use the script. The interval determines how often the script is run, fall determines how many times the script must return a failure before the state is changed to FAULT, and rise determines how many times a success is returned before the instance goes back to a BACKUP or MASTER state.

File: /etc/keepalived/keepalived.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
vrrp_script check_for_file {
    script "/etc/keepalived/check.sh"
    interval 5
    fall 2
    rise 2
}
vrrp_instance example_instance {
    ...
    track_script {
        check_for_file
    }
    ...
}

Restart your BGP daemon and keepalived.

sudo systemctl restart lelastic
sudo systemctl restart keepalived

To test this health check, create the trigger file on whichever Compute Instance is in a MASTER state.
touch /etc/keepalived/trigger.file
Check the log file on that Compute Instance to make sure it enters a FAULT state. Once it does, check the log file on the other Compute Instance to verify that it enters a MASTER state.
tail -F /tmp/keepalived.state
```
===================================
Date:  14-Oct-2022 14:30:54
[INFO] Now Master
```

Additional Recommended Security Settings

By default, Keepalived attempts to run the scripts using a keepalived_script user. If that doesn’t exist, it uses the root user. Since running these scripts as the root user introduces many security concerns, this section discusses creating the keepalived_script user.

Create a limited user account called keepalived_script. Since it is never used to log in, that feature can be disabled.
sudo useradd -r -s /sbin/nologin -M keepalived_script
Edit the sudoers file.
visudo /etc/sudoers

Within this file, grant permission for the new user to restart and stop the BGP daemon. The example below uses lelastic.

File: /etc/sudoers

1
2
3
# User privilege specification
root    ALL=(ALL:ALL) ALL
keepalived_script ALL=(ALL:ALL) NOPASSWD: /usr/bin/systemctl restart lelastic, /usr/bin/systemctl stop lelastic

Update the ownership of the /etc/keepalived directory (and all of the files within it).
sudo chown -R keepalived_script:keepalived_script /etc/keepalived
Once again, edit the Keepalived configuration file and paste the following snippet to the top of that file.
File: /etc/keepalived/keepalived.conf
1 2 3 4
global_defs { enable_script_security } ...

Example Configuration Files

The links below contain complete working configuration files along with the specified example IP addresses. Please review them if you would like to see all of the recommended settings for each Compute Instance combined into a single file.

Shared IP: 203.0.113.57 (configured on the loopback interface)
Compute Instance A: 192.0.2.173 ( keepalived.conf)
Compute Instance B: 198.51.100.49 ( keepalived.conf)

Why Linode

Get to Know Us Better

Compute

Storage

Databases

Services

Networking

Developer Tools

Delivery

Security

Solutions

Pricing

Community

Engage With Us

Search Results

No Results

Filters

Use Keepalived Health Checks with BGP-based Failover

Install and Configure Keepalived

Create the Notify Script

Configure the Health Check (VRRP Script)

Additional Recommended Security Settings

Example Configuration Files

Your Feedback Is Important

Search Results

No Results

Filters

Use Keepalived Health Checks with BGP-based Failover

Configure IP Sharing and BGP Failover

Install and Configure Keepalived

Create the Notify Script

Configure the Health Check (VRRP Script)

Additional Recommended Security Settings

Example Configuration Files

Your Feedback Is Important