HAProxy Enterprise Documentation 2.1r1

Active/Standby Clustering

In an active-standby cluster, one instance of HAProxy Enterprise receives traffic while a second instance is put on standby. In case the active node fails, the standby takes over. With HAProxy Enterprise, you configure an active-standby cluster by installing the VRRP module.

HAProxy Enterprise's VRRP module assigns a virtual, static IP address to your load balancer. If the load balancer fails, the IP address will instantly transfer to a backup instance of HAProxy Enterprise, avoiding any break in service. Your router can continue to send traffic to the same IP address for the load balancer, never knowing that there was a failover. The module utilizes a stable version of Keepalived, which implements the Virtual Router Redundancy Protocol (VRRP).

In this guide, we set up two load balancers: one active and the other on standby.

[Active/Standby with VRRP]

Installation

  1. Install the VRRP module using your system's package manager on both instances of HAProxy Enterprise that will participate in your load balancer cluster:

    $ # On Debian/Ubuntu
    $ sudo apt-get install hapee-extras-vrrp
    $ # On CentOS/RedHat/Oracle
    $ sudo yum install hapee-extras-vrrp
    $ # On SUSE
    $ sudo zypper install hapee-extras-vrrp
    $ # On FreeBSD
    $ sudo pkg install hapee-extras-vrrp
  2. On the instance that will serve as your active load balancer, edit the file /etc/hapee-extras/hapee-vrrp.cfg. It looks like this:

    # Check for the presence of the SSH daemon. That way, if SSH dies, we prefer
    # the other node which remains remotely manageable.
    
    vrrp_script chk_sshd {
            script "pkill -0 sshd"          # pkill -0 is cheaper than pidof
            interval 5                      # check every 5 seconds
            weight -4                       # remove 4 points of prio if missing
            fall 2                          # check twice before setting down
            rise 1                          # check once before setting up
    }
    
    # Check for the presence of the load balancer daemon (hapee-lb) itself. The
    # weight is higher than for SSHD, because if one node only has SSHD and the
    # other one only has the LB running, we prefer the LB.
    
    vrrp_script chk_lb {
            script "pkill -0 hapee-lb"      # pkill -0 is cheaper than pidof
            interval 1                      # check every second
            weight 6                        # add 6 points of prio if present
            fall 2                          # check twice before setting down
            rise 1                          # check once before setting up
    }
    
    # This is an example of how it would be possible to check if the LB sees some
    # operational servers, and to use the result to decide to be primary or backup.
    # The "/are-you-ok" url should be referenced as a "monitor-uri" in hapee-lb,
    # and this vrrp_script should be referenced in the "track_script" block of the
    # concerned VRRP instances.
    
    vrrp_script chk_servers {
            script "echo 'GET /are-you-ok' | nc 127.1 8080 | grep -q '200 OK'"
            interval 2                      # check every 2 seconds
            weight 2                        # add 2 points of prio if OK
            fall 2                          # check twice before setting down
            rise 2                          # check twice before setting up
    }
    
    vrrp_instance vrrp_1 {
            interface eth0                  # or bond0 or whatever prod interface
            state MASTER                    # try to be primary (MASTER) without waiting
            virtual_router_id 51            # use a distinct value for each instance
            priority 101                    # 101 on primary, 100 on backup
            virtual_ipaddress_excluded {
                    55.55.55.55             # your shared service IP address(es)
            }
            track_interface {
                    eth0 weight -2          # interfaces to monitor
                    # eth1 weight -2
            }
            track_script {
                    chk_sshd
                    chk_lb
            }
    }

    In the vrrp_instance section, change the interface value from eth0 to the name of a network interface that receives traffic on the server. This is the same interface that you bind HAProxy Enterprise's frontend to. For example, if traffic reaches the load balancer using interface enp0s8, then set this line to interface enp0s8. Also, reference the same name in the track_interface block.

    Next, change the IP address listed in the virtual_ipaddress_excluded block to the virtual IP address you'd like to assign to this interface. HAProxy Enterprise will bind to this address to receive traffic. If needed, add more IP addresses here, each on its own line. The new address(es) should fall within the interface's IP subnet, but should not be assigned to any server already.

    The following snippet demonstrates these changes:

    vrrp_instance vrrp_1 {
            interface enp0s8                # Change network interface name
            state MASTER
            virtual_router_id 51
            priority 101
            virtual_ipaddress_excluded {
                    192.168.50.10           # NEW IP address
            }
            track_interface {
                    enp0s8 weight -2        # Change network interface name
            }
            track_script {
                    chk_sshd
                    chk_lb
            }
    }

    Note

    On Ubuntu 18.04 and newer, there is an issue that causes the virtual IP to be lost from the network interface when the systemd-networkd service is restarted, such as when you call netplan apply if using Netplan (the default). To avoid releasing assigned virtual IP addresses when the service restarts, set the critical field to true in your /etc/netplan Netplan YAML file.

    network:
    version: 2
    ethernets:
        eth0:
            critical: true
            addresses:
            [---]

    If you have disabled Netplan, you can set the CriticalConnection field to true in your interface file in the /etc/systemd/network directory.

    [DHCP]
    CriticalConnection=true
  3. Start the hapee-extras-vrrp service:

    $ sudo systemctl start hapee-extras-vrrp
    $ sudo systemctl enable hapee-extras-vrrp
  4. Edit your HAProxy Enterprise configuration to listen on the virtual IP address. Note that you must append the field transparent, indicating that the address will be bound even if it does not belong to the local machine, which is necessary since the virtual IP will float to the active load balancer.

    frontend myfrontend
       mode http
       bind 192.168.50.10:80 transparent
       default_backend web_servers

    You can listen on all IP addresses assigned to the server by replacing the address with an asterisk or omitting the IP portion altogether:

    frontend myfrontend
       mode http
       bind *:80
       default_backend web_servers
  5. On your backup load balancer, the VRRP configuraton must be almost exactly the same. Edit the VRRP configuration file and the HAProxy Enterprise configuration in the same way, except:

    • change the priority field in the vrrp_instance block to have a lower number than the active instance; The load balancer with the highest priority is promoted to be the active instance. For example, if the priority on the active instance is 101, then set the backup instance's priority to 100.

    • change state to BACKUP.

    vrrp_instance vrrp_1 {
            interface enp0s8
            state BACKUP
            virtual_router_id 51
            priority 100                    # A lower priority value than the primary
            virtual_ipaddress_excluded {
                    192.168.50.10           # Same IP address as on primary
            }
            track_interface {
                    enp0s8 weight -2
            }
            track_script {
                    chk_sshd
                    chk_lb
            }
    }

Failover triggers

The following events can trigger a failover:

  • The active instance lowers its weight below one of the backup instances due to a failed health check.

  • A backup instance is reconfigured with a weight larger than the current active instance.

  • The active instance stops emitting its heartbeat packet to the cluster.

VRRP health check scripts

In the VRRP configuration, the vrrp_script blocks define health checks that can trigger a failover. By default, our configuration checks whether the SSH daemon is running and whether HAProxy Enterprise is running. You may add more checks.

Consider this block, which checks the status of the SSH daemon every five seconds. If the service is not available, then the VRRP instance's weight (i.e. it's priority) is reduced by 4 points:

vrrp_script chk_sshd {
  script "pkill -0 sshd"          # pkill -0 is cheaper than pidof
  interval 5                      # check every 5 seconds
  weight -4                       # remove 4 points of prio if missing
  fall 2                          # check twice before setting down
  rise 1                          # check once before setting up
}

The HAProxy Enterprise health check is similar, except that if the hapee-lb process is running, it adds 6 points. The reason is in case one load balancer has SSH working, but not hapee-lb, while the other has hapee-lb, but not SSH, the one with hapee-lb will become the active instance, even though it is not manageable remotely.

vrrp_script chk_lb {
        script "pkill -0 hapee-lb"      # pkill -0 is cheaper than pidof
        interval 1                      # check every second
        weight 6                        # add 6 points of prio if present
        fall 2                          # check twice before setting down
        rise 1                          # check once before setting up
}

The following fields define a vrrp_script block:

Name

Argument type

Description

script

string

Command to run; Running pkill -0 checks if any process with the given name is running. If it returns 0, the check passes; If it returns 1, the check fails.

interval

integer

Interval (in seconds) between two checks

weight

integer

Points to add or remove to instance weight

fall

integer

Number of consecutive negative checks before considered as down

rise

integer

Number of consecutive positive checks before considered as up

VRRP instance

The following fields define a vrrp_instance block:

Name

Argument type

Description

vrrp_instance

string

Describes a new VRRP instance.

interface

string

Interface managed by the VRRP module.

state

string

State at startup until VRRP priority negotiation completes. Can be either MASTER or BACKUP.

virtual_router_id

integer

VRRP instance identifier which must be common to all nodes of the same cluster. Its value ranges from 0 to 255.

priority

integer

Weight of local node in this VRRP instance. NOTE This will not work on cloud provider networks that disable IP multicast.

virtual_ipaddress

string

List of virtual IP addresses to add or remove when the state changes to active or backup. All VRRP nodes in a cluster must own the same IP. NOTE This will not work on cloud provider networks that disable IP multicast.

virtual_ipaddress_excluded

string

Same as virtual_ipaddress; but IPs are not announced in the VRRP heartbeat packet, which reduces bandwidth usage when sending packets.

track_interface

string

Tracks interface status and updates priority accordingly.

track_script

string

Runs the health check scripts and updates priority accordingly.


Next up

AWS