Prepare EC2 Instances

First, you have to make sure you have 2 or more up and running AWS EC2 instances.
Let’s call our main EC2 instance Boyega and our secondary EC2 instance Bumaye. Also, make sure to also create a role for both with permissions to perform changes to EC2 instances.

Install HAProxy

You need to install HAProxy on both servers. You can do so by using the link below to compile and then Install HAProxy if you are using Amazon Linux 2

How to Install HAProxy in Amazon Linux 2

Once done installing then change the following configuration file in /etc/haproxy/haproxy.cfg to

global
    log         127.0.0.1 local0 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

    errorfile 503 /etc/haproxy/errorfiles/503.http

frontend http-in
    mode http
    bind *:80 
    default_backend varnish

backend varnish
    mode http
    balance roundrobin
    option httpchk HEAD / HTTP/1.1
    server varnish 127.0.0.1:6081 check

The actual varnish backend does not exist. In your own environment, you would make sure of course that you have a proper backend configured. What is important for now is that we configured a 503 error file as an easy check to see if our failover works.

Make sure the file /etc/haproxy/errorfiles/503.http exists with the following content.

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
  <head>
    <title>503 -  Service Unavailable</title>
  </head>
  <body>
    <div>
          <h2>This is {servername}.</h2>
    </div>
  </body>
</html>

In order to see that the failover works, make sure to change the text of {servername} to the name of the server. This way we can show on which server we are currently running.

Let’s configure HAProxy to start on boot and to make sure it’s up and running.

systemctl enable haproxy
systemctl start haproxy
systemctl status haproxy -l

If you access both HAProxy instances through your browser, you will see the error pages you configured.

Elastic Network Interface

Our next step is to create an ENI with a static ipv4 in AWS

Install keepalived

Next up, we have to install Keepalived on both servers.
Keepalived will keep track of which server is currently the master server and when a failover should occur.

yum install keepalived

Boyega is our main server. Therefore we tell keepalived that this is the master.

Make sure that the config file /etc/keepalived/keepalived.conf looks like this.

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 2
}

vrrp_instance VI_1
{
    interface eth0
    state MASTER
    virtual_router_id 1
    priority 110
    unicast_src_ip 10.10.10.10

    unicast_peer
    {
        10.10.10.11
    }

    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Make sure to set the unicast_src_ip to the private IP of the current server. Set the unicat_peer to the private IP of the Bumaye server.

Bumaye is our secondary server. Therefore we tell Keepalived that this is the backup server. Make sure that the config file /etc/keepalived/keepalived.conf looks like this.

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1
{
    debug 2
    interface eth0
    state BACKUP
    virtual_router_id 1
    priority 100
    unicast_src_ip 10.0.1.42

    unicast_peer
    {
        10.0.1.131
    }

    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Make sure to set the unicast_src_ip to the private IP of the current server. Set the unicat_peer to the private IP of the Boyega server or your secondary/backup server.

Next, create the /etc/keepalived/failover.sh on both servers. Make sure that the ENI_IP_CREATED has the ipv4 IP of the AWS ENI you created.

Please note, the subnet you created the AWS ENI in must be the same subnet as your AWS Instances.

#!/bin/bash -xe
MAC_ADDR=$(ip addr show dev eth0 | sed -n 's/.*ether \([a-f0-9:]*\).*/\1/p') && 
IP=($(curl "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDR/local-ipv4s" 2>/dev/null)) && 
MAIN=${ENI_IP_CREATED} &&
export AWS_DEFAULT_REGION=eu-west-1 &&
echo "Moving IP address: $MAIN to system with primary IP address $IP" &&
INTERFACE=`aws ec2 describe-network-interfaces --filters Name=private-ip-address,Values=$MAIN --output json | jq .NetworkInterfaces[].NetworkInterfaceId -r` &&
TONODE=`curl --silent http://169.254.169.254/latest/meta-data/instance-id` &&
echo "Moving eni: $INTERFACE to instance id: $TONODE " &&
DETACH=`aws ec2 describe-network-interfaces --filters Name=private-ip-address,Values=$MAIN --output json | jq .NetworkInterfaces[].Attachment.AttachmentId -r` &&
INTERFACESTATUS=`aws ec2 describe-network-interfaces --filters Name=private-ip-address,Values=$MAIN --output json | jq .NetworkInterfaces[].Attachment.Status -r` &&
while [ "$INTERFACESTATUS" = 'attached' ] || [ "$INTERFACESTATUS" = 'in-use' ]
do
    echo "Will sleep 1 second"
    sleep 1
    INTERFACESTATUS=`aws ec2 describe-network-interfaces --filters Name=private-ip-address,Values=$MAIN --output json | jq .NetworkInterfaces[].Attachment.Status -r`
    echo "$DETACH is to be detached. Current interface status is $INTERFACESTATUS"
    aws ec2 detach-network-interface --attachment-id $DETACH --force
    echo "Command to detach Interface $INTERFACE submitted"
done &&
echo "Will attach interface $INTERFACE to $TONODE " &&
sleep 5s &&
aws ec2 attach-network-interface --instance-id $TONODE --network-interface-id $INTERFACE --device-index 1

Make sure the file is executable.

chmod 700 /etc/keepalived/failover.sh

Let’s configure Keepalived to start on boot and to make sure it’s up and running.

systemctl enable keepalived
systemctl start keepalived
systemctl status keepalived

Test your setup

If you do a tail -f /var/log/messages you will see the following message appear at Boyega.

Keepalived_vrrp[1196]: VRRP_Instance(VI_1) Entering MASTER STATE

Whereas Bumaye has the following message.

Keepalived_vrrp[784]: VRRP_Instance(VI_1) Entering BACKUP STATE

If this is the case, you should be good to go. If you now call the AWS ENI IP in your browser, you would see the welcomes message from Boyega.

Now stop HAProxy on Boyega.

systemctl stop haproxy

Reload your browser. The message of Bumaye should appear.

You have now created a failover for HAProxy.

How to setup Automatic Failover for HAProxy using Keepalived

Prepare EC2 Instances

Install HAProxy

Elastic Network Interface

Install keepalived

Test your setup

Comments

Leave a Reply Cancel reply

Keep up, get in touch.

Contact

Follow