HAproxy

HAProxy + Keepalived Load Balancer Pair — Setup Guide

A step-by-step guide to build a highly available, active/passive load balancer pair on two Ubuntu servers using HAProxy (Layer 4 TCP passthrough) and Keepalived (VRRP virtual IP).

1. Overview

Two Linux servers run HAProxy. Keepalived manages a single floating Virtual IP (VIP) using VRRP. Only one node owns the VIP at any time (the MASTER). If it fails, the BACKUP node takes over the VIP within a few seconds. Clients only ever talk to the VIP.

Because TLS is passed through (not terminated), HAProxy runs in TCP mode: it forwards the raw encrypted TCP stream to the backend web servers, which hold the certificates. HAProxy never decrypts the traffic.

Environment

Role	Hostname	IP Address	Notes
Load Balancer 1	haproxy01	10.10.10.4	Keepalived MASTER
Load Balancer 2	haproxy02	10.10.10.5	Keepalived BACKUP
Virtual IP (VIP)	—	10.10.10.10	Client-facing, floats between nodes
Web Server 1	web01	10.10.11.1	Backend
Web Server 2	web02	10.10.11.2	Backend

Load-balanced ports: 443, 7443, 8443, 9443

Traffic flow

Client
  │  connects to VIP 10.10.10.10 : {443,7443,8443,9443}
  ▼
Keepalived VIP (owned by MASTER node)
  ▼
HAProxy01  ──or──  HAProxy02   (TCP passthrough, encrypted)
  │
  ├──► WEB01  10.10.11.1
  └──► WEB02  10.10.11.2

Prerequisites

Two Ubuntu servers (this guide targets a modern apt-based Ubuntu release).
Root or sudo access on both.
Static IPs configured: 10.10.10.4 and 10.10.10.5 in the DMZ segment 10.60.85.0/28.
Backend web servers reachable from both load balancers on the target ports.
Firewall rules in place (see Section 9).

Do every step on BOTH servers unless a step is explicitly marked MASTER only or BACKUP only.

2. Prepare both servers

Run on both haproxy01 and haproxy02.

2.1 Update the OS

1
sudo apt update && sudo apt -y upgrade

2.2 Set hostnames (optional but recommended)

On haproxy01:

1
sudo hostnamectl set-hostname haproxy01

On haproxy02:

1
sudo hostnamectl set-hostname haproxy02

Add both hosts and the backends to /etc/hosts on each server:

1
2
3
4
5
6
sudo tee -a /etc/hosts >/dev/null <<'EOF'
10.10.10.4   haproxy01
10.10.10.5   haproxy02
10.10.11.1   web01
10.10.11.2   web02
EOF

2.3 Set the timezone and enable time sync (recommended)

1
sudo timedatectl set-ntp true

3. Kernel networking settings

HAProxy on the BACKUP node must be able to bind to the VIP even though the VIP is not yet present on that node. Enable non-local binding and confirm IP forwarding is off (LB does not route).

Run on both servers:

1
2
3
4
5
6
sudo tee /etc/sysctl.d/60-haproxy.conf >/dev/null <<'EOF'
# Allow HAProxy/Keepalived to bind to the floating VIP not yet assigned locally
net.ipv4.ip_nonlocal_bind = 1
EOF

sudo sysctl --system

Verify:

1
2
sysctl net.ipv4.ip_nonlocal_bind
# expected: net.ipv4.ip_nonlocal_bind = 1

4. Install HAProxy and Keepalived

Run on both servers:

1
2
sudo apt update
sudo apt -y install haproxy keepalived

Check versions:

1
2
haproxy -v
keepalived -v

Do not enable them to auto-start yet — configure first, then start.

5. Configure HAProxy (TCP passthrough)

The HAProxy config is identical on both servers. Because we bind explicitly to the VIP, only the node currently holding the VIP will actually receive client traffic.

5.1 Back up the default config

1
sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig

5.2 Write the new config

Run on both servers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
sudo tee /etc/haproxy/haproxy.cfg >/dev/null <<'EOF'
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         /dev/log local0
    log         /dev/log local1 notice
    chroot      /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user        haproxy
    group       haproxy
    daemon
    maxconn     20000

#---------------------------------------------------------------------
# Default settings applied to all frontends/backends below
#---------------------------------------------------------------------
defaults
    log         global
    mode        tcp
    option      tcplog
    option      dontlognull
    timeout connect 5s
    timeout client  50s
    timeout server  50s
    retries     3
    default-server inter 3s fall 3 rise 2

#---------------------------------------------------------------------
# Statistics dashboard (HTTP) — bound to local IP only for admin use
#   Reachable at http://<node-ip>:8404/stats
#---------------------------------------------------------------------
frontend stats
    bind        *:8404
    mode        http
    stats       enable
    stats       uri /stats
    stats       refresh 10s
    stats       admin if TRUE
    # Uncomment to require a login for the stats page:
    # stats auth admin:ChangeMe123!

#---------------------------------------------------------------------
# Port 443
#---------------------------------------------------------------------
frontend ft_443
    bind        10.10.10.10:443
    default_backend bk_443

backend bk_443
    balance     roundrobin
    server      web01 10.10.11.1:443 check
    server      web02 10.10.11.2:443 check

#---------------------------------------------------------------------
# Port 7443
#---------------------------------------------------------------------
frontend ft_7443
    bind        10.10.10.10:7443
    default_backend bk_7443

backend bk_7443
    balance     roundrobin
    server      web01 10.10.11.1:7443 check
    server      web02 10.10.11.2:7443 check

#---------------------------------------------------------------------
# Port 8443
#---------------------------------------------------------------------
frontend ft_8443
    bind        10.10.10.10:8443
    default_backend bk_8443

backend bk_8443
    balance     roundrobin
    server      web01 10.10.11.1:8443 check
    server      web02 10.10.11.2:8443 check

#---------------------------------------------------------------------
# Port 9443
#---------------------------------------------------------------------
frontend ft_9443
    bind        10.10.10.10:9443
    default_backend bk_9443

backend bk_9443
    balance     roundrobin
    server      web01 10.10.11.1:9443 check
    server      web02 10.10.11.2:9443 check
EOF

5.3 Configuration notes

mode tcp — Layer 4. The TLS handshake and certificates stay end-to-end between the client and the web servers. HAProxy never sees plaintext.
bind 10.10.10.10:<port> — HAProxy listens on the VIP only. Combined with ip_nonlocal_bind = 1, both nodes can start even when they don’t currently hold the VIP.
check — a plain TCP connect health check (opens the port, closes it). A backend is marked down after fall 3 consecutive failures and back up after rise 2 successes, probed every inter 3s.
balance roundrobin — alternates connections between web01 and web02. Change to leastconn if connections are long-lived, or add stick tables if you need session affinity (not usually needed for TLS passthrough).

5.4 Validate the config

Run on both servers:

1
2
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
# expected: "Configuration file is valid"

Do not start HAProxy yet — configure Keepalived first.

6. Configure Keepalived (VRRP virtual IP)

Keepalived config differs between the two nodes (state, priority). We also add a health check so the VIP fails over if HAProxy dies on the MASTER.

6.1 Health-check script (both servers)

1
2
3
4
5
6
7
sudo tee /etc/keepalived/check_haproxy.sh >/dev/null <<'EOF'
#!/bin/bash
# Return 0 if HAProxy is running, non-zero otherwise.
/usr/bin/killall -0 haproxy 2>/dev/null
EOF

sudo chmod +x /etc/keepalived/check_haproxy.sh

killall -0 sends no signal; it only checks that at least one haproxy process exists. Ensure psmisc is installed (it usually is): sudo apt -y install psmisc.

6.2 Keepalived config — MASTER (haproxy01 only)

Run on haproxy01:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
sudo tee /etc/keepalived/keepalived.conf >/dev/null <<'EOF'
global_defs {
    router_id haproxy01
    enable_script_security
    script_user root
}

vrrp_script chk_haproxy {
    script   "/etc/keepalived/check_haproxy.sh"
    interval 2      # run every 2 seconds
    weight   -20    # drop priority by 20 if HAProxy is down
    fall     2      # 2 failures = down
    rise     2      # 2 successes = up
}

vrrp_instance VI_1 {
    state           MASTER
    interface       eth0            # <-- set to this node's NIC (see note below)
    virtual_router_id 51            # must be IDENTICAL on both nodes
    priority        150             # MASTER higher than BACKUP
    advert_int      1
    unicast_src_ip  10.10.10.4      # this node
    unicast_peer {
        10.10.10.5                  # the other node
    }
    authentication {
        auth_type PASS
        auth_pass HA!VRRP24        # must match on both nodes
    }
    virtual_ipaddress {
        10.10.10.10/24              # VIP with the segment prefix
    }
    track_script {
        chk_haproxy
    }
}
EOF

6.3 Keepalived config — BACKUP (haproxy02 only)

Run on haproxy02:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
sudo tee /etc/keepalived/keepalived.conf >/dev/null <<'EOF'
global_defs {
    router_id haproxy02
    enable_script_security
    script_user root
}

vrrp_script chk_haproxy {
    script   "/etc/keepalived/check_haproxy.sh"
    interval 2
    weight   -20
    fall     2
    rise     2
}

vrrp_instance VI_1 {
    state           BACKUP
    interface       eth0            # <-- set to this node's NIC (see note below)
    virtual_router_id 51            # must be IDENTICAL on both nodes
    priority        100             # BACKUP lower than MASTER
    advert_int      1
    unicast_src_ip  10.10.10.5      # this node
    unicast_peer {
        10.10.10.4                  # the other node
    }
    authentication {
        auth_type PASS
        auth_pass HA!VRRP24        # must match on both nodes
    }
    virtual_ipaddress {
        10.10.10.10/28
    }
    track_script {
        chk_haproxy
    }
}
EOF

6.4 Important settings to check

interface eth0 — replace with the real NIC name on each node. Find it with:
1
ip -br addr
It might be ens160, eth0, enp3s0, etc. Both nodes must point at the interface that carries the 10.10.10.0/24 subnet.
virtual_router_id 51 — must be the same on both nodes, and unique on the L2 segment (if another VRRP/HSRP group uses 51, pick a different number 1–255 on both).
auth_pass — must match on both nodes. Change HA!VRRP24 to your own value.
unicast_src_ip / unicast_peer — uses VRRP unicast, which avoids multicast issues on NSX overlay segments. If your network requires multicast VRRP instead, remove both lines.
/24 on the VIP matches the DMZ segment mask 10.10.10.0/24.

6.5 Validate the Keepalived config

1
2
sudo keepalived -t -f /etc/keepalived/keepalived.conf
# no output / exit 0 means the syntax is OK

7. Start the services

Start HAProxy first, then Keepalived, on both servers (MASTER first if you want a predictable initial owner).

1
2
sudo systemctl enable --now haproxy
sudo systemctl enable --now keepalived

Check status:

1
2
sudo systemctl status haproxy --no-pager
sudo systemctl status keepalived --no-pager

If you change a config later, reload without dropping connections:

1
2
sudo systemctl reload haproxy
sudo systemctl reload keepalived

8. Verify the setup

8.1 Confirm the VIP is on the MASTER

On haproxy01 (MASTER):

1
2
ip addr show | grep 10.10.10.10
# You should see 10.10.10.10 listed on the interface.

On haproxy02 (BACKUP):

1
2
ip addr show | grep 10.10.10.10
# You should see NOTHING — the BACKUP does not hold the VIP while MASTER is healthy.

8.2 Watch Keepalived state transitions

1
2
sudo journalctl -u keepalived -f
# MASTER logs "Entering MASTER STATE"; BACKUP logs "Entering BACKUP STATE".

8.3 Confirm HAProxy is listening on the VIP

On the MASTER:

1
2
sudo ss -tlnp | grep haproxy
# expect 10.10.10.10:443, :7443, :8443, :9443 and *:8404 (stats)

8.4 Test the backends through the VIP

From a client (or from a load balancer itself):

1
2
3
4
5
6
# TLS handshake test through the VIP for each port
for p in 443 7443 8443 9443; do
  echo "=== Port $p ==="
  echo | openssl s_client -connect 10.10.10.10:$p -servername your.hostname.example 2>/dev/null \
    | openssl x509 -noout -subject -dates 2>/dev/null
done

A returned certificate subject/expiry proves the passthrough path reaches a web server.

8.5 View the HAProxy stats dashboard

Open in a browser (from an allowed admin host):

http://10.10.10.4:8404/stats
http://10.10.10.5:8404/stats

Backends should show UP (green). If you enabled stats auth, log in with those credentials.

9. Test failover

Simulate a MASTER failure and confirm the BACKUP takes over.

9.1 Failover by stopping HAProxy (tests the track script)

On haproxy01 (MASTER):

1
sudo systemctl stop haproxy

Within a few seconds the health check drops haproxy01’s priority below haproxy02’s, so the VIP moves. Confirm on haproxy02:

1
2
ip addr show | grep 10.10.10.10   # VIP now appears here
sudo journalctl -u keepalived -n 20 --no-pager   # "Entering MASTER STATE"

Restore:

1
2
# on haproxy01
sudo systemctl start haproxy

Because haproxy01 has the higher base priority, it reclaims the VIP (preemption is on by default). To avoid the VIP flapping back, add nopreempt to the BACKUP’s vrrp_instance and set haproxy01’s state to BACKUP as well (both nodes BACKUP + priority decides owner).

9.2 Failover by rebooting

1
2
# on the current MASTER
sudo reboot

Confirm the VIP moves to the other node and clients keep connecting to 10.10.10.10.

10. Firewall requirements

Ensure these are permitted. VRRP is IP protocol 112.

#	Source	Destination	Port / Proto	Purpose
1	Clients / DMZ proxy / VPN / DNATs	10.10.10.10 (VIP)	TCP 443, 7443, 8443, 9443	Client → VIP
2	10.10.10.4, 10.10.10.5	10.10.11.1, 10.10.11.2	TCP 443, 7443, 8443, 9443	HAProxy → web servers
3	10.10.10.4, 10.10.10.5	10.10.10.4, 10.10.10.5	IP proto 112 (VRRP)	Keepalived peer sync
4	Admin hosts	10.10.10.4, 10.10.10.5	TCP 8404	HAProxy stats page (optional)
5	Admin hosts	10.10.10.4, 10.10.10.5	TCP 22	SSH management

With unicast VRRP (as configured above) the peers exchange VRRP packets directly between 10.10.10.4 and 10.10.10.5, so rule #3 is unicast rather than multicast 224.0.0.18.

11. Troubleshooting

Symptom	Likely cause	Fix
Both nodes hold the VIP (split-brain)	VRRP packets blocked between peers	Allow IP proto 112 (rule #3); verify `unicast_peer` IPs; check `virtual_router_id` matches
VIP never appears	Wrong `interface` name	Set the real NIC from `ip -br addr`
HAProxy won’t start on BACKUP: cannot bind socket	Non-local bind not set	Confirm `net.ipv4.ip_nonlocal_bind = 1` (Section 3)
Backends show DOWN in stats	Web servers not listening / firewall	Test `nc -vz 10.10.11.1 443` from the LB; check rule #2
VIP flaps back and forth	Preemption on	Add `nopreempt` and set both nodes to `state BACKUP`
Keepalived ignores the check script	Script security	Ensure `enable_script_security` + script is `chmod +x` and root-owned
Clients get cert errors	Passthrough reaching wrong backend / SNI mismatch	Verify with `openssl s_client` (Section 8.4); confirm certs live on web01/web02

Useful commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Live HAProxy logs
sudo journalctl -u haproxy -f

# Live Keepalived logs / state changes
sudo journalctl -u keepalived -f

# Which node owns the VIP right now
ip -br addr | grep 10.10.10.10

# Runtime backend status via the admin socket
echo "show servers state" | sudo socat stdio /run/haproxy/admin.sock

(Install socat if needed: sudo apt -y install socat.)

12. Changing the VIP

To move the load balancer to a different Virtual IP, the VIP must be updated in two config files on both nodes (plus the cert and external dependencies if applicable). Changing it in only one place will break the service.

12.1 Keepalived — the floating VIP itself

File: /etc/keepalived/keepalived.conf on both haproxy01 and haproxy02.

    virtual_ipaddress {
        10.10.10.10/28      # <-- change to the new VIP (keep the correct prefix)
    }

If the new VIP is in a different subnet, also review:

The /28 prefix — set it to the new subnet’s mask.
interface — must be the NIC carrying the new subnet.
unicast_src_ip / unicast_peer — only if the node IPs themselves change.

12.2 HAProxy — the `bind` lines

File: /etc/haproxy/haproxy.cfg on both nodes. Every frontend binds to the VIP — there are four, one per port:

    bind        10.10.10.10:443      # ft_443
    bind        10.10.10.10:7443     # ft_7443
    bind        10.10.10.10:8443     # ft_8443
    bind        10.10.10.10:9443     # ft_9443

Update all four at once (adjust the target IP):

1
sudo sed -i 's/10\.10\.10\.10:/10.10.10.NEW:/g' /etc/haproxy/haproxy.cfg

12.3 Certificate (only if TLS terminated / SAN lists the VIP)

This passthrough build has no cert on HAProxy, so no change is needed here. If you later switch to TLS termination and the certificate’s subjectAltName includes the VIP by IP, reissue it with the new VIP so IP-based clients don’t get a name-mismatch warning.

12.4 Apply — order matters

On both nodes, validate then reload Keepalived first (brings up the new VIP) and HAProxy second:

1
2
3
sudo haproxy -c -f /etc/haproxy/haproxy.cfg     # validate
sudo systemctl reload keepalived                # new VIP comes up on MASTER
sudo systemctl reload haproxy                   # rebinds to the new VIP

Verify:

1
2
ip addr show | grep <new-VIP>          # present on MASTER only
sudo ss -tlnp | grep haproxy           # listening on new VIP:443/7443/8443/9443

12.5 Don’t forget the external dependencies

Firewall / NSX DFW
DNS

13. Quick reference

Item	Value
VIP	10.10.10.10
MASTER	haproxy01 — 10.10.10.4 (priority 150)
BACKUP	haproxy02 — 10.10.10.5 (priority 100)
Backends	web01 10.10.11.1, web02 10.10.11.2
Ports	443, 7443, 8443, 9443
Mode	TCP passthrough (Layer 4, no TLS termination)
Health check	TCP connect (`check`)
VRRP id	51 (unicast)
Stats page	http://<node>:8404/stats

Change log

v1.0 — Initial guide: HAProxy TCP passthrough + Keepalived unicast VRRP, TCP health checks.

rsync Guide Network Shares

HAproxy

HAProxy + Keepalived Load Balancer Pair — Setup Guide

1. Overview

Environment

Traffic flow

Prerequisites

2. Prepare both servers

2.1 Update the OS

2.2 Set hostnames (optional but recommended)

2.3 Set the timezone and enable time sync (recommended)

3. Kernel networking settings

4. Install HAProxy and Keepalived

5. Configure HAProxy (TCP passthrough)

5.1 Back up the default config

5.2 Write the new config

5.3 Configuration notes

5.4 Validate the config

6. Configure Keepalived (VRRP virtual IP)

6.1 Health-check script (both servers)

6.2 Keepalived config — MASTER (haproxy01 only)

6.3 Keepalived config — BACKUP (haproxy02 only)

6.4 Important settings to check

6.5 Validate the Keepalived config

7. Start the services

8. Verify the setup

8.1 Confirm the VIP is on the MASTER

8.2 Watch Keepalived state transitions

8.3 Confirm HAProxy is listening on the VIP

8.4 Test the backends through the VIP

8.5 View the HAProxy stats dashboard

9. Test failover

9.1 Failover by stopping HAProxy (tests the track script)

9.2 Failover by rebooting

10. Firewall requirements

11. Troubleshooting

Useful commands

12. Changing the VIP

12.1 Keepalived — the floating VIP itself

12.2 HAProxy — the bind lines

12.3 Certificate (only if TLS terminated / SAN lists the VIP)

12.4 Apply — order matters

12.5 Don’t forget the external dependencies

13. Quick reference

Change log

12.2 HAProxy — the `bind` lines