Recently I completed a project that required a pair of TCP load balancers in a failover configuration balancing connections to 6 http/https sites spanning 3 web servers.  This cluster handles a couple terabytes of traffic per month and each machine serves as the marketing mail server for one or more domains.  Total hardware list:

• 2 load balancers with connection to both switches
• 2 gigEth switches – internal and external
• 3 web servers plugged into the internal switch

In this case the data is NFS mounted from a storage machine, but for the purposes of this I’m going to assume that you know how to do those things or that each site is copied somehow to each other web server.  This diagram should assist in illustrating this setup:

Load Balancer Diagram Overview

Let us use these as the sites:

www.mysite.com resolves to 10.0.0.1

www.myothersite.com resolves to 10.0.0.3

www.mythirdsite.net resolves to 10.0.0.5

we’ll call the balancers balancer1 and balancer2 (this is very important, actually) and they will resolve to 10.0.0.254 and 10.0.0.253.

the webservers we’ll call hostA, hostB, and hostC

hostA has an external ip 10.0.0.101 and internal ips 192.168.0.1, 192.168.0.2, and 192.168.0.3

hostB has an external ip 10.0.0.102 and internal ips 192.168.0.4, 192.168.0.5, and 192.168.0.6

hostC has an external ip 10.0.0.103 and internal ips 192.168.0.7, 192.168.0.8, and 192.168.0.9

assume, please, that the first internal ip for each server represents it’s own apache virtual host for the first site and so forth.

You’re going to need to install LVS and heartbeat as well as enable ipv4 packet forwarding.  Each distro is going to do this a little bit differently, so I am skimping on the details in that respect, but if you can manage those packages you should be doing alright.

All your config files should be in /etc/ha.d, the heartbeat directory, but you may also have an /etc/ldirectord directory as well.  Moving on.  The files we care about are ha.d/haresources, ha.d/authkeys, ha.d/ha.cf and the ldirectord.cf file.  How you choose to firewall this thing is your business, but I’ll provide an idea of how I handle things at the end.

in authkeys:

auth 1
1 crc

in ha.cf:

debugfile /var/log/ha-debug #where to find debug info
logfile /var/log/ha-log #where to find logged indo
logfacility     local0
keepalive 2  #time between checks
deadtime 10  #time before it decides a host is down
bcast   eth0 eth1       # Linux
auto_failback on  #do you need help figuring this one out?
node    balancer1   # result of uname -n
node    balancer2   # if these don’t match above, then won’t start

and haresources:

balancer1 IPaddr::10.0.0.1/24/eth0 IPaddr::10.0.0.2/24/eth0 IPaddr::10.0.0.3/24/eth0 IPaddr::10.0.0.21/24/eth0 IPaddr::10.0.0.22/24/eth0 IPaddr::10.0.0.23/24/eth0 ldirectord

balancer1 192.168.0.254/32/eth1

While the other files don’t require much of an explanation, the haresources config certainly does. First, you’ll notice that the first line references “balancer1″ and that there is no mention of balancer2.  This is because the master balancer is balancer1.  The name “balancer1″ MUST be the result of uname -n on the local server, too, or the director and heartbeat may fail to start.  Note also the way the config is structured.  Each of the websites’ IPs must be listed here because they will be swapped over during a failover to balancer2.  Note that these use CIDR notation and specify an interface.  Also note that .21-.23 are here as well.  This is so we can SNAT outbound mail from these servers to appear to come from these internal servers as well as be able to connect back to these servers directly on port 25.  Note that the second line specifies the internal webservers’ gateway IP address.  The active load balancer must be the gateway of these NAT’d machines.

in ldirectord.cf:

# Virtual Server for www.mysite.com
logfile=”local0″
quiescent = no
virtual = 10.0.0.1:0
real = 192.168.0.1 masq
real = 192.168.0.2 masq
real = 192.168.0.3 masq
service = http
protocol = tcp
checktype = negotiate
negotiatetimeout = 20
persistent=3600
scheduler = wlc

# Virtual Server for www.myothersite.com
logfile=”local0″
quiescent = no
virtual = 10.0.0.3:0
real = 192.168.0.4 masq
real = 192.168.0.5 masq
real = 192.168.0.6 masq
service = http
protocol = tcp
checktype = negotiate
negotiatetimeout = 20
persistent=3600
scheduler = wlc

# Virtual Server for www.mythirdsite.com
logfile=”local0″
quiescent = no
virtual = 10.0.0.5:0
real = 192.168.0.7 masq
real = 192.168.0.8 masq
real = 192.168.0.9 masq
service = http
protocol = tcp
checktype = negotiate
negotiatetimeout = 20
persistent=3600
scheduler = wlc

Now, because we’re dealing with private networks, we are going to need to crack into iptables rules to ensure that things are going to be allowed where they need to go.  We’re using NAT, so expect to see NAT rules, and I’ve included rules used to ensure that only required services get through.  Look for comments within the firewall rules to explain things in more detail:

#!/sbin/runscript
#####!/bin/sh
#
# Set up some definitions for Firewalling
#
opts=”start stop open restart”

depend() {
after net
need logger
}

# Loopback
LOIF=’lo’
LOIP=’127.0.0.1′

# EXTIF is the interface facing the Internet
# INTIF is the internal LAN that you want to masquarade
EXTIF=’eth0′
INTIF=’eth1′

# Interface on which the VIPs are assigned
VIPIF=’eth0:0′

# Virtual IP Section – the IPs of our websites
VIPIP=’10.0.0.1′
VIPIP2=’10.0.0.2′
VIPIP3=’10.0.0.3′

# External IP Section
EXTIP=`ifconfig $EXTIF | awk ‘/inet addr/ { gsub(“.*:”, “”, $2) ; print $2 }’`

# Internal IP Section with Broadcast and Netmask info
INTIP=`ifconfig $INTIF | awk ‘/inet addr/ { gsub(“.*:”, “”, $2) ; print $2 }’`
INTBCAST=`ifconfig $INTIF | awk ‘/Bcast/ { gsub(“.*:”, “”, $3) ; print $3 }’`
INTMASK=`ifconfig $INTIF | awk ‘/Mask/ { gsub(“.*:”, “”, $4) ; print $4 }’`

# Associate Private IPs with hostnames so we can SNAT outbound mail to appear to come from public IPs
HOSTA=’10.0.0.21′
HOSTB=’10.0.0.22′
HOSTC=’10.0.0.23′

# Put in a thing to dynamically find the DNS entries
start(){

# Flush out the tables, and deny everything before we allow forwarding
/sbin/iptables -t nat -F
/sbin/iptables -t filter -F
/sbin/iptables -t filter -P INPUT DROP
/sbin/iptables -t filter -P OUTPUT ACCEPT
/sbin/iptables -t filter -P FORWARD ACCEPT
/sbin/iptables -t nat -P PREROUTING DROP
/sbin/iptables -t nat -P POSTROUTING ACCEPT
/sbin/iptables -t nat -P OUTPUT ACCEPT

# Turn on Kernel forwarding of packets – note that this has to be enabled in your kernel
echo “1″ > /proc/sys/net/ipv4/ip_forward
echo “1″ > /proc/sys/net/ipv4/ip_dynaddr

# PREROUTING chain
# Do some checks for obviously spoofed IP Addresses
# Dump them as soon as we see them
# This rule is for geoip in case you have this compiled into your kernel and you want to block access from, say, China
#/sbin/iptables -t nat -A PREROUTING -i $EXTIF -m geoip ! –src-cc US,CA -j DROP

/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 10.0.0.0/8 -j DROP
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 127.0.0.0/8 -j DROP
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 172.16.0.0/12 -j DROP
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 192.168.0.0/16 -j DROP
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 224.0.0.0/4 -j DROP
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 240.0.0.0/5 -j DROP
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -s 0.0.0.0/8 -j DROP

# Do the same for the virtual port too
#/sbin/iptables -t nat -A PREROUTING -d $VIPIP -m geoip ! –src-cc US,CA -j DROP
# Do the same for the virtual port too

# ENABLE MASQUARADING AND ALLOW INSIDE LAN TO GET OUT FOR FORWARDING
# HANDLE FORWARDING RULES

# POSTROUTE RULES
# these rules will ensure that outbound traffic from specific IPs:ports
/sbin/iptables -t nat -A POSTROUTING -o eth0 -s $HOSTA -p tcp –dport 25 -j SNAT –to 10.0.0.21 #HOSTA
/sbin/iptables -t nat -A POSTROUTING -o eth0 -s $HOSTB -p tcp –dport 25 -j SNAT –to 10.0.0.22 #HOSTB
/sbin/iptables -t nat -A POSTROUTING -o eth0 -s $HOSTC -p tcp –dport 25 -j SNAT –to 10.0.0.23 #HOSTC
/sbin/iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# ACCEPT ANYTHING ON LOOPBACK AND PPP0
/sbin/iptables -A INPUT -i $LOIF -j ACCEPT
/sbin/iptables -A OUTPUT -o $LOIF -j ACCEPT
/sbin/iptables -A INPUT -i tun+ -j ACCEPT
/sbin/iptables -A INPUT -i $INTIF -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p tcp -d $EXTIP –dport 22 -j ACCEPT

#port 80
/sbin/iptables -t nat -A PREROUTING -p tcp -d $VIPIP –dport 80 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -p tcp -d $VIPIP2 –dport 80 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -p tcp -d $VIPIP3 –dport 80 -j ACCEPT

#port 443
/sbin/iptables -t nat -A PREROUTING -p tcp -d $VIPIP –dport 443 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -p tcp -d $VIPIP2 –dport 443 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -p tcp -d $VIPIP3 –dport 443 -j ACCEPT

#port 1194 – openVPN port
#/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p udp -d $EXTIP –dport 1194 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p udp -d $EXTIP –dport 161 -j ACCEPT

#icmp rules
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p icmp –icmp-type echo-request -d $EXTIP -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p icmp –icmp-type echo-request -d $VIPIP -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p icmp –icmp-type echo-request -d $VIPIP2 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p icmp –icmp-type echo-request -d $VIPIP3 -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p tcp -d 10.0.0.21  –dport 25 -j DNAT –to-destination $HOSTA
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p tcp -d 10.0.0.22  –dport 25 -j DNAT –to-destination $HOSTB
/sbin/iptables -t nat -A PREROUTING -i $EXTIF -p tcp -d 10.0.0.23  –dport 25 -j DNAT –to-destination $HOSTC

#next rule is for the openVPN tunnel
/sbin/iptables -t nat -A PREROUTING -i tun+ -j ACCEPT
/sbin/iptables -t nat -A PREROUTING -i $INTIF -j ACCEPT

# Inside of firewall gets anything
# Allow the forwarding of ESTABLISHED and related packets from the outside

/sbin/iptables -A FORWARD -i $EXTIF -m state –state ESTABLISHED,RELATED -j ACCEPT

# Now allow the inside to create new connections
/sbin/iptables -A FORWARD -i $INTIF -m state –state NEW,ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A FORWARD -i tun+ -j ACCEPT

# We did forwarding above, Now handle input packets for the external interface
/sbin/iptables -A INPUT -i $EXTIF -p icmp –icmp-type echo-request -d $EXTIP -j ACCEPT
/sbin/iptables -A INPUT -i $EXTIF -p icmp –icmp-type echo-request -d $VIPIP -j ACCEPT
/sbin/iptables -A INPUT -i $EXTIF -p icmp –icmp-type echo-request -d $VIPIP2 -j ACCEPT
/sbin/iptables -A INPUT -i $EXTIF -p icmp –icmp-type echo-request -d $VIPIP3 -j ACCEPT

/sbin/iptables -A INPUT -i $EXTIF -p tcp -d $EXTIP –dport 22 -m state –state NEW -j ACCEPT

/sbin/iptables -A INPUT -p tcp -d $VIPIP –dport 80 -m state –state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp -d $VIPIP2 –dport 80 -m state –state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp -d $VIPIP3 –dport 80 -m state –state NEW -j ACCEPT

/sbin/iptables -A INPUT -p tcp -d $VIPIP –dport 443 -m state –state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp -d $VIPIP2 –dport 443 -m state –state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp -d $VIPIP3 –dport 443 -m state –state NEW -j ACCEPT

/sbin/iptables -A INPUT -i $EXTIF -p udp -d $EXTIP –dport 1194 -m state –state NEW -j ACCEPT
/sbin/iptables -A INPUT -i $EXTIF -p udp -d $EXTIP –dport 161 -m state –state NEW -j ACCEPT

/sbin/iptables -A INPUT -i $EXTIF -p ALL -d $EXTIP -m state –state ESTABLISHED,RELATED -j ACCEPT

/sbin/iptables -A INPUT -p ALL -d $VIPIP -m state –state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -p ALL -d $VIPIP2 -m state –state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -p ALL -d $VIPIP3 -m state –state ESTABLISHED,RELATED -j ACCEPT

# This sets the rate limits
#/sbin/iptables -A OUTPUT -m limit –limit 3/minute –limit-burst 3 -j LOG –log-level DEBUG –log-prefix “IPT OUTPUT packet died: “
eend $?
}

stop(){
/sbin/iptables -t nat -F
/sbin/iptables -t filter -F
/sbin/iptables -t filter -P INPUT DROP
/sbin/iptables -t filter -P OUTPUT ACCEPT
/sbin/iptables -t filter -P FORWARD ACCEPT
/sbin/iptables -t nat -P PREROUTING DROP
/sbin/iptables -t nat -P POSTROUTING ACCEPT
/sbin/iptables -t nat -P OUTPUT ACCEPT
eend $?

}
open(){
/sbin/iptables -t nat -F
/sbin/iptables -t filter -F
/sbin/iptables -t filter -P INPUT ACCEPT
/sbin/iptables -t filter -P OUTPUT ACCEPT
/sbin/iptables -t filter -P FORWARD ACCEPT
/sbin/iptables -t nat -P PREROUTING ACCEPT
/sbin/iptables -t nat -P POSTROUTING ACCEPT
/sbin/iptables -t nat -P OUTPUT ACCEPT

# Turn on Kernel forwarding of packets
echo “1″ > /proc/sys/net/ipv4/ip_forward
echo “1″ > /proc/sys/net/ipv4/ip_dynaddr
/sbin/iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
eend $?

}
restart() {
svc_stop
svc_start
eend $?
}

With that you should be able to get your http/https load balancers up, running, and managing your connections.  This config allows traffic to ports 80 and 443 and maps the appropriate IPs and port 25 for mail.

If you want to see the number of active connections through your load balancer is to execute ipvsadm –list and view the number of active and inactive connections at each real and virtual IP.  Additionally when your webservers send mail it will appear to come from the real IP assigned for each host.