bond interfaces in Xen

CentOS 5.2 x86_64 / RHEL 5U2

bond type: 802.3ad

We're going to bond eth0 and eth1 in a 802.3ad bond (bond0), and make that the primary bridge for Xen (xenbr0). It will be in vlan access mode (i.e. no vlan trunk). It will also be the primary interface for dom0. Eth2 and Eth3 are present, but not used.

Configure the switchports where eth0 and eth1 are plugged in for 802.3ad, lacp.

/etc/modprobe.conf (bond interface must be first, then load drivers for other network interfaces)

alias bond0 bonding
alias eth0 e1000
alias eth1 e1000
alias eth2 e1000
alias eth3 e1000

change this line in /etc/xen/xend-config.sxp

(network-script 'network-bridge netdev=bond0')

Ensure you have a line in /etc/sysconfig/network to define that bond0 should be used as the default gateway:

GATEWAYDEV=bond0

/etc/sysconfig/network-scripts/ifcfg-bond0 (adjust for your ip settings, use a unique MAC address):

DEVICE=bond0
IPADDR=10.10.10.119
MACADDR=00:00:10:01:01:19
NETMASK=255.255.255.0
NETWORK=10.10.10.0
BROADCAST=10.10.10.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="mode=4 miimon=100"

/etc/sysconfig/network-scripts/ifcfg-eth0 (adjust according to your device's true mac address):

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
HWADDR=00:1E:68:37:FA:92

/etc/sysconfig/network-scripts/ifcfg-eth1 (adjust according to your device's true mac address):

DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
HWADDR=00:1E:68:37:FA:93

/etc/sysconfig/network-scripts/ifcfg-eth2 (adjust according to your device's true mac address):

DEVICE=eth2
BOOTPROTO=dhcp
HWADDR=00:1E:68:37:FA:94
ONBOOT=no

/etc/sysconfig/network-scripts/ifcfg-eth3 (adjust according to your device's true mac address):

DEVICE=eth3
BOOTPROTO=dhcp
HWADDR=00:1E:68:37:FA:95
ONBOOT=no

802.3ad link aggregation

From wikipedia’s article on 802.3ad:

The most common way to balance the traffic is to use L3 hashes. These hashes are calculated when the first connection is established and then kept in the devices' memory for future use. This effectively limits the client bandwidth in an aggregate to its single member's maximum bandwidth per session. This is the main reason why 50/50 load balancing is almost never reached in real-life implementations, more like 70/30. More advanced distribution layer switches can employ an L4 hash, which will bring the balance closer to 50/50.
...
A limitation on link aggregation is that it would like to avoid reordering Ethernet frames. That goal is approximated by sending all frames associated with a particular session across the same link[3]. Depending on the traffic, this may not provide even distribution across the links in the trunk.

So, while 802.3ad does provide us redundancy, and the bandwidth of all connections to and from all hosts over a 802.3ad link will potentially aggregate to the combined bandwidth of the bonded interfaces, each individual client-server connection will be limited to the bandwidth available on a single component link of the bond.

It is arguable, then, that ALB (adaptive load balancing) may be a simpler and equally reliable method for bonding multiple interfaces on a server. In the case of ALB, no switch configuration is required, meaning that you can more readily swap ports on a switch.
…downside to ALB is that, because the server then presents one ip address with two different MAC addresses (one for each physical interface), when one interface loses connectivity, communication with any remote host that was using that physical interface will be lost until an ARP request causes the remote host to associate the IP address with the MAC of the remaining physical interface; some tcp sessions may timeout and fail entirely.

I am unclear as to whether gratuitous ARP is used to accelerate this failover.

I learned something new today.