XenServer boot from iSCSI

UPDATE: XenServer boot from iSCSI (at least with iBFT) is just plain awful. Don't bother.  Multipath does not work. Too many hacks to try to get things to work. It is not supported, anyway, and it will probably break every time you do an upgrade.  Plus, the NICs that you use for iSCSI boot will be unusable for any other purpose.  This is aspect of XenServer is very immature and not robust.

1. Credits (Special thanks for pointers from:)
1. https://www.krystalmods.com/index.php?title=xenserver-6-supports-iscsi-boot-undocumented-feature&more=1&c=1&tb=1&pb=1
2. http://serverfault.com/questions/598773/install-xenserver-on-iscsi-target
3. http://serverfault.com/questions/431864/boot-from-iscsi-how-does-it-work
2. Notes/Warnings
1. This was originally writtenf or XenServer 6.2.
2. The NIC that is used for iSCSI boot will not be available for use for any other purposes (admin network, regular storage network, VM network, etc.)
3. This is with Intel I350 Gbit NICs, as on a Supermicro X9DRT motherboard, which uses the ibft module.
4. Once booted, you're in a Busybox environment... some commands will be limited or may not work the way in which you are accustomed on a full Linux system. Consider the information here for more troubleshooting steps to confirm that iSCSI works (LUNs are reachable, access is granted, etc.)
5. As a best practice, use all lower-case named target and initiator names (some firmwares may silently convert case).
6. WARNING: If you do not have your LUNs properly masked, do not specify the installation target correctly, and so forth, you may destroy your data. Know what you are doing, and use at your own risk.
3. Connect up the Intel NICs
4. Enable the Option ROM in the BIOS and add to the boot order; configure the iSCSI boot-enabled NICs in the NIC ROM set-up (perhaps ctrl-D when prompted after POST and prior to OS boot); You'll want to configure networking, initiator and target name; consider following these SAN best practices
5. Configure the iscsi target to allow access from this initiator
6. Boot off the CD and enter the command shell:
1. Insert the XenServer 6.2 CD and power on the computer.  At the "boot: " prompt, type "shell" and press enter.
7. Prepare iscsi:
1. echo "InitiatorName=iqn.2014-01.local:hostname" > /etc/iscsi/initiatorname.iscsi (replace the initiator name and/or hostname as appropriate)
2. echo "node.session.initial_logon_retry_max = 60" >> /etc/iscsi/iscsid.conf
3. modprobe iscsi_ibft
4. modprobe scsi_transport_iscsi
5. modprobe iscsi_tcp
6. iscsid -c /etc/iscsi/iscsid.conf -i /etc/iscsi/initiatorname.iscsi -f &
8. set up multipath (if you have multiple paths)
1. modprobe dm-multipath
9. Start the installer
1. /opt/xensource/installer/init --use_ibft
2. (use --mpath if you have set up multipath)
3. (use --device_mapper_multipath=true)
4. (use --network_device=eth0 , or the correct iSCSI network device, if needed)

Multipath doesn't seem to "just work"... at one point, I had to do this, because if I don't, then installer runs without multipath, but when the init does switchroot, the OS hangs and complains that "a device that was previously mounted without multipath is now mounted with multipath."

1. Go to http://debaan.blogspot.com/2014/10/iscsi-stuff-in-busybox-on-linux.html and follow those steps to get logged in to both LUNs and get them seen with multipath.
2. Start the installer as before.  Wait until the last step, when it says that it is ready to reboot.
3. Alt+F2 to get to a command shell (again).
4. Set up a chroot:
1. mount -o bind /dev /tmp/root/dev
2. mount -o bind /sys /tmp/root/sys
3. mount -o bind /proc /tmp/root/proc
4. chroot /tmp/root /bin/bash
5. cd /boot
6. re-run the
   iscsiadm -m discovery -t sendtargets -p 10.10.10.20 commands to make the chroot environment aware of the targets
7. mkinitrd --with=dm-multipath --with=iscsi_ibft --with=scsi_transport_iscsi -f /boot/initrd-2.6.32.43-0.4.1.xs1.8.0.835.170778xen 2.6.32.43-0.4.1.xs1.8.0.835.170778xen
8.  

iSCSI stuff in Busybox on Linux

1. Notes/Warnings
1. This works with Intel I350 Gbit NICs, as on a Supermicro X9DRT motherboard, which uses the igb (1Gbit) or ixgbe (10Gbit) module as the NIC device driver, and the iscsi_ibft module for iscsi boot. You'll have to change the module names if using Broadcom or other chips.
2. Once booted, you're in a Busybox environment... some commands will be limited or may not work the way in which you are accustomed on a full Linux system.
3. As a best practice, use all lower-case named target and initiator names (some firmwares may silently convert case).
4. WARNING: If you do not have your LUNs properly masked, do not specify the installation target correctly, and so forth, you may destroy your data. Know what you are doing, and use at your own risk.
2. Connect up the Intel NICs
3. Configure the iscsi target to allow access from this initiator (hint: this is done on your SAN), and mask out other LUNs to be NOT visible to this initiator
4. Boot off the CD and enter the busybox command environment
5. Manually configure the ip address and enable the interfaces, and verify connectivity
1. ifconfig eth0 inet 10.10.10.15 netmask 255.255.255.0 up
2. route add default gw 10.10.10.1
3. ping 10.10.10.1; ping www.google.com
4. At this point you can enable ssh to access this from a different computer over the network, if you need to:
1. ssh-keygen -f /etc/ssh_host_rsa_key -t rsa  (use an empty passphrase)
2. ssh-keygen -f /etc/ssh_host_dsa_key -t dsa (use an empty passphrase)
3. /usr/sbin/sshd
4. echo "root:supersecret" | chpasswd
5. (now, copy the hashed password for root from /etc/passwd to /etc/shadow
6. Prepare iscsi:
1. mkdir /etc/iscsi
2. echo "InitiatorName=iqn.2014-01.local:hostname" > /etc/iscsi/initiatorname.iscsi (replace the initiator name and/or hostname as appropriate)
3. echo "node.startup = automatic" > /etc/iscsi/iscsid.conf
4. echo "node.session.initial_logon_retry_max = 60" >> /etc/iscsi/iscsid.conf
5. modprobe iscsi_ibft
6. modprobe scsi_transport_iscsi
7. modprobe iscsi_tcp
8. iscsid -c /etc/iscsi/iscsid.conf -i /etc/iscsi/initiatorname.iscsi -f &
7. detect iscsi targets (where 10.10.10.20 is the IP address of the iSCSI SAN target) (only do this if you are not going to use the ibft boot stuff from an OS installer)
1. iscsiadm -m discovery -t sendtargets -p 10.10.10.20
2. iscsiadm -m node -l
8. set up multipath (if you have multiple paths)
1. modprobe dm-multipath
2. multipath -l -r
3. multipath -l -r (verify that the paths exist and are recognized...)
9. At this point, you can use the multipath device to access your iSCSI LUNs as needed.
10. If you want to run an OS installer, at this point start the installer
1. (in CentOS or XenServer installation media, append --use_ibft)
2. (use --mpath if you have set up multipath)
3. (use --network_device=eth0 , or the correct iSCSI network device, if needed)

SNMP hints

First, install any vendor custom mib:  On RHEL, CentOS, and related, copy the vendor MIB file to /usr/share/snmp/mibs/ , with a .txt extension; rename the file to be the same as the MIB name, e.g., NETAPP-MIB.txt

(in the examples below, supersecret is the community string, and myhostname is the hostname / ip addr of the target.

To get a specific value, based on textual MIB name:
snmpget -v 2c -c supersecret myhostname NETAPP-MIB::dfFileSys.1

To walk, with textual/name (not numeric) descriptions/OIDs:
snmpwalk -v 2c -c supersecret -m +ALL myhostname 1

...the one at the end means "give me everything".  THe "-m +ALL" means "use all MIBs".

Manipulate windows firewall from CLI

The windows firewall UI is a bit cumbersome.  This method will let you set up a host reliably to be secure, and to allow only inbound traffic that you want.

Enable firewall in all profiles (home / domain / public)
netsh advfirewall set allprofiles state on

By default, deny all inbound and allow all outbound traffic
netsh advfirewall set allprofiles firewallpolicy blockinbound,allowoutbound

Add rule to allow SMTP traffic inbound to a specific port from a specific network range
netsh advfirewall firewall add rule name="Allow Inbound TCP/25 from SMTP relay hosts" protocol=TCP dir=in localport=25 action=allow remoteip=10.20.30.0/24

Add rule to allow all HTTP and HTTPS traffic inbound
netsh advfirewall firewall add rule name="Allow Inbound TCP/80 from everywhere" protocol=TCP dir=in localport=80 action=allow

Centos AD Authentication and users and groups

To configure your linux workstation to pull user and group, and authentication information, from AD, run these commands. They do the dirty work of configuring pam, samba+winbind, nscd, and Kerberos.

..substitute your admin user account where mine is used below (admin-username), your AD dns domain/realm where domainname.com is used, and the netbios domain name where domainname is used.

yum install samba pam_krb5.x86_64 pam_smb.x86_64 nscd

authconfig --enableshadow --passalgo=sha512 --disablenis --disableldap --disableldapauth --disableldaptls --disablesmartcard --disablerequiresmartcard --enablekrb5 --krb5kdc=dc1.domainname.com --krb5adminserver=dc1.domainname.com --krb5realm=DOMAINNAME.COM --enablekrb5kdcdns --enablekrb5realmdns --disablesmbauth --smbworkgroup=DOMAINNAME --smbservers=dc1.domainname.com,dc2.domainname.com --enablewinbind --disablewinbindauth --smbsecurity=ads --smbrealm=DOMAINNAME.COM --smbidmapuid=10000000-20000000 --smbidmapgid=10000000-20000000 --winbindseparator=\\ --winbindtemplatehomedir=/home/%D/%U --winbindtemplateshell=/bin/bash --enablewinbindusedefaultdomain --enablewinbindoffline --winbindjoin=admin-username --disablewins --disablehesiod --enablecache --enablelocauthorize --enablepamaccess --disablesysnetauth --enablemkhomedir --updateall

Note that all users on the domain will now be able to log in to your computer over the network, unless you either:

1. Set up a ssh AllowUsers or AllowGroups parameter in /etc/ssh/sshd_config (see man page for sshd_config); or

2. 2. Use pam_access (see man page for pam_access)

bond interfaces in Xen

CentOS 5.2 x86_64 / RHEL 5U2

bond type: 802.3ad

We're going to bond eth0 and eth1 in a 802.3ad bond (bond0), and make that the primary bridge for Xen (xenbr0). It will be in vlan access mode (i.e. no vlan trunk). It will also be the primary interface for dom0. Eth2 and Eth3 are present, but not used.

Configure the switchports where eth0 and eth1 are plugged in for 802.3ad, lacp.

/etc/modprobe.conf (bond interface must be first, then load drivers for other network interfaces)

alias bond0 bonding
alias eth0 e1000
alias eth1 e1000
alias eth2 e1000
alias eth3 e1000

change this line in /etc/xen/xend-config.sxp

(network-script 'network-bridge netdev=bond0')

Ensure you have a line in /etc/sysconfig/network to define that bond0 should be used as the default gateway:

GATEWAYDEV=bond0

/etc/sysconfig/network-scripts/ifcfg-bond0 (adjust for your ip settings, use a unique MAC address):

DEVICE=bond0
IPADDR=10.10.10.119
MACADDR=00:00:10:01:01:19
NETMASK=255.255.255.0
NETWORK=10.10.10.0
BROADCAST=10.10.10.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="mode=4 miimon=100"

/etc/sysconfig/network-scripts/ifcfg-eth0 (adjust according to your device's true mac address):

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
HWADDR=00:1E:68:37:FA:92

/etc/sysconfig/network-scripts/ifcfg-eth1 (adjust according to your device's true mac address):

DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
HWADDR=00:1E:68:37:FA:93

/etc/sysconfig/network-scripts/ifcfg-eth2 (adjust according to your device's true mac address):

DEVICE=eth2
BOOTPROTO=dhcp
HWADDR=00:1E:68:37:FA:94
ONBOOT=no

/etc/sysconfig/network-scripts/ifcfg-eth3 (adjust according to your device's true mac address):

DEVICE=eth3
BOOTPROTO=dhcp
HWADDR=00:1E:68:37:FA:95
ONBOOT=no

802.3ad link aggregation

From wikipedia’s article on 802.3ad:

The most common way to balance the traffic is to use L3 hashes. These hashes are calculated when the first connection is established and then kept in the devices' memory for future use. This effectively limits the client bandwidth in an aggregate to its single member's maximum bandwidth per session. This is the main reason why 50/50 load balancing is almost never reached in real-life implementations, more like 70/30. More advanced distribution layer switches can employ an L4 hash, which will bring the balance closer to 50/50.
...
A limitation on link aggregation is that it would like to avoid reordering Ethernet frames. That goal is approximated by sending all frames associated with a particular session across the same link[3]. Depending on the traffic, this may not provide even distribution across the links in the trunk.

So, while 802.3ad does provide us redundancy, and the bandwidth of all connections to and from all hosts over a 802.3ad link will potentially aggregate to the combined bandwidth of the bonded interfaces, each individual client-server connection will be limited to the bandwidth available on a single component link of the bond.

It is arguable, then, that ALB (adaptive load balancing) may be a simpler and equally reliable method for bonding multiple interfaces on a server. In the case of ALB, no switch configuration is required, meaning that you can more readily swap ports on a switch.
…downside to ALB is that, because the server then presents one ip address with two different MAC addresses (one for each physical interface), when one interface loses connectivity, communication with any remote host that was using that physical interface will be lost until an ARP request causes the remote host to associate the IP address with the MAC of the remaining physical interface; some tcp sessions may timeout and fail entirely.

I am unclear as to whether gratuitous ARP is used to accelerate this failover.

I learned something new today.