Syslog on NetApp

Data Ontap has the ability to send system log messages to an industry standard syslog server (see https://library.netapp.com/ecmdocs/ECMP1196979/html/man5/na_syslog.conf.5.html)

To cause your Netapp to start logging to a syslog server named "logs.mycompany.com", you would use the wrfile to (over)write the syslog configuration file, directly from the console; leave a blank line at the end, and use ctrl-c to conclude the edit:

mynetapp> wrfile /vol/vol0/etc/syslog.conf
*.info    /dev/console
*.info    /etc/messages
*.info    @logs.mycompany.com
auth.*    @logs.mycompany.com
cmdsaudit.auditlog    @logs.mycompany.com


mynetapp>

(You should then see "syslogd restarted" shortly, when the NetApp detects the config file change.)

The "cmdsaudit.auditlog" line causes all console commands that are entered to also be logged to syslog -- thus, you have a record of who did what, when.

Storage Best Practices

Comments welcome.  These are just some things I've decided work best and save hassles:

General Storage

1. Compression should be used on the back-end storage, not in the VM or host that is mounting the storage.
2. Data should be grouped in volumes based on general purpose/function, performance requirements, backup requirements, and any isolation requirements. For example:
   1. DB logs should be on a separate hosting aggregate/raid-set from the databases that they help to protect.
   2. General business data should not be on the same volume as Financials data, because they have distinct back-up requirements.
   3. File share data and VM data should not be on the same underlying volume (though they may live on the same RAID set / aggregate.
3. NAS/SAN volumes and shares should be named to say as briefly as possible what they actually are:
   1. Scratch/temp data should always be named such that the use can see that it is "scratch" data. e.g., "localscratch0", "nas_scratch0".
   2. vi_eng_0  (virtual infrastructure for Engineering's VM's, number 0)
   3. volume IT_Software shared out as \\corp.mycompany.com\LS\IT\Software.
4. Security
   1. Permissions should be set for at least the share and directly-contained folders using domain-local security groups (using nested groups) that are specific to just that share and those folders.
   2. "Everyone" should almost never be used.
   3. "Deny" permissions should almost never be used; instead, create a group that includes only the desired people/roles.
5. Scratch data should not be backed up.  Take every relevant opportunity to remind users of that.
6. Linux
   1. LVM
      1. LVM should be used where possible -- always for the OS; for data disks it may be omitted for cases where separate LUNs are presented for data and there is only one filesystem per LUN.
      2. The OS should use a different Volume group from major application data. (simple LAMP boxes with a single disk device may us all on a single volume group; systems with more disk devices, or where one may wish to restore data LUNs from snapshot should have a separate VG_Data including all physical volumes holding application data
      3. All partitions on a single disk device should only belong to the same Volume Group. (this makes re-assembling a broken system easier... if the data volume group has a physical volume missing/broken or restored from snapshot, the system can still boot if the OS volume group is intact; then the data volume group can be re-assembled.).
      4. Logical volumes should be named with "LV" plus the mount path, substituting "_" for "/"; LV_root for the root "/" filesystem, and LV_swap for the swap filesystem.
      5. /boot should not be on a logical volume.
      6. Logical volumes should always be thick-provisioned
      7. Resizing
         1. Before any resize operation, always back up the data to external storage first!
         2. When growing a logical volume, always grow the logical volume first, using round "g" size, then resize the filesystem (don't specify a size, where possible, and let resize detect the new size).  If the hosting physical volume is to be grown, grow the hosting partition/LUN and then pvextend before growing the LUN.
         3. When shrinking a logical volume, always shrink the filesystem first, using roung "g" size, then resize the logical volume, also using round "g" size specification.  If the hosting partition/LUN is also to be shrunk, then it should be shrunk last using round "g" size specification.
   2. Filesystems should be labeled with the mount path, e.g. "/var/log" where it fits; if the path is too long, then the last unique part of the path should be used, e.g. "postgres-backups".
7. Windows
   1. Application Data for major applications and those where one may wish to restore data LUNs from snapshot separately from the OS should place application data on separate disk devices (not just separate partitions)
   2. Filesystems should be labeled based on mount/drive path and purpose, e.g. "C_OS", "D_Data", "F_Logs", "G_MailboxDB00", "C_Appdata_logs"

General SAN

1. LUNs should always be masked on the SAN target device to only allow access from only those initiators who require access to that LUN.
2. LUNs should be formatted using the  and mounted (fstab) mapped using the multipath device
3. Multipathing
   1. Multipathing should be used where possible, with each target (redundant initiators, switches/paths, and targets)
   2. Redundant targets should be used, as well: Initiator A should connect on subnet/switch A to the target LUN on SAN head A; initiator B should connect on subnet/switch B to the target LUN through SAN head B.
   3. Active-active with round-robin load balancing should be used.
   4. LUNs should generally be thinly provisioned (except for LUNs hosting essential databases); the monitoring system (Zabbix) should be configured to alert when the hosting aggregate is at 80, 90, and 95% of capacity..

FC

1. FC switches should use WWN-based zoning
2. Any host OS installation should include *unplugging* the HBA so that the OS does not wipe all visible LUNs. (If the host OS is being installed for boot-from-SAN, then all LUN masks and target attributes should be triple-checked to avoid inadvertent destruction of data.)
3. switches in separate paths should be managed separately. (Combining them into a single management domain would combine them into a single fault domain for administrative errors).
4. WWN's should, where be possible, be aliased all storage devices (switches and SAN) to include the hostname (and optionally a function) such that they can be easily identified, referenced, and so that all related zonings/mappings can be updated only by updating the alias itself. (For example, if an HBA must be replaced, that would invalidate all WWN-based mappings; but since we used an alias for all mappings, we just update the single alias definition that included that WWN.

iSCSI

1. all IQN's should always be typed as all lower-case.
2. Initiator IQN names should be:
   1. iqn.2014-01.local:hostname[-boot]
   2. The - is optional, and is only needed if there are several different initiators on a single host
   3. No iscsi traffic should be routed.  In other words, any given initiator should be connecting to a target on the same subnet.
3. Targets should be mapped using IP address, not Host name.
4. Jumbo frames (9000-Byte) should be used where possible.

SNMP hints

First, install any vendor custom mib:  On RHEL, CentOS, and related, copy the vendor MIB file to /usr/share/snmp/mibs/ , with a .txt extension; rename the file to be the same as the MIB name, e.g., NETAPP-MIB.txt

(in the examples below, supersecret is the community string, and myhostname is the hostname / ip addr of the target.

To get a specific value, based on textual MIB name:
snmpget -v 2c -c supersecret myhostname NETAPP-MIB::dfFileSys.1

To walk, with textual/name (not numeric) descriptions/OIDs:
snmpwalk -v 2c -c supersecret -m +ALL myhostname 1

...the one at the end means "give me everything".  THe "-m +ALL" means "use all MIBs".

Vi/Vim Tips

These are some things I always seem to forget and have to look up again; they don't seem to all be in the same place anywhere, or stated as succinctly.  Some are VIM only.

(type :help

Marking

m  : set a mark at the current position in this file(a-z) (per-file)
m  : set a *global* mark at the current position, reachable from different files.
:marks  : list marks
:marks aF : list marks a, F

Movement

(precede any of these with a number to repeat it that many times)
` : move to the position of a mark in this file (see "Marking")
' : move to the line of a mark in this file (see "Marking")
` : move to the position of a (global) mark in another file (see "Marking")
' : move to the line of a (global) mark in another file (see "Marking")
h   : move one character left
l   : move one character right
k   : move one character/row up
j   : move one character/row down
w   : move to beginning of next word after punctuation
W   : move to beginning of next word after whitespace
b   : move to beginning of previous word after punctuation
B   : move to beginning of previous word after whitespace
e   : move to end of this word
E   : move to the end of this word before whitespace
0   : move to beginning of the current line (column 0)
^   : move to the non-whitespace beginning of the current line
_   : move to the non-whitespace beginning of the current line with count (e.g. 5_)
$   : move to the end of a line
g_  : move to the last non-whitespace character of the line, with count (e.g. 5g_)
gg  : move to first line
G   : move to last line
ng  : move to line number "n"
H   : move to the first line of the screen (home)
M   : move to the middle line of the screen
L   : move to the last line of the screen
z   : re-center the screen on the current line (with the cursor)
zt  : re-center the screen with the current line (with the cursor) at the top
zb  : re-center the screen with the current line (with the cursor) at the top
ctrl-D : move a half-page down
ctrl-U : move a half-page up
ctrl-B : page up (back)
ctrl-F : page down (forward)
ctrl-o : return to last cursor position
ctrl-i : go to next cursor position
%    : jump to the matching (){}[]

Basic Editing

u    : undo last edit
U    : return last changed line to its former state
ctrl-R : redo last undone edit
cw   : change text of current word
c : replace one character
c`: change text from current position to position of marker
c': change text from current line to line of marker

Copy/Cut/Paste

p    : put/paste following this line
P    : put/paste preceding this line
dd   : delete/cut the current line
D    : delete/cut from current character to the end of the line
yy   : copy (yank) the current line
y$   : copy from current position to the end of the line
*y$  : copy from current position to the end of the line, place in system clipboard
*p   : paste from system clipboard to following line


d`: delete from current position to position of marker a-z
d': delete from current line to line of marker a-z
y`: copy (yank) text from current position to position of marker
y': copy (yank) text from current line to line of marker

Search / Replace

/string : search forward for "string"
?string : search backward for "string"
n    : find next occurrence of the search string (in the same direction as the search)
N    : find previous occurrence of the search string
*    : find the next occurrence of the word currently under the cursor
#    : find the previous occurrence of the word currently under the cursor
g*   : find the next occurrence of the search pattern under the cursor
g#   : find the previous occurrence of the search pattern under the cursor

Change Case

~    : Changes the case of current character 
guu  : Change current line from upper to lower. 
gUU  : Change current LINE from lower to upper. 
guw  : Change to end of current WORD from upper to lower. 
guaw : Change all of current WORD to lower. 
gUw  : Change to end of current WORD from lower to upper. 
gUaw : Change all of current WORD to upper. 
g~~  : Invert case to entire line

Global settings

(many of these can be permanently placed in your ~/.vimrc file)
set tabstop=4  : set how wide a "tab" is

Sources:

http://vim.wikia.com

http://stackoverflow.com/a/2966034 

http://stackoverflow.com/a/7764203 

Grow root partition and filesystem in Ubuntu and Linux Mint

Here are the steps to grow the root filesystem on an existing system that does NOT use LVM:

1. First, use LVM; it makes resizing filesystems nearly trivial, robust, without any real downsides.  Just another way that Ubuntu is not enterprise-worthy.  But, we don’t have LVM on these systems’s as they are….
2. Second, before doing any operations on your partitions, always perform and test a backup (to a separate system!) of your data. You may make your system unbootable and/or nuke all of your data!
3. The system I performed these steps on was partitioned thus:
1. /dev/sda – 7GB – /
2. /dev/sda2 – remainder – Extended partition
3. /dev/sda5 – swap – partition within the extended partition.
4. So, to grow “/”, we will become root, delete the swap partition and the extended partition, grow /, and then re-create the swap partition.  Adjust the steps below according to the partition numbers and layout of your particular system:
1. Disable swap:
1. swapoff /dev/sda5
2. Delete and re-create partitions as appropriate.
1. fdisk /dev/sda
2. print out partition information (p)
3. Delete partition 5 (d – 5)
4. Delete partition 2 (d – 2)
5. Delete partition 1 (d)
6. Create partition 1 (n – p – 1)
1. It must start on the exact same sector as before (as seen in the print command)
2. It must end on a sector higher than it did before.  Num_GB*1024*1024*2 = ending sector
7. Create partition 2 (n – p – 2)
8. Change partition 2 to type “Linux Swap” (t – 2 – 82 )
9. Activate partition 1 to make it bootable (a - 1)
10. Double-check everything.
11. Exit (w)
3. recreate the swap partition, using a label:
1. mkswap -L swap /dev/sda2
4. add a label to the root filesystem
1. e2label /dev/sda1  /   (in centos, it’s e4label)
5. Fix up /etc/fstab
1. Fix the “swap” line to use LABEL=swap instead of UUID=
2. Fix the “/” line to use LABEL=/ instead of UUID=
6. swapon /dev/sda2
7. Fix up /boot/grub/grub.cfg
1. Either: Find the “linux” line for the menu option you will boot, change UUID= to LABEL=/
2. Or: update-grub
8. Re-create the initrd
1. update-initramfs -u -k 3.2.0-38-generic
9. Reboot the VM. *Cross your fingers!* This is your moment of truth.
10. Grow the root filesystem
1. resize2fs /dev/sda1  (resize4fs on CentOS, I believe).
11. If it didn't boot, then that's what you get for biting off more than you could chew, and for choosing a distro that doesn't leverage LVM.  Boot off your Ubuntu/Mint install disk, and copy off your data to a USB disk, and start over. (...but you DID back up your data anyway, right?)

Linux bash shell options parsing

Normally, command line parameters come in as separate, positional values, and may be referenced as:

• $# - the number of command line arguments (positional parameters)
• $* - all positional parameters expressed as a single string
• $@ - all positional parameters, but with each as a quoted string (each positional parameter is intact and presented as a quoted string)
• $0 - (the base name of the script itself)
• $1 - The first positional parameter
• $2 - The second positional parameter, and $3, $4, etc.  Starting with 10, they must be expressed as ${10}, ${11}, etc.

Sometimes a variable is passed from a wrapper script to a child script where the positional parameter is in fact several parameters that should be parsed separately.

Here, we test for that case, and if found, we peel off the first parameter within that group of strings, and assign the remaining parameters in the group of springs:

# from the front, and make the rest of the data be the options passed to rsync.
if [ $# -eq 1 ]; then
{
   Params=($1)
   PARAM_1=${Params[0]}
   unset Params[0]
   ChildCommandOpts=${Params[*]}
} fi

Tweaks for Linux Mint

These were started with Linux Mint 14.1 Cinnamon

Menu bar shows up on the wrong screen:

Do this to find out the name of the displays:
sudo xrand -q
Do this to make the menu show up on one display or the other, for example:
sudo xrand --output VGA1 --primary

Sound doesn't come out through the correct device (speaker, headphone, etc.):

run these:
sudo apt-get install alsa-base alsa-utils pavucontrol pavumeter paman
alsamixer
...then in alsamixer, select the desired output "card", then ensure that "automute" is set to disabled or some other device.



I want to watch Netflix on Mint:

Run these commands:
sudo apt-add-repository ppa:ehoover/compholio
sudo apt-get install netflix-desktop



Display locks up with my NVidia card:

During install, select "compatibility mode".

After install, before logging in, select session type "Cinnamon 2D", then
sudo apt-get install nvidia-current nvidia-settings
sudo reboot



The future of CentOS

It appears that the CentOS project is languishing. There are insufficient resources to release 5.6, 6.0, and 6.1 in a timely fashion; installed servers are going unpatched. It appears from a quick health-check that the resulting attrition may prove an existential threat to the distribution. (Once I start moving everything to Debian, Ubuntu LTS, or some such, I'm probably not going to go back.)

Here's my thought on how to address resource constraints; with 100 businesses such as my own, 4-8 full-time devs could earn a decent wage, and CentOS may be able to survive.

Some reasonably sized consumers of CentOS, such as myself, have found that:

1. RHEL subscription costs are not reasonable.

2. CentOS is good enough that we run 80+ physical and virtual hosts.

3. CentOS is free. We like free. But we also recognize the value of the CentOS product to our business, and feel that our business should (and would) be willing to contribute something to the development, in particular if a release schedule can be assured. Perhaps $100 per vm/physical per year... for me, that might be $8000/yr to contribute to the project. Surely there are several more businesses that would be willing to contribute, again, for an assurance in return? A requirement of this, though, is for you to have a formal purchase order and billing process in place.

Bottom line is, my company (a very large company) does not have a process or a desire for "donation". If there's not a process for it, in a big business, it nearly can't be done. But ordering software, licenses and subscriptions, is something that happens every day as part of the regular business process.

I realize that for some devs it may be the love of the work, or a need for a good distro, or prestige more than money; but some who might be willing could jump in and help if their income were supplemented.

It's a great distro, a great product where stability is king. But we also need consistency and steadiness, or stability of lifecycle, if you will.

Check return code of piped command OR Export variable to parent shell

I just spent a day or more with a peer working the problem of "how do we get the return code of the first command in a series of piped commands in bash?"

The problem is: $? will hold the return code of the last command in the pipe sequence. We tried doing various things like ( foocmd ; export OUTERR=$?) | gzip... The problem with that is that, in bash, exported variables are not "global", so the value of $OUTERR is lost as soon as we hit the ). {}'s also did not work.

My partner's hack was to read stdout from foocmd into a variable, which he later cat'ed into gzip. Ick. Since we're dumping databases with foocmd, we're sure to run into architecture-dependent bash variable size limits, not to mention the RAM requirement of storing entire DB dumps into a variable.

As is usually the case, in the end we found that we'd spent so much time because we were going about it the wrong way, and we lacked the simple truth that would help us solve the problem efficiently.

${PIPESTATUS[@]} is an array similar to $?, except that it stores the return code of each component in a piped series. Thus, if we do this:

# /bin/false | tr x y | wc > /dev/null 2>&1

# echo ${PIPESTATUS[@]}

...we get as output...

1 0 0

...the first command, /bin/false returned "1", and the others returned "0" each.

Properly, we would test the value of each array element before considering the execution of the whole to be a success.

Now, that array will be overwritten/cleared the very next command, so the first thing we want to do is copy the array:

FOO_EXITCODE=("${PIPESTATUS[@]}")

So simple in the end.

Remote bulk file edits and administration with SSH and SED ( sed examples )

Want to deploy the zabbix agent to a bunch of Ubuntu Linux systems? Easy. But wait... the config file for each needs to be updated. How about this:

for target in host1 host2 host3 host4; do echo $target; ssh -t $target "apt-get -y install zabbix-agent; sed -i.bak -e \"s/Server=localhost/Server=10.10.1.11/g\" -e \"s/Hostname=localhost/Hostname=$target/g\" /etc/zabbix/zabbix_agent.conf /etc/zabbix/zabbix_agentd.conf; update-rc.d zabbix-agent enable; /etc/init.d/zabbix-agent restart"; done

This will:

1. ssh to each host
2. install the agent on that host
3. replace the default "Server=" and the "Hostname=" lines in the two config files zabbix_agent.conf and zabbix_agentd.conf", where 10.10.1.11 is the zabbix server ip address.
4. make a backup of the two config files
5. configure the zabbix_agent to auto-start
6. restart the zabbix agent to pick up the config file changes.

That was easier and more reliable than trying to complete the procedure on 50 systems.

(think about updating fstab and others for a mass of hosts.)

For a simple file in-place edit of one line of a file (such as to comment out a line on all the systems' config files):

for target in host1 host2 host3 host4; do echo $target; ssh $target "sed -i.bak -e 's/^domain mynisdomain server mynismaster.company.com$/g #domain mynisdomain server mynismaster.company.com' /etc/yp.conf"; done

If you have a file to edit, and the line you want to replace has quote marks, you'll need to escape them with \\\ like so:

for target in host1 host2 host3 host4; do ssh $target "sed -i.bak -e \"s/^ENABLED=\\\"false\\\"/ENABLED=\\\"true\\\"/g\" /etc/default/sysstat "; done

Ubuntu tweaks

I'll try to update this to keep track of the tweaks I find necessary for ubuntu to work well.

Install System-wide Pulseaudio Equalizer:

sudo add-apt-repository ppa:psyke83/ppa

sudo apt-get update

sudo apt-get install pulseaudio-equalizer

Install Pulseaudio tools and alsaplayer plugin, etc:

sudo apt-get install libasound2-plugins "pulseaudio-*" paman padevchooser paprefs pavucontrol pavumeter

Multimedia Playing

sudo apt-get install totem vnc ubuntu-restricted-extras

sudo apt-get install libdvdread4 && sudo /usr/share/doc/libdvdread4/install-css.sh

Send email from cli with attachments

sudo apt-get install mutt

echo "see attached" | mutt -s "file attached" -a Documents/somefile.xls -- jdoe@somewhere.com

Network Troubleshooting Tools

sudo apt-get install iputils-tracepath nmap aircrack-ng

Remote Systems Administration

sudo apt-get install gtkvnc rdesktop tsclient

GENERIC Linux P2V, V2V, P2P HowTo

Generic steps to P2V/ V2V/ V2P/ P2P/ clone Linux with dissimilar hardware and/or hypervisors 

To handle cases where a simple right-click and clone or copy of vhd/vmdk won't suffice.  From my blog at http://debaan.blogspot.com/    -Lane Bryson 

1. Background 

Purpose 

This process will copy a running system (physical or virtual) to a running environment (physical or virtual).  It is bandwidth efficient.  It is not easy, but it works reliably with some caveats: new platforms, new rescue CD's, changes in the standard storage drivers, and so forth, may require additional work. A strong mid- or senior-level Linux administrator should be able to complete these steps. 

Benefits 

Depending on the applications/services on the source system, you don’t have to bring them down for this cloning operation. 

Because this uses rsync rather than a disk-imaging tool: At best, you can run this process potentially with zero downtime; at worst, you can run this process (when you’re good at it) with downtime only slightly longer than the amount of time it takes to make a final 2nd or 3rd rsync (this depends on how much data changes and how quickly). 

Because you configure partitioning yourself, you can take the opportunity to adjust sizes as desired.  With both an imaging tool or this process, you’ll have to rebuild your initrd from a rescue disk if the source and target storage drivers, device names or paths (including logical volumes) have changed. 

This process requires no additional tools such as ghost or Acronis. 

Caveat Emptor 

This process is derived from my own notes and experience.  It works for me every time – probably close to 30-40 times so far.  However, an understanding of the process is needed, as you’ll have to fill in the blanks and apply the process to your specific environment (specific distro, storage config, network, virtualization platform and environment, etc.).  Test it out thoroughly on non-production systems first, until you are comfortable with it. 

Things are constantly changing; there are many different distributions, hypervisors and countless hardware combinations. I will not update this for every case. Use this as a starting point and adjust as appropriate for your situation. 

Use at your own risk!  You may destroy things, and I will not stop you. 

The trick here is that we rsync a running system.  Provided the system is fairly quiet or has static data (like a print server), we won’t likely lose any data.  Any data that changes on the source system from the time of the last rsync until the destination system is brought online (and the source system taken offline) will be lost – unless you stop the service or application on the source before the last rsync.  If this is, e.g., a db server we’ll only have an outage while/if we stop the database service for the final rsync. If this is a static DNS or print server, and we don’t care about losing data in the logs, then we probably don’t need to stop anything, and can make the transition without anybody knowing it’s happened. 

This would not be a suitable process for copying e.g. a system with TB’s of dynamic data on local storage – the rsync’s alone would take so long you’d probably quiet the system and take a scheduled outage.  

If those  caveats are acceptable to you, this process will be very reliable, if very manual. It could be scripted, but usually my source systems are varied quite a bit, and I take this opportunity to do things that require human intelligence like: 

• Re-evaluate RAM and filesystem size, cpu core count 

• convert from static partitions in the source to LVM-based filesystems in the destination 

2. Source Platform Prep 

If the target is a different platform type than the destination, and if you can take the possible performance hit (temporarily) or a required reboot, then uninstall any specialized hardware device drivers that are not absolutely required (e.g. raid monitoring, ATI catalyst drivers, VMware Tools, XenServer tools, etc).  DO NOT uninstall essential network or storage drivers.  While some of this can be done from within the chroot on the target, it is probably easier to do on the source. 

3. Prepare the Target Platform 

Hardware/Physical Target Platform 

Configure the RAID, physical disks and BIOS boot order as desired.  Details are beyond the scope of this document. 

Xen Paravirtualized Target: a special case 

Why so much work here? Because Linux paravirtualized domains run with extremely low overhead on XenServer.  But our utility/rescue CD requires a fully virtualized (“HVM”) vm in which to run.  And there’s not (to my knowledge) a simple way to switch a vm from “HVM” to paravirt. 

If the target will be a paravirtualized Xen vm, then you’ll need to clone a pre-existing post-install one or manually set some flags; it's easier just to install CentOS as paravirtualized to get the correct settings, then just remove the disks and connect new ones in the next steps. 

Create a HVM domain (from the "other OS" template), and attach the storage from the post-install paravirtualized domain.  This HVM domain will be used to set up the disk and copy files from the source machine/vm. 

Specifically: 

• Install a like-versioned (same distro and version as on the source platform) vm from the template.  If you plan to re-use it because you’re doing a lot of P2V or V2V’s, then call it something like “Paravirt Post-Install” and clone it.  In the end of this whole process, this is the vm that will ultimately replace the source system.   
• Remove the storage on the "Paravirt Post-install" VM and customize the VM to what you need, probably the same as in in the source. (Set the CPU count and architecture, RAM, priority, startup parameters, etc; you may set the MAC to be the same as your source Linux system, just don't power it on at the same time as the source!) 
• Create a HVM domain (from the "other OS" template), and create disk devices –these are the disk devices that will end up in the final target VM.  This HVM domain will be used temporarily to set up the disk and copy files from the source machine/vm. 

Now you have a paravirtualized domain ready to receive the finalized disks, and a HVM domain where you will do all the heavy lifting and prepare the disks. 

VMware ESX VM Target Prep 

Create a Vm according to the spec’s you need: cpu count and architecture, RAM, startup settings, disk devices, etc. 

Recommend e1000 network adapter and LSI Logic parallel SCSI adapter. 

Physical System Destination Prep 

Configure the physical box as you require.  The closer to the source configuration, the better. CPU architecture (32-bit or 64 bit) must be the same. 

4. Target System Bringup 

(note that all of these commands/operations will be performed in the target vm/physical platform. 

Note for legacy source OS 

If migrating a legacy (e.g., RHEL/CentOS 4.x or 3.x) Linux guest, where sata devices might be recognized as hda, hdb, etc., then boot with the systemrescuecd 0.4.3 (may have to use the alternative kernel, vmlinuz2), to ensure that the devices show up after boot with the correct (hdaX, not sdaX) device names. 

Boot the LiveCD 

1. Boot the Target platform from the LiveCD.  I prefer to use the System RescueCD (downloadable from http://www.sysresccd.org/ ). 

1. If source is running a 64-bit kernel, then at the boot prompt type “rescue64 docache”, else type “rescue docache”  

2. Configure network on the target: 

1. net-setup eth0 

3. Partition the disk(s) in the target: 

1. fdisk /dev/sda (or /dev/hda) 

2. This is the opportunity to clean things up, divide filesystems, allocate more storage, etc. 

3. Note: I generally convert things to use LVM, which will enable dynamic partition resizing down the road.  In that case, make partition 1 on the first disk of type “Linux” (for /boot), 500 MB, make it active; create a partition to use the remainder of the disk(s) of type “Linux LVM”. 

4. If your intent is to have the target be LVM-based: 

1. pvcreate /dev/sda2 

2. vgcreate VolGroup00 /dev/sda2 

3. lvcreate --size 10G --name LV_root VolGroup00 

4. Create/prepare other physical volumes, volume groups, and logical volumes as appropriate. (see the LVM documentation for help. This doc is not an LVM tutorial) 

4. Format the filesystems in the target (use the appropriate filesystem type, label, and path): 

1. regular filesystems …   ….For example: 

1. mkfs.ext3 -L /boot /dev/sda1 

2. mkfs.ext3 -L / /dev/VolGroup00/LV_root 

2. swap filesystem(s) …   ….For example: 

1. mkswap -L swap /dev/VolGroup00/LV_swap 

3. **Note: If the source system is RHEL/CentOS 3.x or another equally old platform, then the livecd has probably created the filesystem with options that will render it incompatible with the OS that will eventually run on the target platform; you’ll have to use a “-I 128” in the mkfs.ext3 command, and then run this for each ext3 filesystem in order to remove the new ext3 attributes from the filesystem on the target: 

1. tune2fs -O ^dir_index /dev/hda1 

2. debugfs -w /dev/hda1 -R “features ^resize_inode” 

5. Mount your target systems to a sandbox area, tmproot: 

1. mkdir /mnt/tmproot 

2. mount /dev/(root-device) /mnt/tmproot/ 

3. For example:  

1. mount /dev/VolGroup00/LV_root /mnt/tmproot 

2. mkdir /mnt/tmproot/boot 

3. mount /dev/sda1 /mnt/tmproot/boot 

4. (…mount all other filesystems relative to the tmproot sandbox.) 

6. Perform an initial copy of the source to the target 

1. (on System Rescue CD): Enter the Bash shell, because zsh on sysresccd will interpret”--" incorrectly. 

1. /bin/bash 

2. Make a first-pass copy of the source to the destination, where is replaced by the ip address of the source system. 

1. rsync --archive --hard-links --numeric-ids --sparse --exclude=/proc/** --exclude=/sys/** --exclude=/selinux/** --exclude=/media/** --exclude=/tmp/** --exclude=/mnt/** :/ /mnt/tmproot 

2. ** Note: exclude any nfs-mounted paths (such as home), as well as any other paths you don’t want to copy over, in the command above, as these system paths have been excluded. 

3. Optionally re-run the rsync command, just to get a more current consistent snapshot. (the second execution should be much faster.)  At this point, if the source platform can tolerate downtime, you may stop all databases and other data-changing services/processes prior to running the sync, and leave it off; this will ensure that there is no data loss during the copy. 

7. Set up your chrooted environment and boot devices, necessary for initrd config 

1. mount -t proc none /mnt/tmproot/proc 

2. mount -o bind /sys /mnt/tmproot/sys 

3. cp /etc/mtab /mnt/tmproot/mtab 

4. Verify that the device nodes for your boot device are in /mnt/tmproot/dev ; if not, copy them from /dev. 

5. Add the LVM device nodes 

1. chroot /mnt/tmproot /bin/bash 

2. vgscan --mknodes 

6. For RHEL3 / CentOS3 migration to PV (Here are the notes I took while I changed it over to the Paravirt(PV) kernel; this was on XenServer 5 or such; not sure if they still appl, but including for posterity): 

1. Before installing the xs-tools, change your /etc/fstab and /etc/mtab to point to devices that start with "xv" (in other words, translate hda1 to xvda1. Cdrom drive is xvdd). Also, issue these:  mknod /dev/xvda b 202 0  mknod /dev/xvda1 b 202 1  mknod /dev/xvda2 b 202 2  mknod /dev/xvda3 b 202 3  mknod /dev/xvda4 b 202 4  mknod /dev/xvc0 c 204 191  mknod /dev/xvdd b 202 48 

2. Add “alias eth0 xen_net” to /etc/modules.conf 

3. Now start install.sh on the xs-tools.iso.   

4. It should have copied the new kernel and built a new initrd into your /boot. scp these over to the host. I placed mine in /opt/kernels/NameOfVM. 

5. Shutdown the VM. 

6. From the host's console, you'll need to change several parameters.  xe vm-list (gives the list of all VM's.)  xe vm-list uuid=VMUUIDFROMABOVECOMMAND params=all  xe vm-param-set uuid=VMUUIDFROMABOVECOMMAND HVM-boot-policy=""  xe vm-param-set uuid=VMUUIDFROMABOVECOMMAND PV-args="root=/dev/xvda2 xencons=xvc"  xe vm-param-set uuid=VMUUIDFROMABOVECOMMAND PV-bootloader=""  xe vm-param-set uuid=VMUUIDFROMABOVECOMMAND PV-kernel="/opt/kernels/RHEL3/vmlinuz-2.4.21-47.0.1.EL.xs5.5.0.42xenU"  xe vm-param-set uuid=VMUUIDFROMABOVECOMMAND PV-ramdisk="/opt/kernels/RHEL3/initrd-2.4.21-47.0.1.EL.xs5.5.0.42xenU.img"  (Also you need to change the VBD to "bootable".)  xe vm-disk-list uuid=VMUUIDFROMABOVECOMMAND  xe vbd-param-set uuid=VBD!!UUIDFROMABOVECOMMAND bootable=true  (Reboot your VM from XenCenter and it should be good to go, with the exception of possible driver issues.) 

7. exit (back out of changeroot) 

8. Final target chroot fixup prior to initrd/kernel configuration; These are the things that most often result in a non-bootable system, if a detail is missed (e.g., "switchroot failed... panic"): 

1. mount -o bind /dev /mnt/tmproot/dev 

2. chroot /mnt/tmproot/ /bin/bash 

3. Fixup /etc/fstab (particularly, device names, logical volume paths, filesystem labels,  etc., need to be changed to match your target system’s config). 

4. Fixup /etc/grub.conf (or wherever your grub.conf is located) (particularly, device names, logical volume paths, filesystem labels,  etc., need to be changed to match your target system’s config) 

5. On RHEL/CentOS source/target, fixup /etc/sysconfig/network-scripts/ifcfg-eth0 for new mac address (I comment out the HWADDR and MACADDR ; else when it comes up, the target will recognize that the NIC’s MAC has changed, and reconfigure/re-ip the interface.  I presume you want the target to come up at the same IP as the source, and that the source will be powered off. 

6. Configure the Target (chroot) to know to use the right scsi adapters 

1. If this is an import to a paravirt Linux Xen domain, then add this to /etc/modprobe.conf: 

1. alias eth0 xennet 

2. alias scsi_hostadapter xenblk 

2. else, add the appropriate scsi and eth0 drivers.   

1. For vmware, by default: 

1. alias eth0 e1000 

2. alias scsi_hostadapter mptbase 

3. alias scsi_hostadapter1 mptspi 

2. or if this is an older RHEL system: 

3. alias eth0 e1000 

4. alias scsi_hostadapter mptbase 

5. alias scsi_hostadapter1 mptscsih 

3. …or whatever the appropriate driver is for your target (note: in generic HVM, such as RHEL3, ata_piix driver will be used) 

7. Install the appropriate kernel for your target, if needed: 

1. (If I recall correctly, this was true for CentOS 5.x, kernel 2.6.18, and earlier...) If the target is a Xen (Citrix or RHEL/CentOS Xen) paravirtualized vm but the source was not, you need to install the xen-optimized kernel: 

1. yum install kernel-xen 

2. If the target is a Vmware vm or physical host, but the source was a Xen-optimized kernel, you need to install the non-Xen kernel: 

1. yum install kernel 

8. Now, verify that /etc/grub (inside the chroot) is pointing at the correct kernel 

9. Recreate the initrd for your target so that it will have the scsi/raid drivers, device nodes, and initial mounts it needs to boot: 

1. For PVM Xen guest target: 

1. mkinitrd –f –v --with=xenblk --force-scsi-probe --force-lvm-probe /boot/initrd-(kernel version) (kernel version) 

2. For recent HVM Xen guest target: 

1. mkinitrd –f –v --with=xenblk --force-scsi-probe --with=ata_piix --force-lvm-probe  /boot/initrd-(kernel version) (kernel version) 

3. For legacy (e.g., CentOS 3.x) HVM Xen guest target: 

1. mkinitrd –f –v --force-scsi-probe --with=ata_piix --with=dm-mod /boot/initrd-(kernel version) (kernel version) 

4. ..or add “--with=” for your given scsi/raid module name.  For VMware, these are probably “mptspi” and “mptbase” 

5. For Ubuntu/Debian variants: 

1. update-initramfs -c -v -k 2.6.32-24-server  (specify the correct kernel version) 

10. exit (back out of chroot) 

9. Install the bootloader (grub) in the target platform 

1. CentOS: grub-install --recheck  --root-directory=/mnt/tmproot --no-floppy /dev/sda (use hda instead of sda if you had to boot an old rescue cd that uses IDE drivers instead of SCSI for SATA; use another device name as appropriate, e.g., for special RAID devices) 

10. Final fixups and reboot target: 

1. If you have not previously shut down the source application/platform, then you may want to quiesce applications/services on the source and run a final re-synce, this time syncing only the application data (e.g., /var/lib/mysql)).  This will ensure no data loss with a minimal outage window (presuming the target system comes up cleanly.) 

2. If you use selinux: 

1. touch /mnt/tmproot/.autorelabel 

3. To prevent a fsck upon reboot: 

1. rm /mnt/tmproot/.autofsck 

4. Unmount all filesystems in the sandbox 

1. umount /mnt/tmproot/*  

2. (unmount any other filesystems you have in the tmproot) 

3. umount /mnt/tmproot/ 

5. shutdown -h 

6. If the target is a Xen VM, then detach the hard disks from the HVM vm and re-attach it to the vm where this will live permanently. 

11. If you’ve been complete, the target it should boot up. 

1. You should be ready to kill the network on the source platform, unplug it, etc., once you’ve confirmed the target is up. ...namely, once the target gets to the point of starting services, this means that we're past the trickiest parts (initrd and switchroot), and I kill the network on the source, then bring it down gracefully.  I unplug power an re-label it ("HOSTNAME-OLD" or some such) so that it doesn't accidentally get turned on. 

12. Troublshooting the target if it panics on boot-up 

1. Be sure to remove the “quiet” and “graphical” options from the kernel line in the grub.conf, or you may not be able to see what’s happening. 

2. If grub doesn’t do anything, your bootloader is probably not installed correctly or points to an invalid/incorrect kernel and/or initrd.  If you need to edit the grub file from within the xen host, run xe-edit-bootloader.   

3. In grub.conf, be sure the kernel and initrd lines are correct, relative to the root of the filesystem on which /boot resides. Thus, if /dev/sda1 is /boot and /dev/sda2 is /, then your path will be (0,0)/kernel-….  But if /dev/sda1 is /, then the path will be (0,0)/boot/kernel-…. 

4. If you get a panic after switchroot, you either have a wrong value for / in /etc/fstab, in /etc/grub.conf, or you didn’t get the right scsi/raid driver in the initrd, you need to recreate the chroot environment and re-run the mkinitrd with the correct parameters.  mkinitrd seems to look at /etc/mtab and/or /etc/fstab ; since this occurs in the chroot, you may have to edit those files in the chroot so that mkinitrd knows to pull in certain other drivers, mount paths or filesystem labels, etc. 

13. Other Finalizing steps 

1. Configure vm to autostart. 

2. Load xen tools , vmware tools, or whatever is appropriate to the target. 

3. Fix-up specific to certain OS's: 

1. RHEL 4 HVM vm: if you get radical clock skew (ntpd won't even run as the clock slows and speeds up too much), then add "notsr" to the kernel boot line in /etc/grub.conf, and set the vm to use only 1 cpu. 

2. Ubuntu 9.04 Jaunty on Xen - you may not have a functional console upon boot-up; you have to configure the vm to set up a login on the console.  Normally, this is done in /etc/inittab, but Jaunty does not have an inttab.  Instead, add to the bootloader "kernel" line the string "console=hvc0 xencons=tty" (in XenCenter, set the boot properties to that string); then create the file /etc/event.d/hvc0 with this: 

start on stopped rc2 start on stopped rc3 start on stopped rc4 start on stopped rc5  stop on runlevel 0 stop on runlevel 1 stop on runlevel 6  respawn exec /sbin/getty 38400 hvc0 

Conclusion 

As you can see, it's not necessarily a quick and easy recipe; but it should let you see the main steps that need to happen.  But you probably have just a few standardized types of source platforms and target platforms, so it should be very reproducible once you adapt this to your platforms. 

Enjoy!