Storage Best Practices

Comments welcome.  These are just some things I've decided work best and save hassles:

General Storage

1. Compression should be used on the back-end storage, not in the VM or host that is mounting the storage.
2. Data should be grouped in volumes based on general purpose/function, performance requirements, backup requirements, and any isolation requirements. For example:
   1. DB logs should be on a separate hosting aggregate/raid-set from the databases that they help to protect.
   2. General business data should not be on the same volume as Financials data, because they have distinct back-up requirements.
   3. File share data and VM data should not be on the same underlying volume (though they may live on the same RAID set / aggregate.
3. NAS/SAN volumes and shares should be named to say as briefly as possible what they actually are:
   1. Scratch/temp data should always be named such that the use can see that it is "scratch" data. e.g., "localscratch0", "nas_scratch0".
   2. vi_eng_0  (virtual infrastructure for Engineering's VM's, number 0)
   3. volume IT_Software shared out as \\corp.mycompany.com\LS\IT\Software.
4. Security
   1. Permissions should be set for at least the share and directly-contained folders using domain-local security groups (using nested groups) that are specific to just that share and those folders.
   2. "Everyone" should almost never be used.
   3. "Deny" permissions should almost never be used; instead, create a group that includes only the desired people/roles.
5. Scratch data should not be backed up.  Take every relevant opportunity to remind users of that.
6. Linux
   1. LVM
      1. LVM should be used where possible -- always for the OS; for data disks it may be omitted for cases where separate LUNs are presented for data and there is only one filesystem per LUN.
      2. The OS should use a different Volume group from major application data. (simple LAMP boxes with a single disk device may us all on a single volume group; systems with more disk devices, or where one may wish to restore data LUNs from snapshot should have a separate VG_Data including all physical volumes holding application data
      3. All partitions on a single disk device should only belong to the same Volume Group. (this makes re-assembling a broken system easier... if the data volume group has a physical volume missing/broken or restored from snapshot, the system can still boot if the OS volume group is intact; then the data volume group can be re-assembled.).
      4. Logical volumes should be named with "LV" plus the mount path, substituting "_" for "/"; LV_root for the root "/" filesystem, and LV_swap for the swap filesystem.
      5. /boot should not be on a logical volume.
      6. Logical volumes should always be thick-provisioned
      7. Resizing
         1. Before any resize operation, always back up the data to external storage first!
         2. When growing a logical volume, always grow the logical volume first, using round "g" size, then resize the filesystem (don't specify a size, where possible, and let resize detect the new size).  If the hosting physical volume is to be grown, grow the hosting partition/LUN and then pvextend before growing the LUN.
         3. When shrinking a logical volume, always shrink the filesystem first, using roung "g" size, then resize the logical volume, also using round "g" size specification.  If the hosting partition/LUN is also to be shrunk, then it should be shrunk last using round "g" size specification.
   2. Filesystems should be labeled with the mount path, e.g. "/var/log" where it fits; if the path is too long, then the last unique part of the path should be used, e.g. "postgres-backups".
7. Windows
   1. Application Data for major applications and those where one may wish to restore data LUNs from snapshot separately from the OS should place application data on separate disk devices (not just separate partitions)
   2. Filesystems should be labeled based on mount/drive path and purpose, e.g. "C_OS", "D_Data", "F_Logs", "G_MailboxDB00", "C_Appdata_logs"

General SAN

1. LUNs should always be masked on the SAN target device to only allow access from only those initiators who require access to that LUN.
2. LUNs should be formatted using the  and mounted (fstab) mapped using the multipath device
3. Multipathing
   1. Multipathing should be used where possible, with each target (redundant initiators, switches/paths, and targets)
   2. Redundant targets should be used, as well: Initiator A should connect on subnet/switch A to the target LUN on SAN head A; initiator B should connect on subnet/switch B to the target LUN through SAN head B.
   3. Active-active with round-robin load balancing should be used.
   4. LUNs should generally be thinly provisioned (except for LUNs hosting essential databases); the monitoring system (Zabbix) should be configured to alert when the hosting aggregate is at 80, 90, and 95% of capacity..

FC

1. FC switches should use WWN-based zoning
2. Any host OS installation should include *unplugging* the HBA so that the OS does not wipe all visible LUNs. (If the host OS is being installed for boot-from-SAN, then all LUN masks and target attributes should be triple-checked to avoid inadvertent destruction of data.)
3. switches in separate paths should be managed separately. (Combining them into a single management domain would combine them into a single fault domain for administrative errors).
4. WWN's should, where be possible, be aliased all storage devices (switches and SAN) to include the hostname (and optionally a function) such that they can be easily identified, referenced, and so that all related zonings/mappings can be updated only by updating the alias itself. (For example, if an HBA must be replaced, that would invalidate all WWN-based mappings; but since we used an alias for all mappings, we just update the single alias definition that included that WWN.

iSCSI

1. all IQN's should always be typed as all lower-case.
2. Initiator IQN names should be:
   1. iqn.2014-01.local:hostname[-boot]
   2. The - is optional, and is only needed if there are several different initiators on a single host
   3. No iscsi traffic should be routed.  In other words, any given initiator should be connecting to a target on the same subnet.
3. Targets should be mapped using IP address, not Host name.
4. Jumbo frames (9000-Byte) should be used where possible.

parallel xargs and Xen/VMware / performance / storage performance data gathering

I have a problem where certain virtual machines drop offline for brief periods of time.

Why? I don't know, but I suspect an I/O related issue (NFS / network hiccup). There's nothing in the logs.

How can I test this? Well, the built-in tools (sar/sysstat reports) Vcenter, or XenCenter don't keep historical data down to the second, so I will miss any hiccup that lasted only a few seconds. Furthermore, long-term (minute, 5 minute, etc) stats/graphs tend to average out the spikes, so I won't necessarily see a spike of a fine resolution (seconds), unless it's enough to register on a scale of a time resolution an order of magnitude greater.

I know that when local disk I/O is blocked, then iostat shows the cpu spending 100% in iowait, and an individual device queue maxes out (disk device may also go to 100% utilization, with no transactions or data actually read/written). So, how to get/keep historical data to compare against my Nagios alerts? How about gathering data with 1 second granularity, and then when an alert comes through, compare that timestamp against what I've found in the logs.

how to do that in a quick, parallel, automated fashion without manually logging in to 50 servers? Parallel execution with xargs and ssh:

First, distribute a public key for ssh to use, so you don't have to manually type the password on each box. Then,

echo xenhost1 xenhost2 xenhost3 vm1 vm2 vm3 vm4 vm5 | xargs -P 100 -d" " --replace="TARGET" bash -c 'ssh -i /root/.ssh/ssh_keyfile TARGET "iostat -xnt 1 || iostat -nt 1" > TARGET.log'

(the || is because Citrix XenServer 5.5.0 uses a different iostat version and syntax than the other RHEL/CentOS systems I have)

When I get an that "vm xyz is unavailable", I can look at the log file for each of the xen hosts, the nfs server, and the vm itself to see whether there was an I/O, cpu load, or other problem.

Also, once I have all the data, I can grep out lines that I care about and import them into excel to have trending data with 1 second granularity, e.g.:

egrep "nfsserver:/export/home" xenhost1.log > foo.csv

Of course, you could use this xargs and logfile method to snapshot many aspects of a system over time to troubleshoot a problem or for your own records (what process list, network connections, etc.), things that may not be caught by syslogd, dmesg, cacti, etc. I did something like this to capture total IOPS across our enterprise when we were planning to consolidate a bunch of local disk to a central NAS+SAN. Had to capture iops per-server over a period of a week. It made for a 30MB spreadsheet, but in the end I know my numbers were accurate for the peaks and valleys of our i/o load.

CentOS NFS permission denied on mount

NFS mount results in a permission denied error. Check the export permissions, and those are right.

The solution is to add (on the exporting host):

nfsd /proc/fs/nfsd nfsd auto,defaults 0 0

to /etc/fstab and then type:

$mount -a

on the client.

not sure why this error exists.

Solution found here: http://www.centos.org/modules/newbb/viewtopic.php?forum=28&topic_id=6891&viewmode=thread