Get SSL Certificate Vitals in Linux

This script will let you programmatically get a certificate start date, number of days remaining, and certificate hash, suitable for example for automated checking for expired or changed certificates, as with Zabbix:


function printHelpTextd
        echo "######################################################################"
        echo "#                                                                    #"
        echo "#  This script takes these parameters, in this order:                #"
        echo "#  1. check type, one of: certstartdate, certdaysleft, or certhash.  #"
        echo "#  2. Host connection target (IP address or host name (fqdn)).       #"
        echo "#  3. TCP port number to connect to.                                 #"
        echo "#                                                                    #"
        echo "#  This script returns, depending on the check type, one of:         #"
        echo "#  - certstartdate: a text string of the cert start date             #"
        echo "#  - certdaysleft: an integer of the number of days until the cert   #"
        echo "#    expiration; if the cert has expired, then a negative number.    #"
        echo "#  - certhash: a hash of the cert, useful for detecting changes.     #"
        echo "#                                                                    #"
        echo "######################################################################"


#  Function getCertStartDate
#  parameters (ordered)
#    * Host connection target (IP address or host name (fqdn)).
#    * TCP port number to connect to.
#  returns a text string of the certificate start date.
function getCertStartDate
        startdate=`echo quit | openssl s_client -host $host -port $port 2>/dev/null | awk '/BEGIN/{s=x}{s=s$0"\n"}/END CERTIFICATE-----/{print s}' 2>/dev/null | openssl x509 -noout -dates 2>/dev/null | head -n 1 | cut -d "=" -f 2- | awk -F " " '{ print $1" "$2" "$4" "$3" "$5 }'`
        echo $startdate

#  Function getCertDaysLeft
#  parameters (ordered)
#    * Host connection target (IP address or host name (fqdn)).
#    * TCP port number to connect to.
#  returns a number of days remaining
function getCertDaysLeft
        enddate=`echo quit | openssl s_client -host $host -port $port 2>/dev/null | awk '/BEGIN/{s=x}{s=s$0"\n"}/END CERTIFICATE-----/{print s}' | openssl x509 -noout -dates 2>/dev/null | tail -n 1 | cut -d "=" -f 2-`
        formattedenddate=`echo $enddate | awk -F " " '{ print $1" "$2" "$4" "$3" "$5 }'`
        enddateseconds=`date -d "$formattedenddate" +%s`
        # expiration date minus todays date = the number of days left (in seconds)
        secondsleft=$(expr $enddateseconds - $(date +%s))
        daysleft=$(expr $secondsleft / 86400)
        echo $daysleft

#  Function getCertHash
#  parameters (ordered)
#    * Host connection target (IP address or host name (fqdn)).
#    * TCP port number to connect to.
#  returns the hash of the cert, as a string
function getCertHash
        hash=`echo quit | openssl s_client -host $host -port $port 2>/dev/null | awk '/BEGIN/{s=x}{s=s$0"\n"}/END CERTIFICATE-----/{print s}' | openssl x509 -noout -hash 2>/dev/null`
        echo $hash

if [ "$#" -ne 3 ]; then
        echo "ERROR: Illegal number of parameters."
        exit $ERR_BADNUMPARAMS
}; else
        case $Operation in
                getCertStartDate $TargetHost $TargetPort
                getCertDaysLeft $TargetHost $TargetPort
                getCertHash $TargetHost $TargetPort
                        echo "ERROR: Bad check type."
                        exit $ERR_BADCHECKTYPE
}; fi


Syslog on NetApp

Data Ontap has the ability to send system log messages to an industry standard syslog server (see https://library.netapp.com/ecmdocs/ECMP1196979/html/man5/na_syslog.conf.5.html)

To cause your Netapp to start logging to a syslog server named "logs.mycompany.com", you would use the wrfile to (over)write the syslog configuration file, directly from the console; leave a blank line at the end, and use ctrl-c to conclude the edit:

mynetapp> wrfile /vol/vol0/etc/syslog.conf
*.info    /dev/console
*.info    /etc/messages
*.info    @logs.mycompany.com
auth.*    @logs.mycompany.com
cmdsaudit.auditlog    @logs.mycompany.com


(You should then see "syslogd restarted" shortly, when the NetApp detects the config file change.)

The "cmdsaudit.auditlog" line causes all console commands that are entered to also be logged to syslog -- thus, you have a record of who did what, when.


View package changelog prior to updating

On Redhat/CentOS systems, have a look at the Yum "Changelog" plugin, "yum-plugin-changelog" or "yum-changelog" package

See “man yum-changelog”.

This lets you see what has change on a package between the version that is already installed and the latest available version.

Go to a system, and run “yum  update --changelog”

Or, for a narrower view, try: “yum update kernel --changelog”

We should use this when we patch, to understand what is changing and to scope potential impact, rather than to simply “patch and pray”.  Alone it is not enough (release notes should also be reviewed where available), but it is a good start and may help in flagging potential problems.

In particular, we should look at this on critical infrastructure servers, especially for those that get their software from external repositories where the software changes may be more impactful than standard RedHat/CentOS packages (which generally remain on the same version, with only back-ported bug and security fixes).


Make OpenLDAP work with SSL/TLS

These instructions are for CentOS 7:

Many applications require the use of LDAP over TLS. This is also a best-practice.

However, the Linux / openldap libraries and clients now perform certificate validation.  Many private enterprises will not validate "out of the box".  This is how we assist it.

Take a company, "mycompany.com", with two Active Directory domain controllers, "dc1" and "dc2", and their own internal CA:

openssl s_client -host dc1.mycompany.com -port 636 | awk '/BEGIN/{s=x}{s=s$0"\n"}/
END CERTIFICATE-----/{print s}'> /etc/openldap/certs/DC1.MYCOMPANY.COM.crt

openssl s_client -host dc2.mycompany.com -port 636 | awk '/BEGIN/{s=x}{s=s$0"\n"}/
END CERTIFICATE-----/{print s}'> /etc/openldap/certs/DC2.MYCOMPANY.COM.crt

openssl s_client -showcerts -host dc1.mycompany.com -port 636
(copy the second certificate, between “BEGIN” and “END”, for the MyCompany CA into /etc/openldap/certs/MYCOMPANY_CA.crt)

cacertdir_rehash /etc/openldap/certs
This last command creates sym links named with the cert hash and pointing to the cert file for each cert in that directory.

CAVEAT EMPTOR: the certs will expire, and will (typically, in Active Directory) be automatically renewed; but your openldap clients/libraries will NOT be able to validate the renewed certs. You'll simply find that LDAP does not work anymore until you repeat this process to update the locally stored certs.


XenServer boot from iSCSI

UPDATE: XenServer boot from iSCSI (at least with iBFT) is just plain awful. Don't bother.  Multipath does not work. Too many hacks to try to get things to work. It is not supported, anyway, and it will probably break every time you do an upgrade.  Plus, the NICs that you use for iSCSI boot will be unusable for any other purpose.  This is aspect of XenServer is very immature and not robust.

  1. Credits (Special thanks for pointers from:)
    1. https://www.krystalmods.com/index.php?title=xenserver-6-supports-iscsi-boot-undocumented-feature&more=1&c=1&tb=1&pb=1
    2. http://serverfault.com/questions/598773/install-xenserver-on-iscsi-target
    3. http://serverfault.com/questions/431864/boot-from-iscsi-how-does-it-work
  2. Notes/Warnings
    1. This was originally writtenf or XenServer 6.2.
    2. The NIC that is used for iSCSI boot will not be available for use for any other purposes (admin network, regular storage network, VM network, etc.)
    3. This is with Intel I350 Gbit NICs, as on a Supermicro X9DRT motherboard, which uses the ibft module.
    4. Once booted, you're in a Busybox environment... some commands will be limited or may not work the way in which you are accustomed on a full Linux system. Consider the information here for more troubleshooting steps to confirm that iSCSI works (LUNs are reachable, access is granted, etc.)
    5. As a best practice, use all lower-case named target and initiator names (some firmwares may silently convert case).
    6. WARNING: If you do not have your LUNs properly masked, do not specify the installation target correctly, and so forth, you may destroy your data. Know what you are doing, and use at your own risk.
  3. Connect up the Intel NICs
  4. Enable the Option ROM in the BIOS and add to the boot order; configure the iSCSI boot-enabled NICs in the NIC ROM set-up (perhaps ctrl-D when prompted after POST and prior to OS boot); You'll want to configure networking, initiator and target name; consider following these SAN best practices
  5. Configure the iscsi target to allow access from this initiator
  6. Boot off the CD and enter the command shell:
    1. Insert the XenServer 6.2 CD and power on the computer.  At the "boot: " prompt, type "shell" and press enter.
  7. Prepare iscsi:
    1. echo "InitiatorName=iqn.2014-01.local:hostname" > /etc/iscsi/initiatorname.iscsi (replace the initiator name and/or hostname as appropriate)
    2. echo "node.session.initial_logon_retry_max = 60" >> /etc/iscsi/iscsid.conf
    3. modprobe iscsi_ibft
    4. modprobe scsi_transport_iscsi
    5. modprobe iscsi_tcp
    6. iscsid -c /etc/iscsi/iscsid.conf -i /etc/iscsi/initiatorname.iscsi -f &
  8. set up multipath (if you have multiple paths)
    1. modprobe dm-multipath
  9. Start the installer
    1. /opt/xensource/installer/init --use_ibft
    2. (use --mpath if you have set up multipath)
    3. (use --device_mapper_multipath=true)
    4. (use --network_device=eth0 , or the correct iSCSI network device, if needed)
Multipath doesn't seem to "just work"... at one point, I had to do this, because if I don't, then installer runs without multipath, but when the init does switchroot, the OS hangs and complains that "a device that was previously mounted without multipath is now mounted with multipath."
  1. Go to http://debaan.blogspot.com/2014/10/iscsi-stuff-in-busybox-on-linux.html and follow those steps to get logged in to both LUNs and get them seen with multipath.
  2. Start the installer as before.  Wait until the last step, when it says that it is ready to reboot.
  3. Alt+F2 to get to a command shell (again).
  4. Set up a chroot:
    1. mount -o bind /dev /tmp/root/dev
    2. mount -o bind /sys /tmp/root/sys
    3. mount -o bind /proc /tmp/root/proc
    4. chroot /tmp/root /bin/bash
    5. cd /boot
    6. re-run the
      iscsiadm -m discovery -t sendtargets -p commands to make the chroot environment aware of the targets
    7. mkinitrd --with=dm-multipath --with=iscsi_ibft --with=scsi_transport_iscsi -f /boot/initrd-

iSCSI stuff in Busybox on Linux

  1. Notes/Warnings
    1. This works with Intel I350 Gbit NICs, as on a Supermicro X9DRT motherboard, which uses the igb (1Gbit) or ixgbe (10Gbit) module as the NIC device driver, and the iscsi_ibft module for iscsi boot. You'll have to change the module names if using Broadcom or other chips.
    2. Once booted, you're in a Busybox environment... some commands will be limited or may not work the way in which you are accustomed on a full Linux system.
    3. As a best practice, use all lower-case named target and initiator names (some firmwares may silently convert case).
    4. WARNING: If you do not have your LUNs properly masked, do not specify the installation target correctly, and so forth, you may destroy your data. Know what you are doing, and use at your own risk.
  2. Connect up the Intel NICs
  3. Configure the iscsi target to allow access from this initiator (hint: this is done on your SAN), and mask out other LUNs to be NOT visible to this initiator
  4. Boot off the CD and enter the busybox command environment
  5. Manually configure the ip address and enable the interfaces, and verify connectivity
    1. ifconfig eth0 inet netmask up
    2. route add default gw
    3. ping; ping www.google.com
    4. At this point you can enable ssh to access this from a different computer over the network, if you need to:
      1. ssh-keygen -f /etc/ssh_host_rsa_key -t rsa  (use an empty passphrase)
      2. ssh-keygen -f /etc/ssh_host_dsa_key -t dsa (use an empty passphrase)
      3. /usr/sbin/sshd
      4. echo "root:supersecret" | chpasswd
      5. (now, copy the hashed password for root from /etc/passwd to /etc/shadow
  6. Prepare iscsi:
    1. mkdir /etc/iscsi
    2. echo "InitiatorName=iqn.2014-01.local:hostname" > /etc/iscsi/initiatorname.iscsi (replace the initiator name and/or hostname as appropriate)
    3. echo "node.startup = automatic" > /etc/iscsi/iscsid.conf
    4. echo "node.session.initial_logon_retry_max = 60" >> /etc/iscsi/iscsid.conf
    5. modprobe iscsi_ibft
    6. modprobe scsi_transport_iscsi
    7. modprobe iscsi_tcp
    8. iscsid -c /etc/iscsi/iscsid.conf -i /etc/iscsi/initiatorname.iscsi -f &
  7. detect iscsi targets (where is the IP address of the iSCSI SAN target) (only do this if you are not going to use the ibft boot stuff from an OS installer)
    1. iscsiadm -m discovery -t sendtargets -p
    2. iscsiadm -m node -l
  8. set up multipath (if you have multiple paths)
    1. modprobe dm-multipath
    2. multipath -l -r
    3. multipath -l -r (verify that the paths exist and are recognized...)
  9. At this point, you can use the multipath device to access your iSCSI LUNs as needed.
  10. If you want to run an OS installer, at this point start the installer
    1. (in CentOS or XenServer installation media, append --use_ibft)
    2. (use --mpath if you have set up multipath)
    3. (use --network_device=eth0 , or the correct iSCSI network device, if needed)


Storage Best Practices

Comments welcome.  These are just some things I've decided work best and save hassles:

General Storage

  1. Compression should be used on the back-end storage, not in the VM or host that is mounting the storage.
  2. Data should be grouped in volumes based on general purpose/function, performance requirements, backup requirements, and any isolation requirements. For example:
    1. DB logs should be on a separate hosting aggregate/raid-set from the databases that they help to protect.
    2. General business data should not be on the same volume as Financials data, because they have distinct back-up requirements.
    3. File share data and VM data should not be on the same underlying volume (though they may live on the same RAID set / aggregate.
  3. NAS/SAN volumes and shares should be named to say as briefly as possible what they actually are:
    1. Scratch/temp data should always be named such that the use can see that it is "scratch" data. e.g., "localscratch0", "nas_scratch0".
    2. vi_eng_0  (virtual infrastructure for Engineering's VM's, number 0)
    3. volume IT_Software shared out as \\corp.mycompany.com\LS\IT\Software.
  4. Security
    1. Permissions should be set for at least the share and directly-contained folders using domain-local security groups (using nested groups) that are specific to just that share and those folders.
    2. "Everyone" should almost never be used.
    3. "Deny" permissions should almost never be used; instead, create a group that includes only the desired people/roles.
  5. Scratch data should not be backed up.  Take every relevant opportunity to remind users of that.
  6. Linux
    1. LVM
      1. LVM should be used where possible -- always for the OS; for data disks it may be omitted for cases where separate LUNs are presented for data and there is only one filesystem per LUN.
      2. The OS should use a different Volume group from major application data. (simple LAMP boxes with a single disk device may us all on a single volume group; systems with more disk devices, or where one may wish to restore data LUNs from snapshot should have a separate VG_Data including all physical volumes holding application data
      3. All partitions on a single disk device should only belong to the same Volume Group. (this makes re-assembling a broken system easier... if the data volume group has a physical volume missing/broken or restored from snapshot, the system can still boot if the OS volume group is intact; then the data volume group can be re-assembled.).
      4. Logical volumes should be named with "LV" plus the mount path, substituting "_" for "/"; LV_root for the root "/" filesystem, and LV_swap for the swap filesystem.
      5. /boot should not be on a logical volume.
      6. Logical volumes should always be thick-provisioned
      7. Resizing
        1. Before any resize operation, always back up the data to external storage first!
        2. When growing a logical volume, always grow the logical volume first, using round "g" size, then resize the filesystem (don't specify a size, where possible, and let resize detect the new size).  If the hosting physical volume is to be grown, grow the hosting partition/LUN and then pvextend before growing the LUN.
        3. When shrinking a logical volume, always shrink the filesystem first, using roung "g" size, then resize the logical volume, also using round "g" size specification.  If the hosting partition/LUN is also to be shrunk, then it should be shrunk last using round "g" size specification.
    2. Filesystems should be labeled with the mount path, e.g. "/var/log" where it fits; if the path is too long, then the last unique part of the path should be used, e.g. "postgres-backups".
  7. Windows
    1. Application Data for major applications and those where one may wish to restore data LUNs from snapshot separately from the OS should place application data on separate disk devices (not just separate partitions)
    2. Filesystems should be labeled based on mount/drive path and purpose, e.g. "C_OS", "D_Data", "F_Logs", "G_MailboxDB00", "C_Appdata_logs"

General SAN

  1. LUNs should always be masked on the SAN target device to only allow access from only those initiators who require access to that LUN.
  2. LUNs should be formatted using the  and mounted (fstab) mapped using the multipath device
  3. Multipathing
    1. Multipathing should be used where possible, with each target (redundant initiators, switches/paths, and targets)
    2. Redundant targets should be used, as well: Initiator A should connect on subnet/switch A to the target LUN on SAN head A; initiator B should connect on subnet/switch B to the target LUN through SAN head B.
    3. Active-active with round-robin load balancing should be used.
    4. LUNs should generally be thinly provisioned (except for LUNs hosting essential databases); the monitoring system (Zabbix) should be configured to alert when the hosting aggregate is at 80, 90, and 95% of capacity..


  1. FC switches should use WWN-based zoning
  2. Any host OS installation should include *unplugging* the HBA so that the OS does not wipe all visible LUNs. (If the host OS is being installed for boot-from-SAN, then all LUN masks and target attributes should be triple-checked to avoid inadvertent destruction of data.)
  3. switches in separate paths should be managed separately. (Combining them into a single management domain would combine them into a single fault domain for administrative errors).
  4. WWN's should, where be possible, be aliased all storage devices (switches and SAN) to include the hostname (and optionally a function) such that they can be easily identified, referenced, and so that all related zonings/mappings can be updated only by updating the alias itself. (For example, if an HBA must be replaced, that would invalidate all WWN-based mappings; but since we used an alias for all mappings, we just update the single alias definition that included that WWN.


  1. all IQN's should always be typed as all lower-case.
  2. Initiator IQN names should be:
    1. iqn.2014-01.local:hostname[-boot]
    2. The - is optional, and is only needed if there are several different initiators on a single host
    3. No iscsi traffic should be routed.  In other words, any given initiator should be connecting to a target on the same subnet.
  3. Targets should be mapped using IP address, not Host name.
  4. Jumbo frames (9000-Byte) should be used where possible.


Macbook Pro fails

  1. Turning the thing on after initial setup. I go to install Firefox.  Firefox opens the first time. ...A dialogue box pops up to "do you want to make Firefox your default browser?"  Immediately, a "tour firefox" box pops up in front of that.  I can't do anything, because the first box must be clicked before the second can be acknowledged.  Can't move either, click on either, etc.  Even though Firefox popped up the dialogues, I blame OS X because I should be able to at least switch windows.  In the end, I could only maximize the main window, which allowed me to click on the "behind" default browser window.
  2. Plug and play fail -- plugged in a Microsoft Natural keyboard. ...OS X failed to detect it correctly.  I have to run through steps to get it to detect it correctly. ...and even then, the ctrl, windows/super, and alt keys are all mixed up.
  3. Keyboard key mapping fail
    1. The "end" key does not move the cursor to the end of a line.
    2. The "windows" key / super key acts as the control key; the control key is something else. 
  4. SMB DFS namespace does not work... finder will open it up, but browsing to a linked folder stalls indefinitely -- no timeout, no error, no ability to navigate DFS. ...Linux has had this capability for a decade or more.


Authentification is not a word

Whether one is referring to...

authentication - the validation of credentials (i.e., "Yes, you are who you claim to be")
authorization - the validation of whether access should be granted based on that identity (based on role, group membership, policy, ACL, etc.) (i.e., "Yes, that user is allowed to access that resource")

...the word "authentiFIcation" never comes into play... nor authentify, authentificated, etc.

Identification / identify is a valid word, mind you.

Just saying...


List and delete zomble NetApp snapshots left from backups

Backup software, such as CommVault Simpana, may sometimes leave zombie snapshots -- they're not in use, but will never be cleaned up.  ...the actual space consumed by these snaps increases over time as the delta between that data and the current live copy increases.

In your backup software, you probably have the option to set the snapshot name prefix, which makes it easy enough to distinguish these stale snaps from your regularly scheduled ones.  However, if you have many volumes, it can be tedious to go through and clean those up.

Here's a script that will let you crawl your (Ontap v8) NetApp to list bogus snapshots and optionally delete them:


# snap_cleanup
# Written by Lane Bryson on 2014/01. Provided AS IS, no warranties
# expressed or implied: USE AT YOUR OWN RISK!
# This script is to look for snaps that are left over from backups, that
# are no longer in use, and delete them.
# This script requires two parameters:
# snap_cleanup     (list or delete all stale snaps on the specified servers)

SnapshotString="snapshot_for_backup" # This is the prefix for backup-related snaps

# This function receives a volume name as a parameter, and returns
# the number of snaps that are elligible for deletion, defined by being:
#               1. Having a certain string in the snapshot name;
#               2. Not marked as "busy"
function CountStaleSnaps ()
        local VolToCheck=$1
        local StaleSnaps=`$SshCmd "snap list $VolToCheck" | grep $SnapshotString | grep -v "busy" | wc -l`
        if [ $? -ne 0 ]; then
                echo "getting count of elligible snapshots returned an error"
                exit 1
        } else
                echo $StaleSnaps
        } fi

# Function GetStaleSnapNames
# This function creates, given a volume name, an array of snapshots that are
# candidates for deletion.
# Parameters
#       1. volume to check
function GetStaleSnapNames ()
        local Volume=$1
        StaleSnapNames=( $($SshCmd "snap list $Volume" | grep $SnapshotString| grep -v busy | cut -c 39- | cut -f 1 -d " ") )
        if [ $? -ne 0 ]; then
                echo "getting names of elligible snapshots returned an error"
                exit 1
        } else
                printf -- '%s\n' "${StaleSnapNames[@]}"
        } fi

# Function DelStaleSnaps
# This function deletes stale snaps for the Volume name passed to it.
# Parameters
#       1. volume
#  2. snapshot_name
function DelStaleSnaps ()
        local Volume=$1
        local SnapToDelete=$2
        $SshCmd "snap delete $Volume $SnapToDelete"
        if [ $? -ne 0 ]; then
                echo "error deleting snapshot $Volume:$SnapToDelete"
                exit 1
        } else
                echo "successfully deleted snapshot $Volume:$SnapToDelete"
        } fi

#for Volume in `$SshCmd "vol status -b" | cut -f 1 -d " " | egrep -v "Volume|-----"`; do SNAPS=`$SshCmd "snap list $Volume" | grep $SnapshotString | grep -v busy| wc -l`; echo $target: $SNAPS; done

# Parse command line parameters
if [ $# -ne 2 ] || [ "$1" = "--help" ] || [ "$1" = "-help" ] || [ "$1" = "help" ] || [ "$1" = "-h" ]; then
        echo "This script requires exactly two parameters:"
        echo "  1. a function - list, delete, or help"
        echo "  2. a NetApp host name"
        exit 1
}; fi
case $2 in
                echo "ERROR: you must specify as a second parameter a host name on which you want to delete snapshots."
case $1 in
                echo "ERROR: invalid operation specified on command line.  Please specify either 'list' or 'delete' followed by the servername on which you want to delete the snapshots."
                exit 1

SshCmd="/usr/bin/ssh -i $SshIdentityFile $NasUser@$NasName"

VolumesToCheck=( `$SshCmd "vol status -b" | cut -f 1 -d " " | egrep -v "Volume|-----"` )

for CurrentVol in ${VolumesToCheck[@]}; do
        echo -n "checking $NasName:$CurrentVol... "
        StaleSnaps=`CountStaleSnaps $CurrentVol`
        echo $StaleSnaps       
        if [ $Operation = list ]; then
                if [ $StaleSnaps -ne 0 ]; then
                        GetStaleSnapNames $CurrentVol |  awk '{ print "    " $1 }'
        elif [ $Operation = delete ]; then
                GetStaleSnapNames $CurrentVol
                ArrayOfSnaps=( $(GetStaleSnapNames $CurrentVol) )
                #echo ArrayOfSnaps is ${ArrayOfSnaps[@]}
                #echo "ArrayOfSnaps[0] is ${ArrayOfSnaps[0]}"
                #echo "ArrayOfSnaps[1] is ${ArrayOfSnaps[1]}"
                #echo "ArrayOfSnaps[2] is ${ArrayOfSnaps[2]}"
                for TargetSnap in `printf -- '%s\n' "${ArrayOfSnaps[@]}"`; do
                                DelStaleSnaps $CurrentVol $TargetSnap

}; done