2008/10/29
ldapsearch against AD (or other directory)
ldapsearch -b "dc=mydomain,dc=com" -D "cn=myadminaccount,ou=Users,dc=mydomain,dc=com" -H ldap://mydomaincontroller/ -W -x "(objectclass=Computer)" cn
2008/08/01
Awk and other text processing tips
How to sum the third column?
e.g., calculate total tps across all physical disks from iostat -d output:Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 3.73 30.38 67.80 4864950 10857636
sdb 3.82 30.39 71.74 4866793 11488576
sdc 0.00 0.05 0.00 7208 8
iostat -d |egrep "sd.\ " | awk 'BEGIN {x=0} {x+=$2} END {print x}'
or, less elegantly,
iostat -d| egrep "sd.\ " | awk 'BEGIN {ORS=""}; {print $2"+"}' | ( cat; echo 0)|bc
How to grab just certain columns?
How do I use awk to print the first column, and then the third through the end, for example to grab just the fields I want from an apache log file?awk '{ print $1" " substr($0, index($0,$6)) }' /var/log/httpd/access_log*
gives us something like
10.95.10.20 "POST /license/associateproduct.php HTTP/1.1" 200 8 "-" "Java/1.6.0_17"
10.95.14.248 "POST /license/authorize.php HTTP/1.1" 200 84 "-" "PycURL/7.19.5"
2008/07/16
NDMP Backups with EMC Networker - Quick Start
Table of Contents
Background
Networker Server Setup
Install Base OS and prerequisites
Install and configure Networker App
Configure Licensing for NDMP Backups
Configure backup location, pool, and media (disk based backups are fast for testing)
Client/Management Station Setup
Install Networker Client and Management Center
Configure the Management Console and Networker Server
Configuring for NetApp FAS
Prep the FAS
Prep Networker
Perform the Backup
Configuring for EMC Celerra
Prep Networker
Prep the Celerra
Perform the Backup
Background
Why NetApp and/or EMC has not made such a short and simple guide (that I could find), why a humble sys-admin like myself has had to go through the trial and error process to figure this all out, is only a reflection on the fact that EMC (especially EMC, but to some extent NetApp also,) has become so large as to no longer realize that customers are there. Someone checks "Product Manual" off a release check-list, and calls it good. But none of those people evaluate whether the manual is any good, instead preferring to up-sell to $5000 of videos, or on-site consultants. Nobody thinks that, "Hey, this customer ordered the NDMP option; We should send them a quick-start guide to help them get started with some of the big partners".
This quick guide does provide a quick start guide to performing a disk-based backup via NDMP of your NetApp Filer or EMC Celerra NAS with EMC Networker 7.4. This quick guide does not cover the ins and outs of each platform, data archival and disaster preparedness best practices, etc. This guide was set up using my environment at a company where I know longer work; I cannot provide additional support.
If you are evaluating EMC Networker, ask your sales rep for evaluation license codes for the NDMP and disk backup features; otherwise, you'll need to buy them.
The license codes are all temporary, 30 days.
Networker Server Setup
Install Base OS and prerequisites
Install CentOS 5 i386. Install the X window system (no gnome or KDE is required), as some X libraries are a prerequisite for the networker client (and perhaps other components
Install libgcj and java-1.4.2-gcj-compat for the graphical java components.
service iptables stop
chkconfig iptables off
Disable selinux (set to disabled in /etc/sysconfig/selinux).
Install and configure Networker App
Extract the networker server Linux files as follows:
tar xzf nw74_linux_x86_64.tar.gz
Install the networker server linux component rpms as follows:
rpm -ivh lgtoclnt-7.4-1.i686.rpm lgtolicm-7.4-1.i686.rpm lgtoman-7.4-1.i686.rpm lgtoserv-7.4-1.i686.rpm lgtonode-7.4-1.i686.rpm lgtonmc-3.4-1.i686.rpm
If using DHCP, add DHCP_HOSTNAME=<hostname> to /etc/sysconfig/network-scripts/ifcfg-eth0 . This will cause the DHCP client to also register the hostname in DNS.
Add an entry for the eth0 ip address and hostname to /etc/hosts.
Run these commands to configure and start networker services
/etc/init.d/networker stop
/etc/init.d/gst stop
/opt/lgtonmc/bin/nmc_config
[root@networker ~]# /opt/lgtonmc/bin/nmc_config
NOTE
====
Install has detected the configuration file of a previous lgtonmc
package. Install will attempt to read the configuration parameters
in this file and present them as default values where appropriate.
Please modify any value that is incorrect or needs to be changed.
The Command Line Reports feature of NetWorker Management
Console requires a Java Runtime Environment (JRE) be
installed on this machine. The JRE version should be
1.4.2 or higher, up to (but not including) 1.6.
Is there a supported JRE already installed on this machine [y]? y
Please specify the directory where JRE is installed [/usr/lib/jvm/jre]? /etc/alternatives/jre_1.4.2/
What port should the web server use [9000]?
What port should the GST server use [9001]?
What directory should be used for the LGTOnmc database [/opt/lgtonmc/lgto_gstdb]?
/opt/lgtonmc/lgto_gstdb/lgto_gst.db already exists, do you want to retain this database [y]? n
/opt/lgtonmc/lgto_gstdb/lgto_gst.db already exists, is it okay to remove it [n]? y
Where are the NetWorker binaries installed [/usr/sbin]?
Start daemons at end of configuration [n]?
Creating installation log in /opt/lgtonmc/logs/install.log.
Performing initialization. Please wait...
Installation successful.
/etc/init.d/networker start
/etc/init.d/gst start
chkconfig networker on
chkconfig gst on
Run nsradmin. Type “visual” at the “nsradmin>” prompt to go into visual edit mode. Select type: “NSR”, then you should see your networker server. Select “Edit”, then key down to the “administrator” field. Add ,“*@*” – WARNING: this is only for testing, and will enable anybody on any machine to affect your backups. DON’T FORGET TO CHANGE IT.
Configure Licensing for NDMP Backups
(See client setup for instructions on accessing the Networker Management Console.)
In the Networker Management Console, click Setup LicensingNew, and enter an enabler code for “NetWorker NDMP Client Connection for NetWorker” or “NetWorker NDMP Client Connection – Tier 1”, Networker Server, Storage, DiskBackup, NDMP for NetApp (use your tier in each case) as needed.
Configure backup location, pool, and media (disk based backups are fast for testing)
(See client setup for instructions on accessing the Networker Management Console.)
In the NMC, double-click the server name of your Networker Server.
Go to Devices Nodename Storage Nodes Nodename; right-click and select “Properties”. Change “Node Type” to NDMP.
Add storage location (device) for disk based backups on the Networker Storage node:
Go to Devices Nodename Devices, Right-click, and select “New”. For “Name:”, enter the backup path, and for “Media Type” enter “file”. Click “OK”. Then double-click the newly-created device again to pull up more configuration options.
Create a media pool for the disk-based backup:
Go to Media <Server Name>Media PoolsNew
Label the media for the device. Go to Devices<Server Name>Devices, right-click your disk backup device, and select “Label”. Change the pool to “disk backup pool” (or whatever your pool was). Note: if you don’t label it to the right pool, you’ll get a perpetual notice in /nsr/logs/messages: “NetWorker media: (waiting) Waiting for 1 writable volumes to backup pool 'disk based backup' disk(s) on <servername>”.
Ensure that a volume exists for the media pool. You may have to create this manually
Go to Media <Server Name> Volumes
Client/Management Station Setup
Your workstation will be the management station.
Install Networker Client and Management Center
This table describes the browser, OS, and JRE requirements:
| Platform | Browser |
| AIX | Mozilla 1.7 |
| HP/UX | Mozilla 1.6 |
| Red Hat Enterprise Linux Server 3, 4 and 5 | Mozilla 1.7 |
| Solaris 8 or 9 | Netscape 7 |
| Solaris 10 | Mozilla 1.7 |
| Windows2003 Enterprise or Datacenter Edition | Internet Explorer 6.x |
| Windows XP | Internet Explorer 5.5 or 6.x |
| Windows Vista | Internet Explorer 7.0 |
| Environment | Version |
| Java Runtime Environment | 1.4.2 or greater (but not 1.6) for all, except, |
You may need to disable the Java 6 runtimes on your workstation from the java control panel (access it in the system control panel.) Also within the Java control panel, you may have to set your temporary file size to 300MB.
Map a drive to \a80IT-Software . Then browse to that drive AppsEMCNetworkerNetworker_7.4networker_7.4-win . Run setup.exe . Select “client” and “networker management console”.
Configure the Management Console and Networker Server
Start up the Networker console by opening a web browser on the management workstation to http://localhost:9000/ , click “start”, and use the credentials administrator:administrator .
Note: The Java app that opens will not receive an icon in the task bar.
Accept the EULA. Change the password as desired. Configure the NLM, Database Backup Server, and Networker server for your networker server FQDN. I receive an error about permissions on the database, but it seems to work okay.
In the Networker Management Console, on the Enterprise tab, your networker server should show up. Click on it, and then double-click the application, “Networker”. This is where you’ll configure
Configuring for NetApp FAS
Prep the FAS
First, ensure that manual entries exist in the hosts files for host name and FQDN for the networker server (e.g., networker), storage node, and NDMP server (e.g., fas270c), each of those nodes. (A correct DNS configuration is not necessarily adequate, and the application’s reverse lookups may fail).
Prep Networker
Configure the client Fas270c:
Go to Configuration <Server Name>ClientsNew
Perform the Backup
Run the command on the legato server:
(this example will perform a full NDMP dump of a volume, a qtree, or a subvolume or sub-qtree)
[root@vm-legato ~]# nsrndmp_save -T dump -M -c fas270c.<fqdn> -g Default /vol/vol0/auto-D
Configuring for EMC Celerra
Prep Networker
Configure the client for the Celerra Data mover in the Networker UI:
Set tape buffer size. (Ignore this step in 5.5.30. default of 128KB; networker disk backup requires 32KB; range is 64-1168)
Prep the Celerra
Log in to the Celerra front-end as nasadmin.
$ server_param server_2 -facility NDMP -modify bufsz -value 32
[nasadmin@ns350cs nasadmin]$ server_param server_2 -facility NDMP -list
server_2 :
param_name facility default current configured
maxProtocolVersion NDMP 4 4
convDialect NDMP '8859-1' '8859-1'
scsiReserve NDMP 1 1
dialect NDMP '' ''
includeCkptFs NDMP 1 1
md5 NDMP 0 0
forceRecursiveForNonDAR NDMP 0 0
snapTimeout NDMP 5 5
bufsz NDMP 128 128
snapsure NDMP 0 0
v4OldTapeCompatible NDMP 1 1
create ndmp user on data mover
[root@ns350cs root]# /nas/sbin/server_user server_2 -add -password ndmp
Creating new user ndmp
User ID: 1000
Group ID: 1000
Comment: NDMP user
Home directory: /home/ndmp
Shell:
Changing password for user ndmp
New passwd: ndmppass
Retype new passwd: ndmppass
Perform the Backup
[root@vm-legato ~] nsrndmp_save -T dump -M -c 192.168.123.156 -g Default /siq_daily.2/foo
2008/05/07
PAM notes
Linux PAM (Pluggable Authentication Modules) "is a system of libraries that handle the authentication tasks of applications (services) on the system."
Configuration for individual applications/services may reside in /etc/pam.d . If that directory does not exist, then PAM will look for the single config file /etc/pam.conf . The configuration file(s) define the connection between applications (services) and the pluggable authentication modules (PAMs) that perform the actual authentication tasks.
PAM policy/config file syntax
Each line (rule) in a policy file has 4 parts:
- context (service type) - what aspect of the user's request for a restricted service does this line affect?
- auth -- (authentication) authenticate a user and set up user credentials. Authentication means that the user proves his identity; typically, this involves entering a password, but it may include a hardware based authentication scheme (e.g., smart card). The setting up of user credentials may include setting up group memberships or other privileges.
- account -- (authorization?) -- provide account verification types of service: e.g., has the user's password expired? is this user permitted access to the requested service at this time? Are sufficient system resources available? is this account allowed on the console?
- password -- update authentication tokens, for example, "please enter a new password". Typically, there is one module for each challenge/response based auth type.
- session -- session setup and cleanup; covers things that should be done prior to a service being given, and after it is withdrawn. For example, leaving audit trails, mounting the user's home dir, or unmounting it after logoff.
- control - tells PAM how to handle a "fail" result from a module's authentication task. There are two types of syntax for this control field: the simple one has a single simple keyword; the more complicated one involves a square-bracketed selection of value=action pairs.
- For the historical/simple control field, valid values are
- required -- "if fail, then ultimately fail, but first finish the remaining stacked modules." failure of such a PAM will ultimately lead to the PAM-API returning a failure after the remaining stacked modules (for this service and type) have been invoked.
- requisite -- "if fail, then return to the app now with a failure". like required, but in the case where such a module returns a failure, control is directly returned to the application (without attempting the other stacked modules). The returned value is that associated with the first required or requisite module to fail. This flag may protect, for example, against the possibility of a user getting the opportunity to enter credentials over an unsafe medium; but it may also inform an attacker of valid accounts on a system.
- sufficient -- "if a prior required module has not failed, then a success here is good enough to return to the app immediately with success". success of such a module is enough to satisfy the authentication requirements of the stack of modules (if a prior required module has failed, then the success of this one is ignored). A failure of this module is not deemed as fatal to satisfying the application that this type has succeeded. If the module succeeds, the PAM framework returns success to the application immediately, without trying other modules
- optional -- the success of this module is only important if it is the only module in the stack associated with this service+type.
- include -- include all lines of given type from the configuration file specified as an argument to this control. (On recent RH-based systems, individual application files will tend to include system-auth instead of "other".)
- substack -- include all lines of given type from the configuration file specified as an argument to this control. This differs from include in that evaluation of the done and actions in a substack does not cause skipping the rest of the complete module stack, but only of the substack. Jumps in a substack also can not make evaluation jump out of it, and the whole substack is counted as one module when the jump is done in a parent stack. The reset action will reset the state of a module stack to the state it was in as of beginning of the substack evaluation.
- module path - the PAM module being called
- module arguments (optional) -- options passed to the PAM module. This is a space separated list of tokens that can be used to modify the specific behavior of the given PAM. See individual module documentation for details of that module's options. For arguments that include spaces, surround that argument with square brackets.
The lines/modules are run in the order in which they occur in the file. They're passed the module options (if any) and user/request info, and they return a pass/fail (or other) result. The modules are run until an overall pass/fail result is reached, and that result is passed back to the service to which the user has requested access.
Any line in the config file that is not formatted correctly will generally tend to make the authentication process fail.
Errors are written to syslog.
Detail on Control types
- For the historical/simple control field, valid values are
- required -- failure of such a PAM will ultimately lead to the PAM-API returning a failure after the remaining stacked modules (for this service and type) have been invoked.
- requisite -- like required, but in the case where such a module returns a failure, control is directly returned to the application (without attempting the other stacked modules). The returned value is that associated with the first required or requisite module to fail. This flag may protect, for example, against the possibility of a user getting the opportunity to enter credentials over an unsafe medium; but it may also inform an attacker of valid accounts on a system.
- sufficient -- success of such a module is enough to satisfy the authentication requirements of the stack of modules (if a prior required module has failed, then the success of this one is ignored). A failure of this module is not deemed as fatal to satisfying the application that this type has succeeded. If the module succeeds, the PAM framework returns success to the application immediately, without trying other modules
- optional -- the success of this module is only important if it is the only module in the stack associated with this service+type.
- include -- include all lines of given type from the configuration file specified as an argument to this control. (On recent RH-based systems, individual application files will tend to include system-auth instead of "other".)
- substack -- include all lines of given type from the configuration file specified as an argument to this control. This differs from include in that evaluation of the done and actions in a substack does not cause skipping the rest of the complete module stack, but only of the substack. Jumps in a substack also can not make evaluation jump out of it, and the whole substack is counted as one module when the jump is done in a parent stack. The reset action will reset the state of a module stack to the state it was in as of beginning of the substack evaluation.
- The more complicated syntax is as follows:
- valid control values have the following form: [value1=action1 value2=action2 ...]
- ...where valueN corresponds to the return code from the function invoked in that module line, and is one of these:
- success, open_err, symbol_err, service_err, system_err, buf_err, perm_denied, auth_err, cred_insufficient, authinfo_unavail, user_unknown, maxtries, new_authtok_reqd, acct_expired, session_err, cred_unavail, cred_expired, cred_err, no_module_data, conv_err, authtok_err, authtok_recover_err, authtok_lock_busy, authtok_disable_aging, try_again, ignore, abort, authtok_expired, module_unknown, bad_item, conv_again, incomplete, and default.
- ActionN can be either an unsigned integer, n, signifying an action of "jump over the next n modules in the stack', or it can take one of the following forms:
- ignore -- when used with a stack of modules, the module's return status will not contribute to the return code the application obtains.
- bad -- this action indicates that the return code should be thought of as indicative of the module failing. If this module is the first in the stack to fail, its status value will be used for that of the whole stack.
- die -- equivalent to bad, with the side effect of terminating the module stack and PAM , and immediately returning to the application
- ok -- this tells PAM that the administrator thinks this return code should contribute directly to the return code of the full stack of modules. IE, if the former state of the stack would lead to a return of PAM_SUCCESS, then the module's return code will override thei value. However, if the former state of the stack holds some value that is indicative of a module's failure, this 'ok' value will not be used to override that data.
- done -- equivalent to ok, with the side effect of terminating the module stack and PAM immediately and returning to the application.
- reset -- clear all memory of the state of the module stack and start again with the next stacked module.
- Each of the four keywords, required, requisite, sufficient, and optional, have an equivalent expression in terms of the [...] syntax as follows:
- required -- [success=ok new_authtok=ok ignore=ignore default=bad]
- requisite -- [success=ok new_authtok=ok ignore=ignore default=die]
- sufficient -- [success=done new_authtok=done default=ignore]
- optional -- [success=ok new_authtok_reqd=ok default=ignore]
2008/03/24
Fix Absolute Symbolic Links from tar or rsync SNAFU
- This one won't work for a couple of reasons; first, it can't distinguish between intended relative paths and accidental relative paths; also, cut won't work if any files that have a different character length in any fields of the ls output, such as ownership or date (there's a way to merge whitespace that escapes me, not fmt or pr, but something :
- find . -type l -print0 | xargs -0 ls -laQ | grep ' -> \"relative/path/stuff'| cut --complement -d " " -f 1-8 | sed -e 's/ -> \"/\;\"\//g' | awk -F\; '{ print "ln -sf " $2 " " $1 }' > fix_symlinks
- How about this:
- find . -type l -printf 'ln -sf "/%l" "%p"\n' | grep "\"/accidental/relative/path" > fix_symlinks
- check the fix_symlinks for things that you actually don't want to change.
- . ./fix_symlinks
Also, note the "symlinks" command in the "symlinks" package.
2007/10/02
Pixelated windows graphics fix
The problem was, Dell knew the display was hi-res and small, so they set a registry setting for me to automatically dumb-down the graphics, to make them more legible.
This made web pages render incorrectly, and it made the screen icons look bad. Change this key to fix it.
[HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main]
change
"UseHR"=dword:00000000
Who says only Linux has silly things like this, or that Windows is more user-friendly? My grandma would never have found this fix.
2007/09/27
PostgreSQL Tips and Tricks
# Log in to a postgreSQL database
psql -U username dbname
# Get list of databases.
SELECT datname FROM pg_database;
# or
\l
# or
psql -l
# Get size of tables, sorted in descending order.
SELECT relname, relpages,reltuples FROM pg_class ORDER BY relpages DESC;
# Show tables in a database.
select * from information_schema.tables where table_schema='public' and table_type='BASE TABLE';# or
\dt
# Show structure of a table.
\d tablename
psql Commands
General
| \c[onnect] [DBNAME|- [USER]] | connect to new database (currently "dfdata") |
| \cd [DIR] | change the current working directory |
| \copyright | show PostgreSQL usage and distribution terms |
| \encoding [ENCODING] | show or set client encoding |
| \h [NAME] | help on syntax of SQL commands, * for all commands |
| \q | quit psql |
| \set [NAME [VALUE]] | set internal variable, or list all if no parameters |
| \timing | toggle timing of commands (currently off) |
| \unset NAME | unset (delete) internal variable |
| \! [COMMAND] | execute command in shell or start interactive shell |
Query Buffer
| \e [FILE] | edit the query buffer (or file) with external editor |
| \g [FILE] | send query buffer to server (and results to file or pipe) |
| \p | show the contents of the query buffer |
| \r | reset (clear) the query buffer |
| \s [FILE] | display history or save it to file |
| \w FILE | write query buffer to file |
Input/Output
| \echo [STRING] | write string to standard output |
| \i FILE | execute commands from file |
| \o [FILE] | send all query results to file or |pipe |
| \qecho [STRING] | write string to query output stream (see \o) |
Informational||
| \d [NAME] | describe table, index, sequence, or view |
| \d{t|i|s|v|S} [PATTERN] (add "+" for more detail) | list tables/indexes/sequences/views/system tables |
| \da [PATTERN] | list aggregate functions |
| \db [PATTERN] | list tablespaces (add "+" for more detail) |
| \dc [PATTERN] | list conversions |
| \dC | list casts |
| \dd [PATTERN] | show comment for object |
| \dD [PATTERN] | list domains |
| \df [PATTERN] | list functions (add "+" for more detail) |
| \dg [PATTERN] | list groups |
| \dn [PATTERN] | list schemas (add "+" for more detail) |
| \do [NAME] | list operators |
| \dl | list large objects, same as \lo_list |
| \dp [PATTERN] | list table, view, and sequence access privileges |
| \dT [PATTERN] | list data types (add "+" for more detail) |
| \du [PATTERN] | list users |
| \l | list all databases (add "+" for more detail) |
| \z [PATTERN] | list table, view, and sequence access privileges (same as \dp) |
Formatting
| \a | toggle between unaligned and aligned output mode | ||||||||||||
| \C [STRING] | set table title, or unset if none | ||||||||||||
| \f [STRING] | show or set field separator for unaligned query output | ||||||||||||
| \H | toggle HTML output mode (currently off) | ||||||||||||
| \pset NAME [VALUE] | set table output option (NAME := {format|border|expanded|fieldsep|footer|null| | |numericlocale|recordsep|tuples_only|title|tableattr|pager}) | ||||||||||||
| \t | show only rows (currently off) | ||||||||||||
| \T [STRING] | set HTML
Copy, Large Object
|
2007/09/14
Simple expect ssh example
#!/usr/bin/expect
set timeout 1
set cmd {uname -a}
spawn ssh root@$argv
expect_after eof { exit 0 }
## interact with SSH
expect "yes/no" { send "yes\r" }
expect "password:" { send "rootpasswd\r" }
expect "# "
send "$cmd\r"
expect "$cmd\r"
expect "(.*)\r"
send "exit\r
2007/07/25
120 VAC Outlet / Plug wiring
The three slots/holes and their attachment points are:
Silver color screws, this is where your grounded leg {aka neutral} wire goes, this by North America standards should be a white wire, however it can be white or gray. (looking at the front face of an outlet, with the ground on bottom, this is the slot/screw on the left).
Equipment grounding screw, this is where the circuit grounding conductor wire will be attached, this on most general purpose circuits will be a bare wire (although an equipment grounding conductor can be bare or green), this bare or green wire is also connected to a grounding screw on the back of the electrical box (if the electrical box in use is metallic)
Brass color screws, this is where the ungrounded leg [aka hot conductor] wire is attached, the most frequently used wire colors are black & red but can be any color in rainbow but white, gray, bare or green.
Do a smoke-test with something unimportant; don't touch it while testing it. Measure potential between the new ground and a known good ground.
2007/07/18
Windows Updates from the command line with wuauclt.exe
/configlist - lists WUAUCLT configuration./installAUclient - Install the correct AU client by force