Do You Work With Me?

If you work with me currently please email me from a corporate email address for a password to the company-related (private) posts.

Your Schedule Just Exploded

You have autosys jobs scheduled in a Solaris in a variety of European timezones. Today they went off at the wrong times. Happy European DST. Check your Solaris patches.

Hunting Down Cross-Authorizing SSH Keys

Its well-known that some users insert their public keys into another user’s authorized_keys file to escalate their privileges. Let’s find out who’s playing silly games.

#!/usr/bin/bash

for id in `/usr/bin/ypcat passwd.byname | awk ‘!/’”$USER”‘/ {FS=”:”} { print $1 }’`
do
(ssh -qTn $id@$HOSTNAME id | grep $id >/dev/null) && \
echo “$USER cross-authorizes as $id on $HOSTNAME”
done

Automatically Restart Your Python SimpleHTTPServer

Command Line Fu turned me on to python’s built-in servers a while ago, unfortunately, root tends to kill my long-running processes. Schedule the below in cron to tickle the default python webserver port and start a new instance of the python webserver in the specified <ROOT_DIR> if no response.


(echo '' > /dev/tcp/localhost/8000) 2>/dev/null || (cd /<ROOT_DIR> && /bin/python -m SimpleHTTPServer &)


If this doesn’t work, your BASH was compiled without network support. wget should be able to help you.

Autosys man pages

Because I could never find these myself:

http://www.exploreluxury.com/EL_Autosys/

SSH Connectivity Checks for Autosys

#!/usr/local/bin/perl 

# Find and report on all known hosts that cannot be reached
# via a passwordless ssh connection

use IO::Socket;

($hostname) = split(' ', `hostname`);

@userArr = split(/[\(-\)]/, `id`);
$user = $userArr[1];

$timeout = 5;

@priHosts = ('list', 'high', 'priority', 'hosts', 'here');
$supportingEmail = 'you@your_domain.com';

open (KNOWN_HOSTS, "$ENV{ 'HOME' }/.ssh/known_hosts");
while (<KNOWN_HOSTS>) {
	chomp;

	($host) = split(' ');
	($host) = split(/,/) if ($host =~ /,/);
	push(@lookup, $host);
}
close(KNOWN_HOSTS);

@lookup = sort(@lookup);

foreach $host (@lookup) {

	$isUp = 1;

	$sock = new IO::Socket::INET (
		PeerAddr => $host,
		Proto    => 'tcp',
		Timeout  => $timeout
	) || ($isUp = 0);

        if ($isUp) {
		$stat = `/usr/bin/ssh -x -p22  -C -cblowfish $host /dev/null/discoProc 2>&1`;

		if ($stat !~ /discoProc/) {
			print "USER $user CANNOT SSH TO KNOWN HOST $host!\t";

			if ($stat =~ /Write failed: Broken pipe/) {
				print "invalid SSH KEY/bad permissions. Talk to the sysadmin(s).\n";
			}
			elsif ($stat =~ /Host key verification failed/) {
				print "invalid HOST KEY. Log in to $hostname, su to user $user and ssh manually to $host to clear this error.\n";
			}
			elsif (/no matching cipher found/) {
				print "invalid CIPHER. This scipt uses blowfish, which is apparantly not going to happen.\n";
			}
			else {
				print "USER $user CANNOT SSH TO KNOWN HOST $host!\nSSH CONNECTION LOG BELOW:\n\n$stat\n\n\n";
			}

			if (grep /$host/, @priHosts) {
				print "SENDING A PANIC MAIL\n";
				open (MAIL,  "|/usr/bin/mailx -s 'IMPACTING SSH FAILURE' $supportingEmail");
				print MAIL "user $user on $hostname can't SSH to $host\n";
				close(MAIL);
			}
		}
	}
	else {
		print "USER $user CAN'T SSH TO KNOWN HOST $host!\tSSH FROM HOST $hostname TO HOST $host TIMES OUT AFTER $timeout SECONDS!\n";
	}
}

Balance Remotely Executed Script Runs

Find available SSH proxies and execute script remotely on least busy host as determined by uptime(1) poll

use IO::Socket;

$loud = 1;
$timeout = 1;

@turnstiles = ( 'your', 'hosts',
                'belong', 'here'
);

$jobToRun = join(' ', @ARGV);

$load = 1000;
$lastLoad = 1000;

foreach $turnstile (@turnstiles) {

        $sock = new IO::Socket::INET (   # try opening a
                PeerAddr => $turnstile,  # connection to
                PeerPort => 22,          # the ssh port
                Proto    => 'tcp',       # timeout after
                Timeout  => $timeout     # $timeout seconds
        ) || ($turnstile = 0);

        if ($turnstile) {
                $punch = "$ssh $turnstile /usr/bin/uptime 2>&1";

                open(SSH, "$punch|");
                @_ = split(' ', );
                close(SSH);

                $c = @_; $c = $c - 3;

                $load = $_[$c];
                $load =~ s/,$//;

                if ($loud) {
                        print "$turnstile 5 minute ";
                        print "system load: $load\n"
                }

                $useProxy = $turnstile    if ($load < $lastLoad);
                $lastLoad = $load;
        }
        close($sock) if ($sock);
}
die ("No available proxy!\n") unless (length($useProxy));

print "\nUsing $useProxy ($load)\n\n" if ($loud);
print "Running $ssh $useProxy \"$jobToRun\"\n" if ($loud);

system("$ssh $useProxy $jobToRun");

# exit with error value from above
exit $? >> 8;

XML Excel Spreadsheets

Excel is the de-facto spreadsheet program where I work, and management loves it’s reports. Not surprisingly there’s no perl module for creating Excel spreadhseets on our production hosts. Of course CSV files can be used for basic spreadsheets, but I needed a whizzy layout for charting maintenance windows for production processes. In comes XML Excel.

Starting with some boilerplate at the top of the document:

<?xml version="1.0"?>
<Workbook xmlns="urn:schemas-microsoft-com:office:spreadsheet">
<ExcelWorkbook xmlns="urn:schemas-microsoft-com:office:excel">

<Table>

And closing the XML document with a footer:


</Table>
</Worksheet>
</Workbook>


We now have a valid Excel spreadsheet. Data is added cell-by-cell, row-by-row.


<Row> <Cell> <Data ss:Type=\"String\"> YOUR DATA HERE </Data> </Cell >


Not terribly exciting, but it’s a solid start. I needed a charting script to lay out data for a 24 hour day, with a cell for every 15 minutes. If an application is in a running state, fill the cell in with a yellow background, otherwise fill it with a blue background as below:

Time:	00:00	00:15	00:30	00:45	01:00
Process X:

XML lets us do this by adding a style to the document header:


<Styles>
<Style ss:ID="s65"><Interior ss:Color="#00B0F0" ss:Pattern="Solid"/> </Style>
<Style ss:ID="s62"><Interior ss:Color="#FFDE00" ss:Pattern="Solid"/> </Style>
</Styles>


If theres any interest, I’ll post a short pick-and-choose library for laying out Excel workbooks.

Rememberance

Could you find a moment to remember a very brave operator?

Gladys Gibson's Memorial

Gladys Gibson was the first post-humous recipient of the Theodore Newton Vail award for extrordinary service. The medal was received by her mother.

http://blog.modernmechanix.com/2008/05/05/heroes-of-the-switchboard-and-phone-lines/

http://fultonhistory.com/newspaper%2010/Carmel%20NY%20Putnam%20Country%20Republican/Carmel%20NY%20Putnam%20Country%20Republican%201938-1939%20Grayscale/Carmel%20NY%20Putnam%20Country%20Republican%201938-1939%20Grayscale%20-%200179.pdf

automagic autosys punchlists

Since the dawn of time, NOC management has demanded that operators have a punchlist to check off as jobs run. In our almost fully-automated shop this has become more of a training and DR tool (and likely a way to confirm that operators are not sleeping during their shift). I hate maintaining these things, so (of course) I automated it.

This makefile translates a JIL file into an RTF format punchlist, extracting box name, box description, command name (not job name), and days-of-the-week that that box runs on. If a box has no defined days, a M-F run is assumed. Commands found in the ‘banned’ file will be culled from the final runsheet. Instructions in the ‘specials’ file will be appended underneath the box description.

It is assumed that your autosys job layout looks like ‘$RUNSTREAM… $HOSTNAME’ (mine does) and that your runstream executes in ASCIIbetical order (mine do this, too). Please create a directory for the makefile that is named $SCOPE.

sample output:

BOX_NAME_HERE

“Job description here”

HOST1  /path/to/command                  mo:___ tu:___ we:___ th:___ fr:___

HOST2  /path/to/command                  mo:___ tu:___ we:___ th:___ fr:___

HOST3  /path/to/command                  mo:___ tu:___ we:___ th:___ fr:___

HOST4  /path/to/command                  mo:___ tu:___ we:___ th:___ fr:___