The JSV below calculates the time at which the incoming job will reach the hard runtime limit and if it is after the configured maintenance window begin, modifies the hard runtime limit and notifies the user. This ensures that all jobs running on the cluster will end before the maintenance window begins and that the users know that.

Without further ado, here’s the code. It looks for a file named maintenance in the $SGE_ROOT/$SGE_CELL/common directory that contains the unix timestamp that the maintenance window begins. It’s executed client-side from the cluster-wide sge_request file in the same directory. The script requires the Date and Time libraries for some date math that it does. I’m not a perl programmer, so bear with me.

#!/usr/bin/perl

use strict;
use warnings;
no warnings qw/uninitialized/;

use Env qw(SGE_ROOT SGE_CELL);
use lib "$SGE_ROOT/$SGE_CELL/common/perl/lib/perl5/site_perl/5.8.8/";
use Date::Format;
use Time::Duration;
use lib "$SGE_ROOT/util/resources/jsv";
use JSV qw( :DEFAULT jsv_sub_is_param jsv_sub_add_param jsv_sub_get_param jsv_send_env jsv_log_info jsv_is_param jsv_get_param );

sub hms2s {
	my $input = shift;
	if( $input =~ m/(\d*):(\d*):(\d*)/ ) {
		my $h = $1 || 0;
		my $m = $2 || 0;
		my $s = $3 || 0;
		return $h*3600+$m*60+$s
	} elsif( $input =~ m/(\d+)/) {
		return $1;
	} else {
		return 0;
	}
}

jsv_on_start(sub {
	jsv_send_env();
});

jsv_on_verify(sub {

	my $data_file="$SGE_ROOT/$SGE_CELL/common/maintenance";
	my $success = open(DAT, $data_file); 

	if( !$success ) {
		jsv_accept("No maintenance window scheduled.");
		return;
	}

	my $maintenance_begin = <DAT>;
	close(DAT);
	if( !$maintenance_begin ) {
		jsv_accept("No maintenance window scheduled.");
		return;
	}

	# Allow a 5 minute window for jobs to die before the maintenance officially starts.
	my $delta = 300;
	my $now = time();

	if( $maintenance_begin-$delta < $now ) {
		jsv_log_info('*'x81);
		jsv_log_info('* Maintenance is currently in progress');
		jsv_log_info('* For more information, see http://blahblah');
		jsv_log_info('*'x81);
		jsv_reject();
		return;
	}

	my $requested_rt = hms2s(jsv_sub_get_param('l_hard', 'h_rt'));
	if( $now + $requested_rt > $maintenance_begin - $delta ) {
		my $time_to_run = $maintenance_begin - $delta - $now;
		jsv_sub_add_param('l_hard','h_rt',$time_to_run);
		jsv_log_info('*'x81);
		jsv_log_info('* A maintenance window is scheduled for '.time2str('%m/%d/%Y %H:%M:%S', $maintenance_begin, 'EST'));
		jsv_log_info('* Your job will be allowed to run for '.duration($time_to_run));
		jsv_log_info('* For more information, see http://blahblah');
		jsv_log_info('*'x81);
	}
	jsv_correct('Job is accepted');
	return;
}); 

jsv_main();

The Script

#!/bin/bash

# Configuration values
SGE_ROOT=/usr/share/gridengine
SGE_CELL=default
SVN_URL=file:///home/user/subversion
SVN_IMPORT_DIR=gridconfig
SVN_LOAD=/usr/share/doc/subversion-1.4.2/svn_load_dirs.pl

# The real work
TMPD=`mktemp -d`
source $SGE_ROOT/$SGE_CELL/common/settings.sh
$SGE_ROOT/util/upgrade_modules/save_sge_config.sh $TMPD
find $TMPD -name accounting -exec rm -v {} \;
sed -i "/^load_values/d" $TMPD/execution/*

$SVN_LOAD -no_user_input $SVN_URL $SVN_IMPORT_DIR $TMPD

rm -fr $TMPD

Run this regularly from cron to enable the real power that I’ll discuss below.

Explanation

Lines 4-8 define configuration variables used later on. The SGE_* variables might already be defined for your shell. If not, you know what they should be. SVN_URL is the URL of the Subversion repository you’d like to store the configuration in. It should have already been created using svnadmin create <dir>. SVN_IMPORT_DIR is the name of the directory in your repository that will store the configuration information. The real magic of this script comes from svn_load_dirs.pl, which comes with Subversion. Provide the path to it in the SVN_LOAD variable.

Lines 11-13 setup the SGE environment and dump the current configuration to a temporary directory. The save_sge_config.sh script comes with Grid Engine and is a gem. It writes the grid configuration to a directory structure. It’s worthwhile to run it on your own and browse the output.

Lines 14 and 15 do some post-processing on the dumped configuration. In particular, the accounting file is deleted because in our case, it’s huge and dynamic, so I don’t want to store multiple copies of it in the repository. Another piece of dynamic information is the load_values for each execution host. These will change each time you run the script and don’t provide any useful historical information that you can’t get in better form from Ganglia, so I get rid of them. One last piece of dynamic information is the backup_date file which contains the date and time that save_sge_config.sh. I like to keep this around because it provides some context, but you could also delete that file here.

With the configuration gathered, line 17 simply calls svn_load_dirs.pl to load it into Subversion. A commit message in the form of ‘Load /tmp/sakfjaskfj into gridconfig.’ is attached to the import.

Cool stuff

Now that all the configuration information is stored in Subversion, the world is our oyster. We can use any of the normal Subversion tools to learn more about the configuration.

Finding out what’s changed

By using svn diff, you can find out what’s changed between different dates. For example, here are the changes that I made between October 2nd and 3rd:

user@host ~$ svn diff file:///home/user/subversion/gridconfig -r {2009-10-02}:{2009-10-03}
Index: backup_date
============================================
--- backup_date (revision 177)
+++ backup_date (revision 181)
@@ -1 +1 @@
-2009-10-01_00:10:01
+2009-10-02_00:10:01
Index: usersets/superusers
============================================
--- usersets/superusers  (revision 177)
+++ usersets/superusers  (revision 181)
@@ -2,4 +2,4 @@
 type    ACL
 fshare  0
 oticket 0
-entries johnny
+entries johnny,billy

You’ll notice the backup_date is there, as I mentioned previously. Additionally, you’ll notice I added billy to the superusers userset. Good to know.

WebSVN

WebSVN is an online Subversion repository browser. Set it up correctly to point at your new gridconfig repository and you can get the same diff information as before through your browser.

Rollback your grid configuration

Did something go horribly wrong with your configuration? Rollback to a previous version using the load_sge_config.sh that also comes with Grid Engine and is located in $SGE_ROOT/util/upgrade_modules. Simply checkout the version of the repository you want to load and pass it to load_sge_config.sh. If you do this successfully, let me know. So far we haven’t had any catestrophic configuration changes.

Something else?

If you manage to do something else interesting with your versioned configuration information, leave a comment below or send me a message.

Update: Fixed a typo in the code with the $TMPD variable.