Example Application - Octave

Table of Contents

Overview

This section covers how to use OASIS CVMFS system to run a real application like Octave or MatLab statistical package. For this example, we'll use Octave although Matlab can be substituted without any changes in the code. This example will go through using Matlab to generate a random matrix, doing some simple matrix operations and then calculate the eigenvectors of the matrix.

Setup

First, let's create a work directory. As always, you can use tutorial instead.

$ mkdir -p osg-octave/log
$ cd osg-octave

Testing Octave

Since Octave is installed into CVMFS, it's not available in the normal system paths. We'll need to set up those paths so we can access Octave correctly:

$ source /cvmfs/oasis.opensciencegrid.org/osg/palms/sw/octave/el5/x86_64/default/setup.sh

Once the path is set up, we can run Octave:

$ octave
GNU Octave, version 3.6.4
Copyright (C) 2013 John W. Eaton and others.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type `warranty'.
Octave was configured for "x86_64-unknown-linux-gnu".
Additional information about Octave is available at http://www.octave.org.
Please contribute if you find this software useful.
For more information, visit http://www.octave.org/get-involved.html
Read http://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type `news'.
warning: X11 DISPLAY environment variable not set
warning: readline is not linked, so history control is not available
octave:1> 1+1
ans =  2
octave:2> quit()
$

Running Octave code

Now lets run the create our Octave script:

file: ex1_matrix.octave

A = rand(40, 40)
B = A' * A
[v, d] = eig(A)
diag(d)

Run the script:

$ octave ex1_matrix.octave

This should run fairly quickly.

Building the HTCondor job

Although the previous script ran quickly, suppose we needed to use a 100x100 matrix instead of a 10x10 matrix or do this a few thousand times. Since each invocation is independent of others, we can use condor to easily parallelize this and run it on OSG Connect.
First, we'll need to create a wrapper script to setup the environment for Octave and before running it.

file: octave-wrapper.sh

#!/bin/bash
 
EXPECTED_ARGS=1
 
if [ $# -ne $EXPECTED_ARGS ]; then
  echo "Usage: octave-wrapper.sh file.R"
  exit 1
else
  source /cvmfs/oasis.opensciencegrid.org/osg/palms/sw/octave/el5/x86_64/default/setup.sh
  octave $1
fi

And of course let's make these wrapper executable

$ chmod +x octave-wrapper.sh

Since our previous example only ran for less than a second, let's create a another significantly longer job by operating on a 1000x1000 matrix.

file: ex2_matrix.octave

A = rand(1000,1000)
B = A' * A
[v, d] = eig(A)
diag(d)

Now that we've created a wrapper and a batch job, let's build an HTCondor submit file around it.

file: octave.submit

universe = vanilla
log = log/octave.log.$(Cluster)
error = log/octave.err.$(Cluster).$(Process)
output = log/octave.out.$(Cluster).$(Process)
 
# Setup Octave path, run the ex2_matrix.octave script
executable = octave-wrapper.sh
transfer_input_files = ex2_matrix.octave
arguments = ex2_matrix.octave
+ProjectName="ConnectTrain"
 
requirements = (HAS_CVMFS_oasis_opensciencegrid_org =?= TRUE)
queue 100

Since we're using Octave from CVMFS, we will always need to have a requirement (HAS_CVMFS =?= TRUE) that selects nodes with CVMFS installed.

Submit and analyze

Finally, submit the job to OSG Connect!

$ condor_submit octave.submit

After a short wait, the output from the jobs will be in the ~/octave_tutorial/log/ directory.