Using SWIFT on OSG Connect

Introduction

This is a quick introduction to using SWIFT on OSG Connect. SWIFT is a parallel scripting language that lets you easily incorporate workflows using different applications and convert them into a simple script file.  The SWIFT runtime takes this script and tries to run as much of the workflow in parallel as possible. Once you finish using the exercises on this page you will be able to create workflows in SWIFT.

Conventions

The following conventions are used throughout this document: 

  • All of the following exercises must be done using the BASH shell
  • The setup.sh in the tutorial directory is sourced before running any of the tutorial:

    Cleanup example
    $ source setup.sh
  • Each part of the exercises below is located in a separate directory (e.g. part01, part02, ...)
  • To cleanup the directory and remove all outputs, after running SWIFT, run in the exercise directory

    Cleanup example
    $ ../bin/cleanup

Introductory Exercises

Scripts

The introductory exercises use two different mock "science applications" that act as simple stand-ins for real since applications.

simulate.sh

The simulation.sh script is a simple substitute for a scientific simulation application. It generates and prints a set of one or more random integers in the range 0-29,999 as controlled by its optional arguments.

 See arguments
$ ./app/simulate.sh --help ./app/simulate.sh: usage: -b|--bias offset bias: add this integer to all results [0] -B|--biasfile file of integer biases to add to results [none] -l|--log generate a log in stderr if not null [y] -n|--nvalues print this many values per simulation [1] -r|--range range (limit) of generated results [100] -s|--seed use this integer [0..32767] as a seed [none] -S|--seedfile use this file (containing integer seeds [0..32767]) one per line [none] -t|--timesteps number of simulated "timesteps" in seconds (determines runtime) [1] -x|--scale scale the results by this integer [1] -h|-?|?|--help print this help
ArgumentDescription

-b,--bias

offset bias: add this integer to all results [0]

-B,--biasfile

file of integer biases to add to results [none]

-l,--log

generate a log in stderr if not null [y]

-n,--nvalues

print this many values per simulation [1]

-r,--range

range (limit) of generated results [100]

-s,--seed

use this integer [0..32767] as a seed [none]

-S,--seedfile

use this file (containing integer seeds [0..32767]) one per line [none]

-t,--timesteps

number of simulated "timesteps" in seconds (determines runtime) [1]

-x,--scale

scale the results by this integer [1]

-h,-?,?,--help

print this help

With no arguments, simulate.sh prints 1 number in the range of 1-100. Otherwise it generates n numbers of the form R * scale + bias where R is a random integer. By default it logs information about its execution environment to stderr. Here’s some examples of its usage:

Running simulate application
$ simulate.sh 2>log
       5
$ head -4 log

Called as: /home/wilde/swift/tut/CIC_2013-08-09/app/simulate.sh:
Start time: Thu Aug 22 12:40:24 CDT 2013
Running on node: login01.osgconnect.net

$ simulate.sh -n 4 -r 1000000 2>log
  239454
  386702
   13849
  873526

$ simulate.sh -n 3 -r 1000000 -x 100 2>log
 6643700
62182300
 5230600

$ simulate.sh -n 2 -r 1000 -x 1000 2>log
  565000
  636000

$ time simulate.sh -n 2 -r 1000 -x 1000 -t 3 2>log
  336000
  320000
real    0m3.012s
user    0m0.005s
sys     0m0.006s

stats.sh 

The stats.sh script serves as a trivial model of an "analysis" program. It reads N files each containing M integers and simply prints the average of all those numbers to stdout. Similarly to simulate.sh it logs environmental information to the stderr.

Running stats script
$ ls f*
f1  f2  f3  f4

$ cat f*
25
60
40
75

$ stats.sh f*
50

Part 1 - Run an application under Swift

The first swift script, p1.swift, runs simulate.sh to generate a single random number. It writes the number to a file.  The control flow for the swift script is shown below.

In the p1.swift file below, the app construct is used to tell SWIFT how to use the simulate.sh script and what the script expects for its inputs and outputs. The simulate application gets translated to simulate.sh using the apps file for the mapping. The file construct to indicate the name of the file used to hold the output from the simulate script.

p1.swift
type file;
app (file o) mysim ()
{
  simulate stdout=@filename(o);
}
file f <"sim.out">;
f = mysim();

To run this script:

p1.swift
$ cd part01
$ swift p1.swift

Now, view the output:

Output from simulate
$ cat sim.out
 95

Part 2 - Running an ensemble of many apps in parallel with "foreach" loops

The p2.swift script introduces the foreach loop to run multiple instances of the simulate script in parallel. Output files are named using the file mapper so each instance writes to output/sim_N.out.

The control flow and source for the script are shown below:

p2.swift
type file;
app (file o) mysim ()
{
  simulate stdout=@filename(o);
}
foreach i in [0:9] {
  file f <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
  f = mysim();
}

To run:

 

Running p2.swift
$ cd part02
$ swift p2.swift

Output files will be named output/sim_N.out:

 

Output of p2.swift
$ ls output/
sim_0.out  sim_2.out  sim_4.out  sim_6.out  sim_8.out
sim_1.out  sim_3.out  sim_5.out  sim_7.out  sim_9.out

 

Part 3 - merging/reducing the results of a parallel foreach loop

The p4.swift script introduces a postprocessing step. After all the parallel simulations have completed, the files created by simulation.sh will be averaged by stats.sh.  The @filenames functions is used to store filenames being transfer outputs from the simulate scripts to the stats script. The control flow and source for the SWIFT script follows:

p3.swift
type file;
app (file o) mysim (int sim_steps, int sim_values)
{
  simulate "--timesteps" sim_steps "--nvalues" sim_values stdout=@filename(o);
}
app (file o) analyze (file s[])
{
  stats @filenames(s) stdout=@filename(o);
}
int nsim   = @toInt(@arg("nsim","10"));
int steps  = @toInt(@arg("steps","1"));
int values = @toInt(@arg("values","5"));
file sims[];
foreach i in [0:nsim-1] {
  file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
  simout = mysim(steps,values);
  sims[i] = simout;
}
file stats<"output/average.out">;
stats = analyze(sims);
To run:

 

Running p3.swift
$ cd part02
$ swift p3.swift

The output will be named output/average.out.

 

Output of p3.swift
$ cat output/average.out
51

Part 4 - Running a parallel ensemble on OSG Connect resources

This is the first script that will submit jobs to OSG through OSG connect. It is similar to earlier scripts, with a few minor exceptions. To generalize the script for other types of remote execution (e.g., when no shared filesystem is available to the compute nodes), the application simulate.sh will get transferred to the worker node by Swift, in the same manner as any other input data file. The control flow and source for the SWIFT script follows:

 

p4.swift
type file;

# Application program to be called by this script:
file simulation_prog <"app/simulate.sh">;

# "app" function for the simulation application:
app (file out, file log) simulation (file prog, int timesteps, int sim_range)
{
   sh @prog "-t" timesteps "-r" sim_range stdout=@out stderr=@log;
}

# Command line parameters to this script:
int nsim = @toInt(@arg("nsim", "10")); 

# number of simulation programs to run
int range = @toInt(@arg("range", "100")); 

# range of the generated random numbers# Main script and dataint steps=3;tracef("\n*** Script parameters: nsim=%i steps=%i range=%i \n\n", nsim, steps, range);foreach i in [0:nsim-1] {
   file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
   file simlog <single_file_mapper; file=@strcat("output/sim_",i,".log")>;
   (simout,simlog) = simulation(simulation_prog, steps, range);
}
To run:

 

Running p4.swift
$ cd part04
$ swift p4.swift

The output will be named output/sim_N.out.

Output of p2.swift
$ cat output/average.out
51
SWIFT uses parameters in the sites.xml file to determine parameters to use when submitting jobs to HTCondor.  The key values to note are the pool element which sets the name for the pool that SWIFT submits to.  In addition, there is a condor.+ProjectName value that needs to be set to the project that you're submitting as.

 

sites.xml
<config>
  <pool handle="osg">
    <execution provider="coaster" jobmanager="local:condor"/>
    <profile namespace="karajan" key="jobThrottle">5.00</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <profile namespace="globus" key="jobsPerNode">1</profile>
    <profile namespace="globus" key="maxtime">3600</profile>
    <profile namespace="globus" key="maxWalltime">00:01:00</profile>
    <profile namespace="globus" key="highOverAllocation">10000</profile>
    <profile namespace="globus" key="lowOverAllocation">10000</profile>
    <profile namespace="globus" key="internalHostname">128.135.158.173</profile>
    <profile namespace="globus" key="slots">20</profile>
    <profile namespace="globus" key="maxNodes">1</profile>
    <profile namespace="globus" key="nodeGranularity">1</profile>
    <workdirectory>.</workdirectory> <!-- Alt: /tmp/swift/OSG/{env.USER} -->
    <!-- For UC3: -->
    <profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
    <!-- For OSGConnect -->
    <profile namespace="globus" key="condor.+ProjectName">"con-train"</profile>
    <profile namespace="globus" key="jobType">nonshared</profile>
  </pool>
</config>

The other file that SWIFT uses is the apps file.  This file lays out the mappings between applications used in the SWIFT script files and the actual binaries for each pool.  E,g:

apps
osg sh /bin/bash

 

Part 5 - Linking applications together on OSG-Connect

The p5.swift introduces a postprocessing step. After all the parallel simulations have completed, the files created by simulation.sh will be averaged by stats.sh. This is similar to p3, but all app invocations are done on remote nodes with Swift managing file transfers.  The workflow and source follow:

p5.swift
type file;
# Define external application programs to be invoked
file simulation_prog <"app/simulate.sh">;
file analysis_prog   <"app/stats.sh">;
app (file out, file log) simulation
    (file prog, int timesteps, int sim_range,
     file bias_file, int scale, int sim_count)
{
  sh @prog "-t" timesteps "-r" sim_range "-B" @bias_file
           "-x" scale "-n" sim_count
           stdout=@out stderr=@log;
}
app (file out) analyze (file prog, file s[])
{
  sh @filename(prog) @filenames(s) stdout=@filename(out);
}
# Command line params to this script
int  nsim  = @toInt(@arg("nsim",  "10"));  # number of simulation programs to run
int  steps = @toInt(@arg("steps", "1"));   # number of "steps" each simulation (==seconds of runtime)
int  range = @toInt(@arg("range", "100")); # range of the generated random numbers
int  count = @toInt(@arg("count", "10"));  # number of random numbers generated per simulation
# Perform nsim "simulations"
tracef("\n*** Script parameters: nsim=%i steps=%i range=%i count=%i\n\n", nsim, steps, range, count);
file sims[];                               # Array of files to hold each simulation output
file bias<"bias.dat">;                     # Input data file to "bias" the numbers:
                                           # 1 line: scale offset ( N = n*scale + offset)
foreach i in [0:nsim-1] {
  file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
  file simlog <single_file_mapper; file=@strcat("output/sim_",i,".log")>;
  (simout, simlog) = simulation(simulation_prog, steps, range, bias, 100000, count);
  sims[i] = simout;
}
# Generate "analysis" file containing average of all "simulations"
file stats<"output/stats.out">;
stats = analyze(analysis_prog,sims);

 

To run:

 

Running p4.swift
$ cd part05
$ swift p5.swift

The output will be named output/stats.out.

Output of p2.swift
$ cat output/stats.out
5143382

Part 6 - Specifying more complex workflow patterns

The p6.swift script builds on p5.swift, but adds new apps for generating a random seed and a random bias value.  The script's control flow and source is shown below:


p6.swift
type file;
# Define external application programs to be invoked
file simulation_prog <"app/simulate.sh">;
file analysis_prog   <"app/stats.sh">;
file genbias_prog = simulation_prog;
file genseed_prog = simulation_prog;
# app() functions for application programs to be called:
app (file out) genseed (file prog, int nseeds)
{
  sh @prog "-r" 2000000 "-n" nseeds stdout=@out;
}
app (file out) genbias (file prog, int bias_range, int nvalues)
{
  sh @prog "-r" bias_range "-n" nvalues stdout=@out;
}
app (file out, file log) simulation (file prog, int timesteps, int sim_range,
                                     file bias_file, int scale, int sim_count)
{
  sh @prog "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale
           "-n" sim_count stdout=@out stderr=@log;
}
app (file out) analyze (file prog, file s[])
{
  sh @prog @filenames(s) stdout=@out;
}
# Command line arguments
int  nsim  = @toInt(@arg("nsim",  "10"));  # number of simulation programs to run
int  range = @toInt(@arg("range", "100")); # range of the generated random numbers
int  count = @toInt(@arg("count", "10"));  # number of values generated per simulation
int  steps = @toInt(@arg("steps", "1"));   # number of timesteps (seconds) per simulation
# Main script and data
tracef("\n*** Script parameters: nsim=%i range=%i count=%i\n\n", nsim, range, count);
file seedfile<"output/seed.dat">;        # Dynamically generated bias for simulation ensemble
seedfile = genseed(genseed_prog, 1);
int seedval = readData(seedfile);
tracef("Generated seed=%i\n", seedval);
file sims[];                      # Array of files to hold each simulation output
foreach i in [0:nsim-1] {
  file biasfile <single_file_mapper; file=@strcat("output/bias_",i,".dat")>;
  file simout   <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
  file simlog   <single_file_mapper; file=@strcat("output/sim_",i,".log")>;
  biasfile = genbias(genbias_prog, 1000, 20);
  (simout,simlog) = simulation(simulation_prog, steps, range, biasfile, 100000, count);
  sims[i] = simout;
}
file stats<"output/stats.out">;            # Final output file: average of all "simulations"
stats = analyze(analysis_prog,sims);

To run:

 

Running p6.swift
$ cd part06
$ swift p6.swift

The output will be named output/stats.out.

Verifying output
$ cat output/stats.out
4407482

 

Further information and refernces