Versions Compared

Old Version 35

changes.mady.by.user Suchandra Thapa

Saved on

compared with

New Version Current

changes.mady.by.user Suchandra Thapa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin
h2. Overview In this module we demonstrate job submission to the OSG Connect environment from your laptop with [BOSCO|http://bosco.opensciencegrid.org]. This will allow you to manage jobs running on OSG from your familiar environment. It does not have to be a laptop: Any Linux or Mac host can be used provided it runs RHEL5 or RHEL6 (and Scientific Linux distributions), Debian 6, or Mac OS X (10.5 or later). Here is a diagram showing a picture of how the different resources are connected when you install BOSCO on your
Panel
titleTable of Contents
Table of Contents
outlinetrue
maxlevel4
indent1em
stylenone
Wiki Markup

Overview

In this module we demonstrate job submission to the OSG Connect environment from your laptop with BOSCO. This will allow you to manage jobs running on OSG from your familiar environment. It does not have to be a laptop: Any Linux or Mac host can be used provided it runs RHEL5 or RHEL6 (and Scientific Linux distributions), Debian 6, or Mac OS X (10.5 or later).

Here is a diagram showing a picture of how the different resources are connected when you install BOSCO on your laptop:

Warning

Jobs running on OSG Connect must have a project name set ( +ProjectName in the HTCondor submit file). The best way to do that is by adding in your home directory a file with your default project name: $HOME/.osg_default_project

For example on login.osgconnect.net do: echo ConnectTrain > $HOME/.osg_default_project to use the ConnectTrain account.  Once you create a project, you should replace the ConnectTrain with your project name.

...

  • Download the BOSCO Quickstart Multi-Platform installer from this download page. If you prefer to work in a terminal window, you can also copy the URL that will be printed in the download page and use cURL:

    No Format
    [user@laptop Downloads]$ curl -o bosco_quickstart.tar.gz ftp://ftp.cs.wisc.edu/GET_THE_URL_FROM_THE_PAGE/bosco_quickstart.tar.gz
    Code Block
    languagebash
    titlecurl detailed output
    collapsetrue
    [user@laptop Downloads]$ curl -o bosco_quickstart.tar.gz ftp://ftp.cs.wisc.edu/condor/bosco/1.2/bosco_quickstart.tar.gz 
% Total % Received % Xferd Average Speed Time Time Time Current
                           Dload Upload Total Spent Left Speed
100 2551 100 2551 0 0 4398 0 --:--:-- --:--:-- --:--:-- 10585
    Note

    If you have no curl you can use wget to download the file: 

    wget -O ./bosco_quickstart.tar.gz ftp://ftp.cs.wisc.edu/GET_THE_URL_FROM_THE_PAGE/bosco_quickstart.tar.gz 

...

...
  • Untar 
...
  •  the 
...
  •  bosco_quickstart 
...
  •  script 
...
  •  from 
...
  •  a 
...
  •  terminal 
...
  •  with 
...
  •  a 
...
  •  current 
...
  •  working 
...
  •  directory 
...
  •  of 
...
  •  the 
...
  • ~/Downloads
...
  •  folder 
...
  •  or 
...
  •  the 
...
  •  folder 
...
  •  in 
...
  •  which 
...
  •  you 
...
  •  saved 
...
  •  the 
...
  •  file: 
...
  •  No Format
...
  • [user@laptop Downloads]$ tar xvzf ./bosco_quickstart.tar.gz

...
  • Run the quickstart script and answer the questions. 
     No Format
    [user@laptop Downloads]$ ./bosco_quickstart

...
    • When prompted "Do you want to install Bosco? Select y/n 
...
    •  and 
...
    •  press 
...
    •  [ENTER
...
    • ]:
...
    • " 
...
    •  press 
...
    •  "y" 
...
    •  and 
...
    •  ENTER.
...
    • When 
...
    •  prompted 
...
    •  "
...
    • Type 
...
    •  the 
...
    •  cluster 
...
    •  name 
...
    •  and 
...
    •  press 
...
    •  [ENTER
...
    • ]:
...
    • " 
...
    •  type 
...
    • login.osgconnect.net
...
    •  and 
...
    •  press 
...
    •  ENTER.
...
    • When 
...
    •  prompted 
...
    •  "
...
    • Type 
...
    •  your 
...
    •  name 
...
    •  at 
...
    •  login.osgconnect.net 
...
    •  (default 
...
    •  YOUR_USER) 
...
    •  and 
...
    •  press 
...
    •  [ENTER
...
    • ]:
...
    • " 
...
    •  enter 
...
    •  your 
...
    •  user 
...
    •  name 
...
    •  on 
...
    •  OSG-Connect 
...
    •  and 
...
    •  press 
...
    •  ENTER.
...
    • When 
...
    •  prompted 
...
    •  "
...
    • Type 
...
    •  the 
...
    •  queue 
...
    •  manager 
...
    •  for 
...
    •  login.osgconnect.net 
...
    •  (pbs, 
...
    •  condor, 
...
    •  lsf, 
...
    •  sge, 
...
    •  slurm) 
...
    •  and 
...
    •  press 
...
    •  [ENTER
...
    • ]:
...
    • " 
...
    •  enter 
...
    • condor
...
    •  and 
...
    •  press 
...
    •  ENTER.
...
    • Then 
...
    •  when 
...
    •  prompted 
...
    •  "
...
    • user@login.osgconnect.net's 
...
    •  password:
...
    • " 
...
    •  enter 
...
    •  your 
...
    •  OSG-Connect 
...
    •  user 
...
    •  password.
...
* 
 Code Block
titleclick here to see the output
collapsetrue
[user@laptop Downloads]$ ./bosco_quickstart
Bosco Quickstart
Detailed logging of this run is in ./bosco_quickstart.log
Bosco is not installed. You need Bosco to run this quickstart.
Do you want to install Bosco? Select y/n and press [ENTER]): y
************** Downloading and Installing Bosco ***********
Installing BOSCO.......
BOSCO Installed
************** Starting Bosco: ***********
BOSCO Started
************** Connect one cluster (resource) to BOSCO: ***********
At any time hit [CTRL+C] to interrupt.
Type the submit host name for the BOSCO resource and press [ENTER]: login.osgconnect.net
Type your username on login.osgconnect.net (default marco) and press [ENTER]: user
Type the queue manager for login.osgconnect.net (pbs, condor, lsf, sge, slurm) and press [ENTER]: condor
Connecting login.osgconnect.net, user: user, queue manager: condor
user@login.osgconnect.net's password:
.............................................
login.osgconnect.net connected
************** Testing the cluster (resource): ***********
This may take up to 2 minutes... please wait.............................................................
BOSCO on login.osgconnect.net Tested
************** Congratulations, Bosco is now setup to work with login.osgconnect.net! ***********
You are ready to submit jobs with the "condor_submit" command.
Remember to setup the environment all the time you want to use Bosco:
source ~/bosco/bosco_setenv
Here is a quickstart guide about BOSCO:
https://twiki.grid.iu.edu/bin/view/CampusGrids/BoscoQuickStart
To remove Bosco you can run:
source ~/bosco/bosco_setenv; bosco_uninstall --all
Here is a submit file example (supposing you want to run "myjob.sh"):
universe = grid
Executable = myjob.sh
arguments =
output = myjob.output.txt
error = myjob.error.txt
log = myjob.log
transfer_output_files =
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
queue 1
[user@laptop Downloads]$
 Wiki Markup
  • After 
...
  •  a 
...
  •  successful 
...
  •  installation, 
...
  •  before 
...
  •  changing 
...
  •  directory, 
...
  •  you 
...
  •  can 
...
  •  remove 
...
  •  the 
...
  •  installer 
...
  •  and 
...
  •  its 
...
  •  log 
...
  •  file: 
...
  •  No Format
...
  • [user@laptop Downloads]$ rm bosco_quickstart* 
...

  • Setup the environment 
     No Format
    [user@laptop ~]$ source ~/bosco/bosco_setenv 
...

  • BOSCO has been started for you but in the future you may need to restart it with: 
     No Format
    [user@laptop ~]$ bosco_start
BOSCO Started
...

  •  At this point, 
...
  •  submission 
...
  •  to 
...
  • login.osgconnect.net
...
  • , 
...
  •  which 
...
  •  gets 
...
  •  to 
...
  •  the 
...
  •  full 
...
  •  OSG-Connect 
...
  •  environment 
...
  •  is 
...
  •  now 
...
  •  ready. 
...
  •  The 
...
  •  BOSCO 
...
  •  services 
...
  •  will 
...
  •  remain 
...
  •  running 
...
  •  even 
...
  •  if 
...
  •  you 
...
  •  log 
...
  •  out 
...
  •  unless 
...
  •  explicitly 
...
  •  shut 
...
  •  down.
...
Each time setup the BOSCO environment
Each time you login or start a new shell stup the environment and invoke bosco_start (bosco_start is a no-op if the services are already running): 
 No Format
$ source ~/bosco/bosco_setenv
$ bosco_start
BOSCO Started{noformat}


h2. Create a tutorial directory

Create a new directory to run this tutorial and the log directory for the jobs:
{noformat}
Create a tutorial directory
Create a new directory to run this tutorial and the log directory for the jobs:
 No Format
$ mkdir -p tutorial-bosco/log
$ cd tutorial-bosco
{noformat}


h2. Submit a job to 
Submit a job to OSG-Connect
...
Now 
...
 run 
...
 a 
...
 simple 
...
 job, 
...
 like 
...
 the 
...
Job 
...
 1 
...
 of 
...
 the 
...
 Quickstart 
...
 tutorial 
...
. 
...
 The 
...
 workload 
...
 is 
...
 the 
...
 same, 
...
 the 
...
 submit 
...
 description 
...
 file 
...
 will 
...
 be 
...
 slightly 
...
 different.
...
Create 
...
 a 
...
 workload
...
Inside 
...
 the 
...
 tutorial 
...
 directory 
...
 that 
...
 you 
...
 created 
...
 or 
...
 installed 
...
 previously, 
...
 let's 
...
 create 
...
 a 
...
 test 
...
 script 
...
 to 
...
 execute 
...
 as 
...
 your 
...
 job 
...
 (remember 
...
 to 
...
 make 
...
 the 
...
 script 
...
 executable
...
!):
...
}
 No Format
$ vi short.sh
$ chmod +x short.sh
{noformat}

Here 
...
 is 
...
 the 
...
 content 
...
 of 
...
 short.sh:
...
: = }
 No Format
title
file: 
 short.sh
#!/bin/bash
# short.sh: a short discovery job

printf "Start time: "; /bin/date
printf "Job is running on node: "; /bin/hostname
printf "Job running as user: "; /usr/bin/id

echo "Environment:"
/bin/env | /bin/sort

echo "Dramatic pause..."
sleep ${1-15}    # Sleep 15 seconds, or however much we're told to sleep
echo "Et voila!"
{noformat}


h3. Create a condor submit file:

The next step is to create a submission file for the job.
{noformat}
Create a condor submit file:
The next step is to create a submission file for the job.
 No Format
$ vi bosco01.sub
{noformat}


Here 
...
 is 
...
 the 
...
 bosco01.sub 
...
 content, 
...
 configured 
...
 to 
...
 use 
...
 a 
...
 special 
...
project 
...
 name
...
 on 
...
login01.osgconnect.net
...
. 
...
 This 
...
 is 
...
 a 
...
 general 
...
 purpose 
...
 project 
...
 name 
...
 and 
...
 you 
...
 are 
...
 encouraged 
...
 to 
...
 use 
...
 one 
...
 of 
...
 the 
...
 projects 
...
 that 
...
 you 
...
 are 
...
 member 
...
 of. 
...
 You 
...
 can 
...
 see 
...
 the 
...
 projects 
...
 you 
...
 are 
...
 member 
...
 of 
...
 by 
...
 using 
...
 the 
...
osgconnect_show_projects
...
 command. 
...
 This 
...
 is 
...
 very 
...
 nearly 
...
 the 
...
 minimal 
...
 content 
...
 of 
...
 a 
...
 submission 
...
 file.
...

 Note 
...
 that 
...
 differently 
...
 from 
...
 the 
...
 previous 
...
 examples, 
...
 now 
...
 the 
...
Universe
...
 of 
...
 the 
...
 job 
...
 is 
...
 now 
...
grid
...
. 
...
 This 
...
 tells 
...
 BOSCO 
...
 to 
...
 run 
...
 the 
...
 job 
...
 on 
...
 the 
...
 resource 
...
 added 
...
 during 
...
 the 
...
 setup.
...
 }
 No Format
titlefile: 
 bosco01.sub
########################
# Submit description file for short test program
########################
Universe       = grid
Executable     = short.sh
Error   = log/job.err.$(Cluster)-$(Process)
Output  = log/job.out.$(Cluster)-$(Process)
Log     = log/job.log.$(Cluster)
+ProjectName="ConnectTrain"
Queue 1
{noformat}
 Code Block
titlesubmit file: bosco01.sub
collapsetrue
########################
# Submit description file for short test program
########################
# The UNIVERSE specifies class of workers will be used and how
# Condor will connect.  You will almost always use VANILLA.
# For BOSCO jobs you almost always use GRID
Universe = grid
# EXECUTABLE tells what program your job will run.  It's often useful
# to create a shell script to "wrap" your actual work.
Executable = short.sh
# ERROR and OUTPUT tell Condor how to name your job's standard error
# (stderr) and standard output (stdout).
Error = log/job.err.$(Cluster)-$(Process)
Output = log/job.out.$(Cluster)-$(Process)
# The LOG file is where Condors places information about your job's
# status, success, and resource consumption.
Log = log/job.log.$(Cluster)
# This option ensures that the tutorial queue is available for this
# job. It should not be used except in the tutorials.
+ProjectName="ConnectTrain"
# QUEUE is the "start button" - it launches any jobs that have been
# specified this far.
Queue 1
...