Best practices for running jobs

This page outlines best practices for running jobs on OSG Connect.  This page covers a few different areas.


Job Submission directory:

  • For some scale testing and development, users can use a directory in their home directory area, however for production usage where the output returned is likely to be large, users should use a directory in /stash or ~/stash to hold the logs and output from jobs
  • If users anticipate creating a large number of files (e.g. greater than 50,000) within a single directory, please contact the OSG Connect administrators so that we can create a special area for this

Transferring Files:

  • The condor file transfer mechanism should not be used if the total amount of files being staged in or out to jobs is over 1GB.  In these situations the job should do a http transfer for staging in data and/or use a tool like connect-copy to stage data in or out
  • A tarball should be used to aggregate multiple small files rather than trying to transfer 100+ small files

Other best practices:

  • Job submissions should be done in batches so that the total number of jobs in the condor job queue remains below 25,000 jobs