LSF

General

LSF (Platform Load Sharing Facility) is a Batch-Scheduler. To get an overview of the functionality of a scheduler, go here or to the Scheduling Basics.

#BSUB Usage

If you are writing a jobscript for an LSF batch system, the magic cookie is "#BSUB". To use it, start a new line in your script with "#BSUB". Following that, you can put one of the parameters shown below, where the word written in <...> should be replaced with a value.

Basic settings:

Parameter	Function
-J <name>	job name
-o <path>	path to the file where the job output is written
-e <path>	path to the file for the job error output (if not set, it will be written to output file as well)

Requesting resources:

Parameter	Function	Default
-W <runlimit>	runtime limit in the format [hour:]minute; once the time specified is up, the job will be killed by the scheduler	00:15
-M <memlimit>	memory limit per process in MB	512
-S <stacklimit>	limit of stack size per process in MB	10

Parallel programming (read more here):

Parameter	Function
-a openmp	start a parallel job for a shared-memory system
-n <num_threads>	number of threads to execute OpenMP application with
-a openmpi	start a parallel job for a distributed-memory system
-n <num_procs>	number of processes to execute MPI application with

Email notifications:

Parameter	Function
-B	send email to the job submitter when the job starts running
-N	send email to the job submitter when the job has finished
-u <email_address>	recipient of emails

Job Submission

This command submits the job you defined in your jobscript to the batch system. If the less-than sign < is left out, your job will be submitted, but all the resource requests in your jobscript will be ignored.

$ bsub < jobscript.sh

Just like any other incoming job, your job will first be queued. Then, the scheduler decides when your job will be run. The more resources your job requires, the longer it may be waiting to execute.

You can check the current status of your submitted jobs and their job ids with the following shell command. A job can either be pending PEND (waiting for free nodes to run on) or running RUN (the jobscript is currently being executed). If all of your jobs have finished execution, the command will print No unfinished jobs found.

$ bjobs

If you are interested in the current status of your job, you can try the utility bpeek. It prints the output which has already been written by your job:

$ bpeek <job_id>

In case you submitted a job on accident or realised that your job might not be running correctly, you can always remove it from the queue or terminate it when running by typing:

$ bkill <job_id>

Array and Chain Jobs

#BSUB -J "ChainJob[1-4]%1"

This creates an array job with 4 subjobs where only one may be executed at a time in a random order. An explicit order can be forced by either submitting each subjob at the end of the one before (which may prolong pending) or using the dependencies feature, which results in a chain job.

#BSUB -w <condition>

The set condition is a logical expression, which can also be a combination of multiple logical expressions, connected by the logical operators && (AND), || (OR) or ! (NOT). The entire condition needs to be satisfied for the job to be executed.

Jobscript Examples

This serial job will run a given executable, in this case "myapp.exe".

#!/usr/bin/env zsh

### Job name
#BSUB -J MYJOB

### File where the output should be written
#BSUB -o MYJOB_OUTPUT.txt

### Time your job needs to execute, e. g. 1 h 20 min
#BSUB -W 1:20

### Memory your job needs, e. g. 1000 MB 
#BSUB -M 1000

### Stack limit per process, e. g. 20 MB
#BSUB -S 20

### The last part consists of regular shell commands:
### Change to working directory
cd /home/user/mywork

### Execute your application
myapp.exe

This OpenMP job will start the parallel program "myapp.exe" with 24 threads.

#!/usr/bin/env zsh

### Job name
#BSUB -J OMPJOB

### File where the output should be written
#BSUB -o OMPJOB_OUTPUT

### Time your job needs to execute, e. g. 15 min
#BSUB -W 0:15

### Memory your job needs, e. g. 1000 MB 
#BSUB -M 1000

### Stack limit per process, e. g. 50 MB
#BSUB -S 50

### Request 24 compute slots (in this case: threads)
#BSUB -n 24

### Execute as shared-memory job
#BSUB -a openmp

### Change to working directory
cd /home/user/mywork

### Execute your application
myapp.exe

This OpenMPI job will start the parallel program "myapp.exe" with 4 processes.

#!/usr/bin/env zsh

### Job name
#BSUB -J MPIJOB

### File where the output should be written
#BSUB -o MPIJOB_OUTPUT

### Time your job needs to execute, e. g. 30 min
#BSUB -W 0:30

### Memory your job needs, e. g. 1024 MB 
#BSUB -M 1024

### Stack limit per process, e. g. 50 MB
#BSUB -S 50

### Request 4 compute slots (in this case: processes)
#BSUB -n 4

### Execute as distributed-memory job with OpenMPI
#BSUB -a openmpi

### Change to working directory
cd /home/user/mywork

### Execute your application
myapp.exe

References

More LSF jobscript examples