Batchjobs on Freke

All jobs on Freke must be executed as batchjobs through the queueing system, PBS Pro.
All jobs must be submitted with a apropriate resource request (see below). Currently these queues are defined:

 

Queue	Description	Limits/remarks
qgen	This is the default queue. Jobs in this queue are routed either to q1 (the serial queue) if no (or one) CPU is requested, or to qpar if two or more CPU's are requested.	Max. number of CPU's pr. job is 32. Parallel and high priority jobs suspends serial jobs.
q1	The serial queue. Only available via qgen	Running jobs in this queue are subjects to suspension.
qpar	The parallel queue. Only available via qgen	Jobs to this queue may suspend jobs in q1. Max 40 CPUs in total available to this queue.
qexp	High priority queue for short (test) jobs.	Max 8 CPU's for this queue in total. Each running job are limitted to 1 hour wallclock. Preempts running jobs in q1.
qncwh	High priority queue for special jobs.	For validated users only. Contact Staff for access to this queue.

When a job starts, a uniq directory will be created in the /scratch filesystem. You can refer to this directory via the SCRDIR
environment variable as indicated in the jobscript below.
When the job terminates the scratch-directory and its contents is automatically erased.

Usefull commands for handling batchjobs on Freke:

Submit jobs to the system:

% qsub jobscript
Submits a 1 CPU job
% qsub -l ncpus=N jobscript
Submits a job requiering N CPU's.
% qsub -l ncpus=4,mem=7g jobscript
Submits a job requiering 4 CPU's and 7 GB of memory in total.
% qsub -q qexp -l ncpus=N jobscript
Submits a short testjob requiering N CPU's.
jobscript is the name of the job cammand file.

Delete a job:

% qdel jobid
- where jobid is the uniq identifier of the job, which is showed in the js command output.

Display jobs in the queues:

% js - displays jobs in the queues
% bj [-u] - show the number of CPUSs allocated to groups (and users: -u)
% qs - show the queue status.

Example of a jobscript for a serial batchjob:

#!/bin/csh

cd path/to/my/data
cp *.dat $SCRDIR
cp prog.exe $SCRDIR

cd $SCRDIR
./prog.exe > out

cp out ~/path/to/my/data
#

This job can be submitted with this command:
qsub jobscript

Example of a parallel (MPI-) jobscript:

#!/bin/csh

cd $SCRDIR
mpirun -np 6 /path/to/mpiprogram arg1 arg2 > outdata
grep Result: outdata
#

This job must be submitted with this command:
qsub -l ncpus=6 jobscript

Example of a jobscript for a parrallel (OpenMP) job:

#!/bin/csh

setenv OMP_NUM_THREADS 6
/path/to/par-program arg1 arg2 > outdata
grep Result: outdata
#

This job must be submitted with this command:
qsub -l ncpus=6 jobscript