...
The
...
purpose
...
of
...
the
...
queuing
...
system
...
is
...
(1)
...
to
...
promote
...
the
...
efficient
...
utilization
...
of
...
our
...
computer
...
facilities,
...
and
...
(2)
...
to
...
promote
...
equity
...
of
...
access
...
to
...
those
...
resources
...
across
...
our
...
user
...
community.
...
This
...
page
...
is
...
designed
...
to
...
help
...
users
...
get
...
started
...
with
...
this
...
system.
...
How
...
it
...
works
...
for
...
users
...
Users
...
should
...
edit
...
their
...
shell
...
scripts
...
to
...
add
...
special
...
directives
...
to
...
the
...
queue
...
system,
...
beginning
...
with
...
"#PBS",
...
that
...
request
...
resources,
...
declare
...
a
...
required
...
walltime,
...
and
...
direct
...
standard
...
output
...
and
...
error.
...
Users
...
can
...
then
...
"submit"
...
their
...
job
...
using
...
commands
...
like
...
"qsub,"
...
as
...
described
...
in
...
the
...
...
...
.
...
Advanced
...
details
...
can
...
be
...
found
...
in
...
...
...
of
...
the
...
Torque
...
user
...
manual.
...
A
...
simple
...
example
...
of
...
such
...
a
...
script
...
can
...
be
...
found
...
in
...
the
...
attached apoa.sh
...
,
...
which
...
runs
...
the
...
...
...
.
...
Other
...
examples
...
have
...
been
...
posted
...
on
...
the
...
...
...
page,
...
including
...
an
...
example
...
documenting
...
how
...
to
...
submit
...
multiple
...
similar
...
jobs,
...
using
...
Torque's
...
JobID
...
Array
...
functionality.
...
A
...
job
...
name
...
is
...
assigned
...
with
...
a
...
"#PBS
...
-N"
...
statement,
...
the
...
destination
...
queue
...
is
...
specified
...
using
...
a
...
"#PBS
...
-q"
...
statement:
| Code Block |
|---|
} #PBS -N solution_equilibration_273K { |
| Code Block |
|---|
code} {code} #PBS -q short {code} |
The
...
standard
...
output
...
and
...
standard
...
error
...
can
...
be
...
directed
...
to
...
files
...
using
...
the
...
"#PBS
...
-o"
...
or
...
"#PBS
...
-e"
...
directives.
...
These
...
two
...
streams
...
can
...
be
...
joined
...
using
...
a
...
"#PBS
...
-j
...
oe"
...
directive.
| Code Block |
|---|
} #PBS -e solution_equil.err #PBS -o solution_equil.log {code} |
Users
...
can
...
request
...
resources
...
using
...
a
...
"#PBS
...
-l"
...
statement.
...
Resources
...
include
...
the
...
walltime
...
(in
...
mm:ss
...
or
...
hh:mm:ss
...
format)
...
and
...
the
...
number
...
of
...
nodes
...
and
...
number
...
of
...
processors
...
per
...
node.
...
In
...
the
...
example
...
below,
...
several
...
alternative
...
examples
...
of
...
node
...
requests
...
are
...
given
...
to
...
illustrate
...
the
...
possible
...
syntax;
...
only
...
one
...
would
...
be
...
included
...
in
...
an
...
actual
...
script.
| Code Block |
|---|
} #PBS -l walltime=14:30:00 { |
| Code Block |
|---|
} {code} #PBS -l nodes=1:ppn=4:group0 OR #PBS -l nodes=1:ppn=8:group0 OR #PBS -l nodes=n024:ppn=8 OR #PBS -l nodes=1:ppn=8+nodes=1:ppn=4 OR #PBS -l nodes=n024:ppn=8+nodes=1:ppn=8 OR #PBS -l nodes=1:ppn=8:mem16 {code} |
Multiple
...
job
...
submission
...
through
...
job
...
arrays
...
is
...
available
...
on
...
Cyrus1,
...
but
...
not
...
Quantum2
...
(which
...
has
...
an
...
earlier
...
version
...
of
...
Torque.
...
This
...
feature
...
can
...
be
...
especially
...
helpful
...
to
...
users
...
who
...
need
...
to
...
submit
...
a
...
large
...
number
...
of
...
similar
...
jobs.
...
For
...
more
...
details,
...
see
...
section
...
2.1.1
...
of
...
the
...
...
...
Some or all of these arguments can also be given at the command line. Command-line settings override any settings in the script.
| Code Block |
|---|
|http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml]. Some or all of these arguments can also be given at the command line. Command-line settings override any settings in the script. {code} [bashprompt]$ qsub -q short -l walltime=5:00:00 -l nodes=2:ppn=8 -e test_stderr.txt ./test_simulation.sh {code} |
Some
...
notes
...
and
...
suggestions
...
for
...
users:
...
- Users
...
- should
...
- request
...
- the
...
- lowest
...
- possible
...
- walltime
...
- for
...
- their
...
- jobs,
...
- since
...
- the
...
- queue
...
- system
...
- will
...
- need
...
- to
...
- "block
...
- out"
...
- the
...
- entire
...
- 24-hour
...
- period
...
- when
...
- no
...
- walltime
...
- is
...
- specified.
...
- This
...
- is
...
- analogous
...
- to
...
- a
...
- customer
...
- arriving
...
- at
...
- a
...
- busy
...
- barbershop
...
- and
...
- explaining
...
- that
...
- he
...
- only
...
- needs
...
- a
...
- "very
...
- quick
...
- trim."
...
- If
...
- users
...
- do
...
- not
...
- specify
...
- a
...
- queue
...
- in
...
- their
...
- script,
...
- the
...
- "default"
...
- queue
...
- is
...
- used.
...
- This
...
- queue
...
- has
...
- a
...
- walltime
...
- limit
...
- of
...
- 24
...
- hours,
...
- a
...
- node
...
- count
...
- limit
...
- of
...
- 1,
...
- and
...
- a
...
- default
...
- priority
...
- of
...
- 60;
...
- in
...
- other
...
- words,
...
- it
...
- is
...
- less
...
- desirable
...
- than
...
- any
...
- of
...
- the
...
- queues
...
- listed
...
- in
...
- the
...
...
...
- .
...
- Specifying
...
- a
...
- queue
...
- with
...
- the
...
- "#PBS
...
- -q"
...
- option
...
- is
...
- a
...
- good
...
- idea.
...
- Short
...
- "analysis
...
- scripts"
...
- are
...
- not
...
- exempt
...
- from
...
- the
...
- queuing
...
- system.
...
- If
...
- your
...
- script
...
- runs
...
- anything
...
- other
...
- than
...
- Linux
...
- utilities
...
- (e.g.
...
- VMD,
...
- CHARMM,
...
- custom
...
- C
...
- code
...
- for
...
- MD
...
- analysis),
...
- it
...
- belongs
...
- in
...
- the
...
- queue
...
- system.
...
- Just
...
- add
...
- a
...
- few
...
- lines
...
- as
...
- discussed
...
- in
...
- the
...
- section
...
- above.
...
- Input
...
- and
...
- output
...
- to
...
- files
...
- do
...
- not
...
- seem
...
- to
...
- be
...
- as
...
- immediate
...
- as
...
- when
...
- running
...
- directly
...
- from
...
- the
...
- command
...
- line.
...
- Users
...
- should
...
- not
...
- count
...
- on
...
- immediate
...
- access
...
- to
...
- program
...
- output.
...
- Users
...
- should
...
- test
...
- system
...
- scaling
...
- before
...
- expanding
...
- beyond
...
- one
...
- node;
...
- for
...
- systems
...
- of
...
- 10
...
- to
...
- 20
...
- katoms,
...
- poor
...
- scaling
...
- has
...
- been
...
- observed
...
- beyond
...
- 8
...
- ppn,
...
- while
...
- the
...
- 92
...
- katom
...
- ApoA1
...
- benchmark
...
- case
...
- scales
...
- well
...
- to
...
- 2
...
- nodes
...
- (and
...
- possibly
...
- beyond).
...
| Anchor | ||||
|---|---|---|---|---|
|
...
Managing jobs in the queuing system, and the approach used by Torque
The following commands can be used to submit and manage jobs:
command | purpose |
|---|---|
qsub jobscript | Submit job in script jobscript. Can accept other arguments as discussed above. |
qsub -I -l nodes=1:ppn=4 |
...
Request |
...
interactive |
...
job |
...
with |
...
indicated |
...
resources. |
...
qdel jobID | Delete job number jobID. Seems to kill processes on compute nodes cleanly. |
showq | list jobs by state, with scheduling details. |
showbf | list "backfill," or number of processors available at the moment. |
qstat | List active jobs. Use "-i" for a different format. |
qstat -q | List queue attributes for all queues. |
qstat -f jobID | List detailed information for job number jobID. |
qnodes | List all nodes and their state and properties. |
qnodes -l down | List those nodes currently down. |
qnodes -l active or qnodes -l active | List nodes currently used for jobs. |
qnodes -l free | List nodes currently free. |
qmgr -c "print server" | Print queue configuration details |
administrative command | purpose |
|---|---|
pbsnodes -o nodename | Take node offline after allowing current job to finish. |
pbsnodes -r nodename | Check node and then return to free status if possible. |
checkjob jobid | Check status of job. |
releasehold -a jobid | Release holds on job. |
qrun jobid | Force job to execute now. |
showstats | Show usage statistics. |
showgrid statistic_name | Print a text table of statistic_name by time. |
The PBS queue system allocates a set of nodes and processors to an individual job, either for the walltime specified in the job or the maximum walltime in the queue. It then provides a set of environmental variables to the shell in which the script runs, such as PBS_NODEFILE, the temporary node file describing allocated CPUs. When running with OpenMPI's mpiexec, the submitted script seems to just launch processes without needing that nodefile specified as an argument to mpiexec, although it's not clear whether that behavior is a feature or a bug.
Additional information for queue administrators on PBS and Maui can be found on a helpful page maintained by the University of Cambridge's Theoretical Chemistry department.
| Anchor | ||||
|---|---|---|---|---|
|
Table of queues attributes
The following tables were updated on June 11th to reflect changes made on that date. Note that a dash ("--") indicates no limit for the given queue.
Queue attributes on Cyrus1
| debug | short | long |
|---|---|---|---|
max walltime | 20 min | 24 hr | 5 days |
max nodes per job | 1 | 5 | 1 |
max nodes per queue | – | – | 10 |
max jobs running per user | – | – | 5 |
max jobs queuable per user | – | – | - |
priority | 100 | 80 | 60 |
Queue attributes on Darius1, Darius2, Xerxes and Artaxerxes
| debug | short | long | verylong |
|---|---|---|---|---|
max walltime | 20 min | 24 hr | 5 days | 14 days |
max nodes per job | 1 | 32 | 4 | 4 |
max nodes per queue | – | – | 15 | 5 |
max jobs running per user | – | – | 15 | 5 |
priority | 100 | 80 | 60 | 40 |