Registration Guide Register for HPC Access to HPCNew User Guide How to Run Batch Jobs Portal User Guide About the Service Subscribe to Pay-Per-Use Service Subscribe to Condominium Service Terms and ConditionsUser GuideHigh Performance Workspace Service Utility Storage ServiceResearch Data Management ServiceCentre for Computational Science & Engineering
 |
||
| Need help? | ||
|
||
 |
||
Checking SVU Cluster status (bhosts, bqueues, lsmon)
All commands provided by LSF can be typed from the command prompt. Detailed description of the commands and their optional arguments can be found in their man pages (viewable by typing: man command ). The bhosts command will show you the hosts in the SVU cluster:
> bhosts
Example output:
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV atlas2 ok - - 8 8 0 0 0 atlas2-1 ok - - 5 5 0 0 0 atlas2-c00 ok - - 1 1 0 0 0 atlas2-c01 ok - - 2 2 0 0 0 ... atlas2-c18 ok - - 1 1 0 0 0 atlas2-c19 ok - - 1 1 0 0 0 atlas3-c00 ok - - 0 0 0 0 0 atlas3-c01 ok - - 41 41 0 0 0 ... atlas3-c62 ok - 4 3 3 0 0 0 atlas3-c63 closed - 4 0 0 0 0 0 atlas4-c01 ok - - 2 2 0 0 0 atlas4-c02 ok - 6 6 6 0 0 0 ... atlas4-c87 ok - 6 1 1 0 0 0 atlas4-c88 ok - 6 2 2 0 0 0 atlas5-c01 ok - - 2 2 0 0 0 atlas5-c02 ok - 2 2 2 0 0 0 ... Atlas5-c95 ok - 6 1 1 0 0 0 Atlas5-c96 ok - 6 2 2 0 0 0 venus00 ok - - 1 1 0 0 0 venus01 ok - - 0 0 0 0 0 ... venus15 ok - - 1 1 0 0 0 venus16 ok - - 4 4 0 0 0
The bqueues command shows you the available queues in the cluster:
> bqueues
Example output:
QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP
monitor 45 Open:Active - - - - 0 0 0 0
esvu 44 Open:Active - 6 - - 0 0 0 0
linux64login 44 Open:Active - 6 - - 1 0 1 0
atlas4login 44 Open:Active - 6 - - 21 0 21 0
atlas5login 44 Open:Active - 6 - - 2 0 2 0
vislogin 44 Open:Active - 4 - 6 2 0 2 0
atlas3_large 43 Open:Active - 2 - 2 9 9 0 0
atlas4_large 43 Open:Active - 2 - 2 5 0 5 0
atlas4_128 43 Open:Active - 1 - 1 12 11 1 0
atlas5_large 43 Open:Active - 2 - 2 10 4 6 0
atlas3_parallel 43 Open:Active - 16 2 3 0 0 0 0
atlas4_parallel 43 Open:Active - 16 - 6 0 0 0 0
atlas5_parallel 43 Open:Active - 32 - 6 0 0 0 0
atlas3_parallel 43 Open:Active - 24 2 3 44 0 44 0
atlas4_parallel 43 Open:Active - 16 - 6 230 6 224 0
atlas5_parallel 43 Open:Active - 32 - 6 540 76 464 0
g03_linda 43 Open:Active - 16 - 2 0 0 0 0
g09_linda 43 Open:Active - 16 - 2 0 0 0 0
q_tgauss 40 Open:Active 5 1 - 1 0 0 0 0
normal 30 Open:Inact - - - - 4 4 0 0
linux64 30 Open:Active - 10 - 2 168 13 155 0
PRIO shows the priority of the queues. A job that is run from a queue with higher PRIO number will have higher priority when running and may be able to pre-empt jobs that are lower in priority. A queue has 2 status attributed to it. An "Open" status indicates that it is open for job submission, allowing users to submit and queue a job using the bsub command. An "Active" status indicates that the jobs in the queue can be launched to and started on the appropriate server. For example, an open, inactive queue will continue to accept jobs from users, but will not launch the jobs until the queue is activated by the administrators or by the system.
A more specific variation of bqueues can be used to determine the queues that are serving a particular system. For example, if you are interested to find out which queue to use if you want to run a job on the large memory compute server atlas4-c88, type the following command:
> bqueues -m atlas4-c87
Output:
QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP
monitor 45 Open:Active - - - - 0 0 0 0
esvu 44 Open:Active - 6 - - 0 0 0 0
atlas4_large 43 Open:Active - 2 - 2 5 0 5 0
atlas4_parallel 43 Open:Active - 16 - 6 0 0 0 0
atlas4_parallel 43 Open:Active - 16 - 6 230 6 224 0
g09_linda 43 Open:Active - 16 - 2 0 0 0 0
normal 30 Closed:Inact - - - - 4 4 0 0
linux64 30 Open:Active - 10 - 2 168 13 155 0
bqueues with the -l option and the queue name will give the details of a particular queue:
> bqueues -l atlas4_parallel
Output:
QUEUE: atlas4_parallel
-- For Parallel Jobs on Atlas4 Linux Cluster systems.
PARAMETERS/STATISTICS
PRIO NICE STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SSUSP USUSP RSV
43 16 Open:Active - 32 - 6 308 32 276 0 0 0
Interval for a host to accept two jobs is 100 seconds
PROCLIMIT
4 4 64
SCHEDULING PARAMETERS
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -
adapter_windows ntbl_windows poe css0 csss gm_ports
loadSched - - - - - -
loadStop - - - - - -
SCHEDULING POLICIES: NO_INTERACTIVE
DEFAULT HOST SPECIFICATION: atlas4-c01
USERS: all
HOSTS: atlas4cluster/
ADMINISTRATORS: lsfadmin ccefoog ccetancc cceyeoeh ccezxh ccewjh
PRE_EXEC: /usr/lsf62/mnt/svuscripts/batchsvr_pre_exec.sh
RES_REQ: select[type==X86_64 && model==PC1133] span[ptile=2]
PREEMPTION: PREEMPTABLE[linux64login]
The parameter JL/U indicates the maximum number of CPU cores a user can run at this queue.
Checking LSF hosts load status
The status of the server loads can be checked using the lsload command as shown:
> lsload
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem
atlas2-c13 ok 0.0 2.0 2.0 99% 0.0 0 1e+05 3784M 3818M 3984M
venus08 ok 0.0 1.3 1.1 0% 0.0 3 945 11G 8000M 3374M
atlas4-c53 ok 0.0 7.4 7.5 7% 364.5 0 70464 9824M 8192M 58M
atlas2-c01 ok 0.1 2.0 2.0 100% 0.0 0 93696 3784M 3816M 4046M
atlas3-c09 ok 0.2 4.0 4.1 90% 0.0 0 92160 9608M 16G 4948M
venus11 ok 0.2 0.8 0.5 0% 0.0 4 5224 9712M 8000M 3460M
atlas3-c61 ok 0.3 3.5 3.8 52% 0.0 0 92160 9592M 16G 4980M
venus00 ok 0.4 1.3 1.0 1% 0.0 3 8164 10G 8000M 7424M
...
atlas2-c19 ok 0.5 0.8 1.0 41% 0.0 0 1e+05 3784M 3808M 3950M
atlas3-c27 ok 0.7 3.7 3.7 48% 0.0 0 92160 9592M 16G 4972M
atlas2-c04 ok 1.0 2.0 1.9 100% 0.0 1 90752 3784M 3816M 3656M
atlas3-c06 ok 6.0 6.0 6.0 100% 0.0 0 31376 9448M 16G 6924M
atlas3-c50 ok 7.0 7.1 7.0 100% 0.0 0 92160 9608M 16G 6912M
atlas3-c01 busy 0.3 *61.0 61.8 0% 0.0 13 5 9616M 16G 12G
atlas4-c86 busy 2.0 *16.2 15.2 89% 0.0 0 40352 9536M 16G 57G
atlas4-c51 busy 2.3 *8.4 8.2 92% 0.0 0 52672 9848M 16G 10G
The column ut indicates the overall CPU load on the host, and mem indicates the free memory on the host.
The status of the server loads can be monitored using the lsmon command as shown:
> lsmon -E
Hostname: atlas4-c01 Refresh rate: 10 secs
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem
atlas2-c01 ok 0.0 1.0 1.0 100% 0.0 0 93696 3784M 3816M 4054M
atlas2-c05 ok 0.0 0.0 0.0 0% 0.0 0 1506 3784M 4094M 4496M
atlas2-c13 ok 0.0 1.0 1.0 100% 0.0 0 1e+05 3784M 3818M 3986M
atlas4-c67 ok 0.0 0.0 0.0 0% 0.0 0 92160 9848M 16G 8052M
atlas3-c63 ok 0.0 0.0 0.0 0% 0.0 0 11240 9720M 16G 6964M
atlas2-c12 ok 0.0 1.0 1.0 100% 0.0 0 1e+05 3784M 3812M 4200M
venus11 ok 0.0 0.1 0.1 1% 0.0 4 5224 9712M 8000M 3462M
venus10 ok 0.0 0.0 0.0 0% 0.0 4 165 11G 7892M 3508M
venus09 ok 0.0 0.0 0.0 0% 0.0 0 24624 11G 8000M 3490M
venus06 ok 0.0 0.0 0.1 0% 0.0 1 6832 11G 8000M 3522M
venus02 ok 0.0 0.0 0.1 0% 0.0 2 98 11G 8000M 3444M
atlas3-c00 ok 0.0 0.0 0.0 0% 0.0 0 61536 9512M 16G 31G
atlas4-c15 ok 0.0 1.0 0.9 25% 0.0 0 60416 9848M 16G 15G
atlas2-c03 ok 0.0 1.0 1.0 100% 0.0 0 73408 3784M 3812M 4260M
atlas4-c81 ok 0.0 0.2 0.7 13% 0.0 9 3 9640M 16G 60G
atlas4-c01 ok 0.1 0.0 0.1 1% 0.0 10 2 9536M 16G 15G
To monitor the particular host, for example atlas4-c02, the following command can be used:
> lsmon –E atlas4-c02
