National University of Singapore

Release of the New HPC Cluster, New Parallel File Systems and New LSF Queues for Batch Job Submission

Dear HPC Users               

 We are pleased to inform you that a new HPC system, atlas5 Linux cluster, has been added to the pool of the NUS HPC compute resources. The atlas5 cluster has 96 compute nodes, each node is equipped with 2 quad-core Nehalem E5550 2.66GHz processors and 48GB of memory, so the whole cluster has 8.17 TFlops of compute power and 4,608 GB of memory.

 Together with the release of the atlas5 cluster, we are also pleased to announce that two new parallel GPFS file systems, atlas3tmp and atlas5tmp, each with 40TB of disk storage, will be available to the users.  From now on, the High Performance Workspace for Computational Clusters: atlas3tmp, atla4tmp and atlas5tmp, will serve as the working spaces for users to run I/O intensive applications on the specific clusters.

 To make good use of resources and manage the resources properly, we have upgraded the resource management software LSF from LSF 6.2 to LSF 7.0.  New queues are created and old queues have been modified to cater for different computational requirements.  Please read through the following instructions carefully before you submit your jobs the LSF batch queues:

 1.      LSF Queues

Different queues are configured to run different kinds of jobs. The following queues are available in LSF 7.  Do a “bqueues -l queue_name” to check the queue definitions.

Serial job queues: linux, linux64, ia64

Large memory serial job queues: atlas3_large, atlas4_large, atlas4_128, atlas5_large, ia64_large

Parallel job queues: atlas3_parallel, atlas4_parallel, atlas5_parallel, ia64_4p,

Parallel job test queues: atlas3_parallel_test, atlas4_parallel_test, atlas5_parallel_test

1. 2.      Where to Submit Your Jobs:

Jobs submitted to different queues will run on different clusters from different parallel file systems. You are advised to login to the specific Linux cluster, change directory to the specific parallel file system and submit your parallel or large memory jobs there to the specific LSF queues. For example, if you want to run your jobs on atlas4 cluster, login to atlas4-c01 or atlas4-c81,

> cd /atlas4tmp
> mkdir my_subdir
> cd my_subdir
> bsub -q atlas4_parallel -o myjob.eo -a mvapich -n 32 mpirun.lsf ./my_executable

 3.      File Systems and File Transfer

The user’s home directory (/home/svu/$userid) is a NFS mounted shared file system among all the HPC servers. You can access or copy files in your home directory on any servers.  Small serial jobs can be submitted to the linux64 queue from your home directory, but don’t submit  large and parallel jobs in your home directory to atlas5_large and atlas5_parallel,  this will affect the overall system performance of the atlas5 cluster, because we have a limited bandwidth between the home file system and the atlas5 clusters.

 The other two NSF mounted file systems /lsftmp and /hpctemp will be gradually phased out.  So refrain from using this two file system from now on. If you have files in these two files systems, copy and backup them in a safe place.

 The parallel file system on atlas3 cluster (/atlas3tmp) will also be remotely mounted on the login nodes of the other two Linux clusters atlas4 cluster and altas5 cluster. The same applies for the parallel file systems /atlas4tmp and /atlas5tmp. This is to facilitate file transferring between different parallel file systems. You should NOT try to submit jobs from /atlas3tmp to atlas4 cluster or atlas5 cluster, and vice versa.

1. 4.      Compile Your Code

Due to the different OS version and Infiniband model, executables compiled on atlas3 and atlas4 cluster may not run on atlas5 cluster. If you want to submit your jobs to the atlas5_parallel and atlas5_large, login to atlas5-c01 or atlas5-c49 to recompile your code.  The following compilers are available on the new atlas5 cluster:

Intel compilers: ifort, icc

MPI compilers: mpif90, mpif77, mpicxx, mpicc

GNU compilers: gfortran, g77, gcc

1. 5.      Jobs Submitted to LSF 6 Before Today

From now on, you’ll not be able to submit jobs to the LSF 6.  Those jobs submitted before today to the LSF 6 queues and are still running will be allowed to run until they are finished.  But for those jobs which are still pending, we advise users to stop them and re-submit them through LSF7 to the new LSF queues. You can expect you jobs be dispatched to the servers in a shorter time in the LSF 7 queues as a large among of systems are idle at the beginning of the system release.

 To check the jobs status and terminate the old jobs, please use LSF commands with a 6 as suffix, e.g.: bjobs6, bkill6, lsload6, etc.

 Please check the HPC website for more information and instructions. Should you have any problems or queries, don’t hesitate to contact us through ccesvuhelp@nus.edu.sg.

 Wish you enjoy using the new HPC resources and have a happy and auspicious Chinese New Year!