CAHL Cluster is a parallel computing facility originally housed at the Computational Aero & Hydrodynamics Laboratory (CAHL) in the Department of Mechanical and Aerospace Engineering at the George Washington University. It was initially deployed in 2011, and since then primarily managed by team members of the lab. In 2017, the team rebuilt the cluster to update its operating and provisioning systems. Several hard drive failures led to another rebuild in 2019 using OpenHPC's collection of open-source packages in a "stateless" way (OS runs in RAMs instead of on hard drives). The cluster now runs CentOS 7, employs Warewulf as the provisioning system, Slurm as the job management tool, and Lmod as the package management tool. The cluster serves as the main testing and debugging platform of the lab for developing parallel flow and structure solvers. In the Fall of 2019, the cluster was relocated to Clarkson University following the move of the lab.
Hardware
The cluster features a total number of 252 CPU cores, with 12 on the master node and 240 on the compute nodes. The detailed configurations are:
Compute nodes: 5 DELL™ PowerEdge C6100 chassis with a total number of 20 nodes, each node featuring dual Intel® Xeon X5670 2.93GHz 6-core processors and 36GB RAM
Storage device: DELL™ Powervault MD1200 12-bay storage chassis with 6 × 2TB hard disk drives installed
InfiniBand switch: Mellanox™ IS5035 36-port non-blocking 40Gb/s InfiniBand switch for inter-processor communication
User's Guide
To run jobs on the cluster, a user must have basic knowledge in at least three aspects: logging in the system, accessing preinstalled packages, and managing jobs.
Logging in
To login to the system, use the following command in your local terminal:
For Windows users, PuTTY is a good choice to serve as your local terminal. Alternatively, enable the Windows Subsystem for Linux (WSL) feature that comes with Windows for the purpose.
If one wants to use remote software (such as Tecplot) with graphic user interface (GUI), use the -X option to login:
Note that, the local system must have X-Window support for the remote GUI to display correctly. Mac and Linux come with X-Window support naturally. For Windows, third-party software, such as Xming, is required.
Finally, to transfer files between local and remote systems, use the scp or sftp command. One can also use GUI-based sftp clients, such as WinSCP or FileZilla, to transfer or sync files.
Package Management
This section describes how to access preinstalled packages, such as compilers, numerical libraries, and many other tools.
Show available packages
[user@cluster ~]$ module avail
----------------------------------- /opt/ohpc/pub/moduledeps/intel ------------------------------------
hdf5/1.10.4 metis/5.1.0 ocr/1.0.1 pdtoolkit/3.25 py3-numpy/1.15.3
impi/2019.3.199 mpich/3.3 openmpi/1.10.7 plasma/2.8.0 scotch/6.0.6
likwid/4.3.3 mvapich2/2.3 openmpi3/3.1.3 py2-numpy/1.15.3 superlu/5.2.1
-------------------------------------- /opt/ohpc/pub/modulefiles --------------------------------------
EasyBuild/3.8.1 clustershell/1.8 gnu8/8.3.0 pmix/2.2.2
Tecplot/2018r1 (L) cmake/3.13.4 hwloc/2.0.3 prun/1.3
autotools gnu/5.4.0 intel/19.0.3.199 singularity/3.1.0
charliecloud/0.9.7 gnu7/7.3.0 papi/5.6.0 valgrind/3.14.0
Where:
L: Module is loaded
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
Note that, this command does not necessarily display all the preinstalled packages. As the output suggests, one can use module spider to see a complete list.
Load a package
To compile, debug, or run a code, a user must first load the necessary packages. For example, one can use the following command to load the Intel compilers:
[user@cluster ~]$ module load intel/19.0.3.199
With that, the Intel Fortran compiler ifort, Intel C/C++ compiler icc, and the related Intel libraries are available for use.
To compile an MPI code, one must load one of the preinstalled MPI packages (mpich, mvapich2, openmpi, openmpi3). We recommend using the openmpi package on CAHL Cluster (for the best performance). The following command will load it:
[user@cluster ~]$ module load openmpi/1.10.7
After that, one can compile MPI Fortran code using mpifort or mpif90, MPI C code using mpicc, and MPI C++ code using mpicxx. It is worth noting is that, once an MPI package is loaded, an environment variable MPI_DIR is also created. This is helpful if one wants to manually link the MPI libraries to some applications.
The widely used domain decomposition package Metis is also preinstalled, and is loadable through:
[user@cluster ~]$ module load metis/5.1.0
Once loaded, commands such as gpmetis and mpmetis are available for use. Meanwhile, four environmental variables METIS_BIN, METIS_DIR, METIS_INC, and METIS_LIB are also created.
To automatically load these packages the next time a user logs into the system, one can add the above commands to the .bashrc file in his/her home directory. Alternatively, one can use the module save command to save the loaded module list.
Show loaded packages
[user@cluster ~]$ module list
Currently Loaded Modules:
1) Tecplot/2018r1 2) intel/19.0.3.199 3) openmpi/1.10.7 4) metis/5.1.0
Unload a package
For example, if a user wants to use the GNU compiler instead of the Intel compiler, he/she must first unload the Intel compiler (if it's already loaded) using the following command:
[user@cluster ~]$ module unload intel/19.0.3.199
and then load the GNU compiler using the usual module load command.
Job Management
This section gives information on how to submit jobs to the compute nodes and how to manage jobs. Note that, it is generally discouraged to run long-time jobs on the master (login) node.
Show available resources
To see how busy or idle the cluster is, use the sinfo command:
[user@cluster ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
defq* up infinite 12 alloc node[01-12]
defq* up infinite 8 idle node[13-20]
The output simply says that the defq (default queue) partition currently has 12 busy (allocated) nodes and 8 idle nodes. Note that, there is currently no enforced time limit for a job in this queue, but it is suggested that a user limit his/her run time to one week for the fairness of all users.
Submit a job
There are a couple of ways to run jobs on the compute nodes. But the most common way is to use a job script. Here is a sample script called jobsub.sh for submitting an MPI job. The contents of the script are also listed below with comments:
#!/bin/bash # shell environment
#SBATCH -J YourJobName # job name
#SBATCH -t 7-00:00:00 # requested maximum run time (day-hour:min:sec)
#SBATCH -p defq # requested partition (queue)
#SBATCH -n 60 # requested number of CPU cores
#SBATCH -e slurm%j.err # system error file
#SBATCH -o slurm%j.out # system output file
#SBATCH --mail-type=ALL # email notification type
#SBATCH --mail-user=xxx@xxx.edu # user's email address to recieve notifications
mpirun ./YourCode > output.out 2>&1 # mpi command to run the job
Do not forget to modify items in the script where necessary before submitting your job. Note that, the requested number of cores usually should be multiples of 12, because each compute node has 12 cores. In the rare case that a user has a series of jobs each reuire just 1 core, use the srun command with for loops in your script.
To submit a job, use the following sbatch command:
The output tells the user that the submitted job has a job ID 58 in this example.
One can also submit interactive jobs using the salloc command. More information regarding this command can be found in Slurm's manual.
Check the status of jobs
[user@cluster ~]$ squeue -u username
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
58 defq prop user PD 0:00 12 (Resources)
57 defq prop user R 0:48 12 node[01-12]
The output says that the user has one running job and one pending job in the queue, with job ID 57 and 58, respectively.
Cancel a job
[user@cluster ~]$ scancel 58
The above command will cancel/terminate the job with job ID 58.
Admin's Guide
Add a new user
To add a new user whose preferred user name is xxx:
[root@cluster ~]# adduser xxx
Then set up a password for the user:
[root@cluster ~]# passwd xxx
And, that's it, no more action is needed. Note that, the cluster is configured to automatically synchronize users' passwords to all compute nodes every five minutes using the cron service and Warewulf's file sync function. For this reason, a new user might have to wait for up to five minutes before he/she can run a job.
Delete a user
To delete the user xxx together with his/her home directory:
[root@cluster ~]# userdel -r xxx
Omit the -r if want to keep the user's home directory.
Install packages
All packages should be installed into the best category (e.g., apps, compiler, libs, utils, etc.) under the /opt/ohpc/pub directory (which is an NFS directory shared to all compute nodes).
Once a package is installed, the admin must set up a module file for this package under the /opt/ohpc/pub/modulefiles directory. In this way, a user can access this package through module load. Sample module files are found in any sub-directories of /opt/ohpc/pub/modulefiles, and also in /opt/ohpc/pub/examples.
Intel compiler license
The Intel compiler is free for students and educators. Its license is valid for one year but is renewable here. Once a new license file is received, go to /opt/intel/licenses to replace the expired license file with the new one.
Reboot the cluster
The master node can be rebooted when the compute nodes are either on or off, but not when any of them are in the process of booting up.
The compute nodes can be rebooted only when the master node is on.
More information
For more information, please consult BZ or Google.
CAHL Cluster is a parallel computing facility originally housed at the Computational Aero & Hydrodynamics Laboratory (CAHL) in the Department of Mechanical and Aerospace Engineering at the George Washington University. It was initially deployed in 2011, and since then primarily managed by team members of the lab. In 2017, the team rebuilt the cluster to update its operating and provisioning systems. Several hard drive failures led to another rebuild in 2019 using OpenHPC's collection of open-source packages in a "stateless" way (OS runs in RAMs instead of on hard drives). The cluster now runs CentOS 7, employs Warewulf as the provisioning system, Slurm as the job management tool, and Lmod as the package management tool. The cluster serves as the main testing and debugging platform of the lab for developing parallel flow and structure solvers. In the Fall of 2019, the cluster was relocated to Clarkson University following the move of the lab.