The configuration of the Opteron cluster is shown in Fig. 1. It consists of 114 compute nodes connected by Infiniband and gigabit networking. All machines run Red Hat Enterprise Linux 4 and Platform OCS (a vendor supported "Rocks"-like distribution) All compute nodes are are accessed from the front end. A RAID array is directly attached to the front end and contains the home directories and 2 globally accessible scratch spaces of 1 TB each. Also, there is a set of 3 IO nodes which serve as a 1.5 TB parallel file system and is accessible from the nodes and the front end via the infiniband network.
The following table lists the details of each machine in the cluster:
| # | Node name | architecture | processor | memory | local disk |
|---|---|---|---|---|---|
| 1 | Dell PowerEdge 2970 | pzt.wm.edu | Opteron 2220 / 2.8 GHz | 8 GB | 380 GB |
| 98 | Dell PowerEdge SC1435 | c1-[1-28],c2-[29-57],c4-[58-86],c5-[87-102] | Opteron 2218 / 2.6 GHz | 8 GB | 120 GB |
| 4 | Dell PowerEdge SC1435 | c5-[103-106] | Opteron 2222 SE / 3.0 GHz | 32 GB | 420 GB |
| 8 | Dell PowerEdge SC1435 | c5-[107-114] | Opteron 2222 SE / 3.0 GHz | 16 GB | 420 GB |
Table 2 shows the various filesystems that users may access within the cluster. The /home, /scr1 and /scr2 filespaces are all mounted via NFS over the gigabit ethernet network. Currently a 10GB quota is set for all users /home directory space. /scr1 and /scr2 are to be used as global scratch space. The global scratch space is to be used for temporarily storing large files. Neither filesystem is backed up and purging is not done automatically. Users with large amounts of data (>100 GB) will be made to reduce their usage when the scratch space starts to get too full (> ~70%).
| name | size | notes |
|---|---|---|
| /home/$USER | 1 TB | users get a 10 GB quota |
| /scr1/$USER | 1 TB | Global NFS scratch 1 |
| /scr2/$USER | 1 TB | Global NFS scratch 2 |
| /pvfs | 1.5 TB | Global parallel scratch |
| /lscr | 120 GB / 420 GB | Local scratch on compute node |
The following lists the compilers, libraries, mpi environments and computational packages that are currently installed on the cluster. Most packages have an environment module in place for users to load. Please send email to Eric J. Walter if you need additional software installed.
| Command | |||||
|---|---|---|---|---|---|
| Description | Version(s) | Module | C | C++ | Fortran |
| Portland Group / PGI | 7.1, 7.1-3, 7.2-3 | pgi/7.1, pgi/7.1-3, pgi/7.2, pgi/7.2-3 | pgcc | pgCC | pgf90/pgf95/pgf77 |
| Pathscale | 3.0 | pathscale | pathcc | pathCC | pathf90/pathf95/pathf77 | Intel | 9.1.036, 10.1.012, 10.1.015 | intel/9.1, intel/10.1, intel/10.1v2 | icc | icpc | ifort |
| Absoft | 10.1 | absoft | NA | NA | f90/f95/f77 |
| Name | Location |
|---|---|
| AMD Core Math Library | /share/apps/acml4.0.1 |
| Hierarchical Data Format | /share/apps/absoft10.1/extras/hdf(4,5) |
| fftw v2.1.5 | /share/apps/fftw2 |
| fftw v3 | /share/apps/fftw3 |
| cmkl - Intel's Cluster Math Kernel Library | /share/apps/intel/cmkl/10.0.1.014,/share/apps/intel/cmkl/10.0.3.020 |
| ATLAS | /share/apps/absoft10.1/extras/ATLAS |
| Name | Version | Module |
|---|---|---|
| openmpi | 1.2.4,1.2.5,1.2.8(compiled using intel v10.1.015) | openmpi-1.2.4, openmpi-1.2.5, openmpi-1.2.8_intel |
| mvapich2 | 1.0.1, 1.0.2p2 | mvapich2-1.0.1, mvapich2-1.0.2 |
| mvapich1 | 1.0p1 (compiled for pgi and intel) | mvapich-1.0-intel, mvapich-1.0-pgi |
| Name | Version | Module | Notes |
|---|---|---|---|
| ABINIT | 5.4.4, 5.4.4p, 5.5.4, 5.6.4 | abinit/abinit-544_mvapich2, espresso/espresso-402_openmpi125, abinit/abinit-544_openmpi125, abinit/abinit-544p_openmpi125, abinit/abinit-554_openmpi125, abinit/abinit-564_openmpi125 | The kpt/spin parallel version is named abinip, the band/fft parallel version is named abinip_bfft for versions earlier than 5.6.4 |
| Quantum Espresso | CVS from 02/27/08, v3.2.3, v4.0.2 | espresso/espresso-CVS_openmpi125, espresso/espresso-323_mvapich2, espresso/espresso-402_openmpi125 | Parallel versions of the codes d3.x, phcg.x, ph.x and pw.x are named p_d3.x, p_phcg.x, p_ph.x, p_pwcond.x, p_pw.x |
| Dacapo | SVN from 03/31/08 | dacapo/dacapo_SVN033108 | |
| Siesta | v2.0.1 | siesta/siesta-2.0.1 |