Information Technology Services: Topsail
Group Contact
Ruth Marinshaw
Assistant Vice Chancellor

Campus Box 3420
Chapel Hill, NC 27599-3420
Phone: 919.962.4314
E-mail: ruth_marinshaw[at]unc.edu

Topsail

System Information

The Topsail cluster is intended for parallel computing although single processor runs are also supported.

  • Login node : topsail.unc.edu 8 CPUs @ 2.3 GHz Intel EM64T with 2x4M L2 cache (Model E5345/Clovertown), 12 GB memory
  • Compute nodes : 4,160 CPUs @ 2.3 GHz Intel EM64T with 2x4M L2 cache (Model E5345/Clovertown), 12 GB memory
  • Shared Disk : (/ifs1) 39 TB IBRIX Parallel File System
  • Interconnect: Infiniband 4x SDR
  • Resource management is handled by LSF v.6.2, through which all computational jobs are submitted for processing

Getting an Account

  • You may request an account on Topsail by sending an email to research@unc.edu. Please include your Onyen, full name, campus address, campus phone number, department, faculty sponsor's name if you are not a faculty member, and a description of the work you plan to do. You must also provide your email.unc.edu or first_last@unc.edu address; that is the address that will be used for communication with you. Topsail accounts are not available to collaborators from other institutions; you must be a UNC-Chapel Hill faculty member, staff member or a graduate student with a faculty sponsor in order to be considered for an account. Note that commercial applications software is not, in general, installed on the cluster.

Logging In

  • Use ssh to connect to topsail.unc.edu and login with your Onyen.
  • On the first login, ssh-keygen will run. Accept the defaults. If this environment is not established correctly, jobs will fail with "permission denied" messages.
  • Your home directory will be /ifs1/home/ONYEN. Any user home directory in excess of 4GB will be excluded from backups. Home is not intended to hold more than basic code and environment files. All data and output files should be directed to /ifs1/scr. Create a directory there for your work files. Note that /ifs1/scr will never be backed up and files not associated with a currently running process will be deleted after 21 days. Users have a hard quota of 500GB for data stored in /ifs1/scr.
  • Interactive use on the login node must be restricted to compiling and debugging. Other processes running on the login node will be terminated.

Software Development Tools

  • Intel Compiler Suite v.9.1 for Fortran77, Fortran90, C and C++, and v.9.0 Math Kernel Library

Compiling

The development suite is toggled using the module command. To see current environment: module list module show hpc/mvapich-intel To change to gcc: module unload hpc/mvapich-intel module load hpc/mvapich-gcc In either environment the available mpi compiler commands are mpicc mpif77 mpif90 mpiCC* mpicxx* *mpiCC and mpicxx rely on mpi2c++ which is currently only available under gcc

Submitting Parallel Jobs

 bsub -n "< number CPUs >" -o out.%J -e err.%J -a mvapich mpirun ./mycode

or

#### run_hpl #BSUB -n "< number CPUs >" #BSUB -e err.%J #BSUB -o out.%J #BSUB -a mvapich mpirun ./mycode ######## bsub < run_hpl

For more basic LSF commands refer to the brief Fundamental LSF Commands.

Queue Structure

  • A description of the queues available to users in the Topsail cluster can be found below. The list is in order of priority. You can also use the bqueues command to list the properties of a specific queue. For example, you could type "bqueues -l debug" to find out more about the debug queue. Additional queues may be added as need dictates.
  • All queues share a common fairshare allocation policy that governs what job will be dispatched next based on the recent runtime history of each user or group with jobs queued to run.
  • Time limits are measured by wallclock time.
  • All queues have exclusive enabled. A user can indicate with the bsub option -x that their job should be sent to hosts that have no other jobs running, and that no further job can be dispatched to those hosts until the job completes.
Queue Name> Min/Max CPU Limit per Job Time Limit Comments
debug 1 to 4 2 hours  
128cpu 32 to 128 2 days Default queue.
32cpu 4 to 32 2 days  
Limited Access Queues
512cpu 256 to 512 2 days Job must be shown to scale
reserve_rack     Research project specific.
chunk     Research project specific.
staff     Administrative purposes.