Intermediate Guide to FAS HPC Resources

1. Intermediate Guide to FAS HPC Resources Teresa Kaltz, PhD Research Computing rchelp@fas.harvard.edu

2. FAS Research Computing • FAS RC Sciences Division • Massive compute and storage resources collectively referred to as "Odyssey" • Staff with both computer and domain expertise • Anyone with FAS faculty sponsor can get an account • Web site: http://rc.fas.harvard.edu • Questions: rchelp@fas.harvard.edu

3. What is “Odyssey”? (April 2008)

4. "Odyssey" Today (April 2010)

5. Presentation Outline • Part 1: Technical Background • Compute • Storage • Networking • Part 2: Using Odyssey • Linux & the command line • Running programs • LSF scheduling software

6. Odyssey Compute Resources • The computers that make up Odyssey have many names: "server", "host", "node" • They all refer to the same thing! • Each server has: • CPU & memory (RAM) • local disk (hard drive) • network • filesystems (access to home directory, lab space, etc) • Odyssey currently has over a thousand servers!

7. What is a CPU? • Single piece of silicon that plugs into a "socket" • Server motherboards can have multiple sockets • Multiple processing "cores" per CPU • Each core is capable of executing one "instruction stream" • For example: • A 4 core 2 socket server has 8 processing cores total; hence it can run 8 serial jobs • Processor used to refer to both core and CPU

8. Intel Xeon X5560 (Nehalem)

9. How Can I Learn About CPUs? • cat /proc/cpuinfo will give speed, type and number of cores for a particular host • Listed frequency might be lower ("power save mode") • Double actual core count (hyperthreading) • lshosts lists subset of CPU info for all servers on Odyssey • Odyssey has both Intel and AMD (Opteron) CPU's • 1 - 4 cores, 2-3Ghz • Some lower speed CPU's actually perform much better than higher speed ones

10. Partial Screen Shot of /proc/cpuinfo

11. Do I Care Which CPU I Use? • Contrary to marketing hype, you don't care much about the speed • You *might* care about processor type • "Fatal Error: This program was not built to run on the processor in your system." • Built with architecture specific instructions • Contact rchelp@fas if this is an installed application • Check whether you are using compiler flags specifying architecture (-x for Intel, -march for gnu)

12. What is memory (RAM)? • Any data your program needs to use must be read off disk and loaded into memory • Your job will share memory with others jobs unless you are using a host exclusively • If your program tries to use more memory than is available, it will usually slow down dramatically (paging and/or swapping) • If it uses a lot more memory than is available, it might crash the server and kill your job!

13. How Much Memory Is Available? • cat /proc/meminfo will give you total amount of memory on the node (MemTotal) • Plus lots of other information beyond the scope of this talk (virtual memory) • lshosts lists total memory (maxmem) for all servers on Odyssey • Odyssey nodes have between 8GB and 64 GB of memory! • We'll cover this in more detail later

14. What is Storage? • Odyssey storage consists trays of disk drives bundled together to create large pools of storage space • These pools are presented to users as "filesystems" • Any individual has access to a one or more directories (folders) on a subset of filesystems • Performance and size varies greatly between filesystems

15. Types of Filesystems on Odyssey • Local • Presented as /scratch • Only that one node has access to the files • Network Filesystem (NFS) • CIFS is used for Windows clients • Files accessible on all nodes • Lustre • High performance filesystem • Not available for general users

16. Some Odyssey Filesystems

17. Home Directories • Home directories are located on a highly available enterprise class storage system • Uses NFS • No single point of failure • Shared with ALL Odyssey users • Filesystem is checkpointed and backed up • Checkpoints in .ckpt in home directory • Lower performance • Store critical data but don't run I/O intensive jobs

18. Local Directories • These filesystems are NOT available on other hosts • You should use /scratch not /tmp • If you fill /tmp the node may crash • You must stage files before and after your job runs • Use a script to copy your data • Good performance since it is not shared • Very useful for serial jobs

19. Scratch Directories • Scratch directories may be located on different types of storage • All NFS • Some on enterprise class (/n/scratchXX/) • Others exported from single servers (/n/nobackupX) • All are shared with other users • Performance varies depending on filesystem • If you want to run jobs with lots of I/O, contact rchelp@fas

20. What is Networking? • Odyssey has two types of networks • Ethernet (Gigabit Ethernet or GbE) • Infiniband (IB) • All servers and storage have Ethernet connections between 1 – 4 Gb/s • A subset of servers and storage also have Infiniband which is a high-speed, low latency technology • Use this for distributed memory parallel jobs

21. Odyssey software: Linux • Almost all Odyssey computers run the Linux operating system • Most powerful way to interact with Linux is via the "command line" • User types commands that are run in a "shell" • Steep learning curve • Rich programming language

22. Shell Power: Real World Example • "I need a program to help me determine the number of unique sequences in a large data set. Can you write this for me?" • Snippet from file at right • Many thousands of these

23. Linux has a Command for this! • No program or script needed

24. What is a "shell"? • Core Linux operating system (OS) is called the "kernel" • Schedules processes • Handles peripheral devices like keyboard input • Stuff the user doesn't need to know about • User cannot directly interact with the kernel • User types commands into an interpreter called a "shell" • The shell translates these commands into kernel-speak and passes them to the kernel

25. Two Types of Shells on Odyssey • bash: Bourne Again Shell • Similar to sh, ksh • Default on Odyssey • tcsh: Enhanced C Shell (Turbo) • Similar to csh • Each has a different syntax and startup files • You should pick one and learn it • Use echo $SHELL to see which shell you are using

26. Searching for files • Find files greater than 1 GB in size: find . -size +1G • Find core files and delete them: find . -name core -exec rm {} \; find . -name core | xargs rm –f • Find all strings with "MPI_": find . -print | xargs grep -n MPI_

27. Searching for text and patterns • Use the grep command and variants • Here are real examples of grep from my command history: rpm -qa | grep lustre find . -print | xargs grep -in MPI_ ps axms | grep -A 1 fct egrep '(processor|physical id)' /proc/cpuinfo

28. Other Linux Commands and Utilities • diff – show differences between files • sort – sort lines of text • uniq – report or omit repeated lines • screen – persistent terminal session http://rc.fas.harvard.edu/tipsandtricks/screen • rsync – powerful file copying command http://rc.fas.harvard.edu/tipsandtricks/rsync • tar – create archives of files

29. What are awk & sed? • These are text processing utilities • awk is good at extracting data in columns; sed is useful for making simple edits to groups of files • Print second column in a file if line contains foo: awk '/foo/{print $2}' file • Replace all occurrences of aaa with bbb sed 's/aaa/bbb/g' <old >new

30. Learn the Command Line! • Command line is more than just tricks to improve workflow efficiency • You will use shell more frequently than any "programming language" unless you are actively developing software • Great "toolbox" reference: UNIX Power Tools, O'Reilly • http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html

31. Let's Take a Break!!

32. Running Programs on Odyssey • You submit programs as "jobs" to the LSF scheduler • These jobs can take different forms • serial or parallel • interactive or batch • The jobs run on hosts different from the one you are logged into • After you reserve the node you can ssh

33. What is a Process? • The kernel sees your program and all commands you execute as "processes" • The kernel itself runs many processes to do important functions of the OS • Allow user programs to access network, filesystems • Processes have memory and various resources associated with them • Your Odyssey program may have one or more processes associated with it

34. What is a Thread? • A thread can be conceptualized as the subset of a process that contains the instruction stream • A process can have one or more threads • Code must be explicitly programmed to use multiple instruction streams or threads • Details of how "multithreading" works varies greatly • There is also an API called Pthreads • POSIX standard for programming threads • Many implementations of the standard

35. Running in Parallel • Some applications can do work using more than one process or thread: these are called "parallel applications" • Distributed memory parallel programs • Can run on multiple hosts • Shared memory parallel programs • Can only run on ONE host • Most applications cannot run in parallel! • They must be explicitly programmed

36. Parallel Programming Methods • MPI is popular for distributed memory parallel applications in the scientific community • OpenMPI, MVapich • OpenMP is for multithreading • Simply add directives to existing code • Speed up is not usually great • Pthreads • Autoparallelism via compiler flags

37. Parallel Applications on Single Host • Programs written using OpenMP or Pthreads • These use multiple threads • Can also run MPI programs on a single host • Can run these "shared memory" parallel applications in the serial queues! • Reserve up to 8 cores per job • Usually can control number of threads via environment variables • OMP_NUM_THREADS for OpenMP

38. Parallel Applications Across Hosts • Programs written using MPI or sockets • Use multiple processes • Usually require some separate method to "launch" the application • Some canned software applications hide this from the user • See http://rc.fas.harvard.edu/sw for details on how to launch particular Odyssey applications

39. Process Memory • Processes have two type of "memory" associated with them • Virtual memory • Creates the appearance of every process having all the memory in a computer • Helps the kernel manage physical memory efficiently • Extremely complicated • Real (or "resident") memory • The physical memory (RAM) on a host

40. What Are My Processes? Using ps • PID == process ID • VSZ == virtual memory size; the amount of memory the process has reserved to use • RSS == resident set size; the amount of physical memory the process is actively using • STAT == status • R running (doing something right now) • S sleeping (not doing anything and doesn't want to) • D uninterruptible (waiting on an event that may or may not happen – could be doing I/O or "hung")

41. Linux ps aux Command

42. Process Memory Usage Guidelines • The RSS of your process should not exceed total memory/number of cores • You can use all memory if you reserved exclusively • If the sum of RSS of all processes exceeds total memory available, node starts paging heavily • Paging can slow down your job by 10x • If RSS continues to grow, then node will start swapping • Node will eventually become unresponsive

43. Measuring Memory Use of a Program • You can log into the node your job is running on and issue ps ux • Lists all processes running under your username • Check if RSS size of your processes • If you use the top command: Determining total memory use on a host can be confusing: there is rarely a lot of "free" memory at any time, even when no user processes are running

44. Am I Running Out of Memory? • Job mysteriously dies when increasing problem size • LSF Summary Output: Is swap > 8GB times number of nodes?

45. Example: Large Memory + 8 Processes

46. How Much Memory Do I Need? • This can be very hard to predict • Estimate from problem size • A 1,000,000 x 1,000,000 matrix (dense) double precision elements would need at least 8000 GB • Check file size of data sets • Run a few cases using smaller problem sizes and extrapolate based on measured usage • Warning: the size command is not accurate

47. The ldd command • This command lists all the shared libraries used by a program • A shared library is just a piece of code this is so commonly used the OS "shares" it amongst the processes that need it • This technique reduces memory usage by programs • If the program cannot load the shared library into memory, then it will not run • "error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory"

48. Output from ldd

49. LD_LIBRARY_PATH • This environment variable lists search path for shared libraries • Use echo $LD_LIBRARY_PATH to see current search path • If your program can't find a shared library, then put it in LD_LIBRARY_PATH • The module load/unload command modifies this variable • Use the module show command to see what it is being set

50. LSF: The Odyssey Batch Scheduler • LSF (Load Sharing Facility) is the name of the software that schedules user jobs on the compute cluster • Commands are found in /lsf/7.0/linux2.6-glibc2.3-x86_64/bin • LSF is responsible for ensuring efficient scheduling of thousands of jobs from hundreds of users