1 / 45

Introduction to Scientific Computing on BU’s Linux Cluster

Introduction to Scientific Computing on BU’s Linux Cluster. Doug Sondak Linux Clusters and Tiled Display Walls Boston University July 30 – August 1, 2002. Outline. hardware parallelization compilers batch system profilers. Doug Sondak Linux Clusters and Tiled Display Walls

lynley
Download Presentation

Introduction to Scientific Computing on BU’s Linux Cluster

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to Scientific Computing on BU’s Linux Cluster Doug Sondak Linux Clusters and Tiled Display Walls Boston University July 30 – August 1, 2002

  2. Outline • hardware • parallelization • compilers • batch system • profilers Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  3. Hardware Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  4. BU’s Cluster • 52 2-processor nodes • specifications • 2 Pentium III processors per node • 1 GHz • 1 GB memory per node • 32 KB L1 cache per CPU • 256 KB L2 cache per CPU Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  5. BU’s Cluster (2) • Myrinet 2000 interconnects • sustained 1.96 Gb/s • Linux Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  6. Some Timings • CFD code, MPI, 4 procs. Origin2000 495 SP 329 Cluster, 2 procs. per box 174 Cluster, 1 proc. per box 153 Regatta 78 Machine Sec.

  7. Parallelization Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  8. Parallelization • MPI is the recommended method • PVM may also be used • some MPI tutorials • Boston University http://scv.bu.edu/Tutorials/MPI/ • NCSA http://pacont.ncsa.uiuc.edu:8900/public/MPI/ Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  9. Parallelization (2) • OpenMP is available for SMP within a node • mixed MPI/OpenMP not presently available • we’re working on it! Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  10. Compilers Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  11. Compilers • Portland Group • pgf77 • pgf90 • pgcc • pgCC Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  12. Compilers (2) • gnu • g77 • gcc • g++ Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  13. Compilers (3) • Intel • Fortran ifc • C/C++ icc Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  14. Compilers (2) Polyhedron F77 Benchmarks http://www.polyhedron.com/ PG gnu Intel AC 8.66 12.38 6.13 ADI 8.48 9.27 6.83 AIR 16.41 15.65 13.45 CHESS 11.67 10.06 10.16 DODUC 21.35 36.23 18.18 LP8 4.31 7.88 4.16 MDB 3.62 3.81 2.94 MOLENR 11.66 12.72 7.61 PI 24.58 41.95 7.08 PNPOLY 3.81 5.24 4.86 RO 10.75 10.31 3.92 TFFT 18.84 20.24 20.18

  15. Compilers (3) • Portland Group • pgf77 generally faster than g77 • Intel • ifc generally faster than pgf77 Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  16. Compilers (4) • Linux C/C++ compilers • gcc/g++ seems to be the standard, usually described as a good compiler Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  17. Portland Group -O2 • highest level of optimization -fast • same as -O2 -Munroll -Mnoframe -Minline • function inlining Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  18. Portland Group (2) -Mbyteswapio • swaps between big endian and little endian • useful for using files created on our SP, Regatta, or Origin2000 -Ktrap=fp • trap floating point invalid operation, divide by zero, or overflow • slows code down, only use for debugging Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  19. Portland Group (3) -Mbounds • array bounds checking • slows code down, only use for debugging -mp • process OpenMP directives -Mconcur • automatic SMP parallelization Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  20. Intel • Need to set some environment variables • contained in /usr/local/IT/intel6.0/compiler60/ia32/bin/iccvars.csh • source this file, copy it into your .cshrc file, or source it in .cshrc • there’s an identical file called ifcvars.cshto avoid (create?) confusion Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  21. Intel (2) -O3 • highest level of optimization -ipo • interprocedural optimization -unroll • loop unrolling Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  22. Intel (3) -openmp -fpp • process OpenMP directives -parallel • automatic SMP parallelization -CB • array bounds checking Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  23. Intel (3) -CU • check for use of uninitialized variables • Endian conversion by way of environment variables setenv F_UFMTENDIAN big • all reads will be converted from big to little endian, all writes from little to big endian Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  24. Intel (4) • Can specify units for endian conversion setenv F_UFMTENDIAN big:10,20 • Can mix endian conversions setenv F_UFMTENDIAN little;big:10,20 • all units are little endian except for 10 and 20, which wil be converted Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  25. Batch System Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  26. Batch System • PBS • different than LSF on O2k’s, SP’s, Regattas • there’s only one queue dque Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  27. qsub • job submission done through script • script details will follow qsub scriptname • returns job ID • in working directory • std. out - scriptname.ojobid • std. err -scriptname.ejobid [sondak@hn003 run]$ qsub corrun 808.hn003.nerf.bu.edu

  28. qstat • Check status of all your jobs qstat • lies about run time • often (always?) zero [sondak@hn003 run]$ qstat Job id Name User Time Use S Queue ---------------- ---------------- ---------------- ------------ - -------- 808.hn003 corrun sondak 0 R dque

  29. qstat (2) • S - job status • Q - queued • R - running • E - exiting (finishing up) • qstat -f gives detailed status exec_host = nodem019/0+nodem018/0 +nodem017/0+nodem016/0 • to specify jobid qstat jobid

  30. Other PBS Commands • kill job qdel jobid • some less-important PBS commands • qalter, qhold, qrls, qmsg, qrerun • man pages are available for all commands Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  31. PBS Script • For serial runs #!/bin/bash # Set the default queue #PBS -q dque # ppn is cpu's per node #PBS -l nodes=1:ppn=1,walltime=00:30:00 cd $PBS_O_WORKDIR myrun

  32. PBS/MPI • For MPI, set up gmi file in PBS script test -d ~/.gmpi || mkdir ~/.gmpi GMCONF=~/.gmpi/conf.$PBS_JOBID /usr/local/xcat/bin/pbsnodefile2gmconf $PBS_NODEFILE > $GMCONF cd $PBS_O_WORKDIR NP=$(head -1 $GMCONF) Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  33. PBS/MPI (2) • To run MPI, end PBS script with (all on one line) mpirun.ch_gm --gm-f $GMCONF --gm-recv polling --gm-use-shmem --gm-kill 5 -np $NP PBS_JOBID=$PBS_JOBID myprog Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  34. PBS/MPI (3) • mpirun.ch_gm • version of mpirun that uses myrinet • --gm-f $GMCONF • access configuration file constructed above • --gm-recv polling • poll continually to check for completion of sends and receives • most efficient for dedicated procs. • That’s us!

  35. PBS/MPI (4) • --gm-use-shmem • enable shared-memory support • may improve or degrade performance • try your code with and without it • --gm-kill 5 • if one MPI process aborts, kill others after 5 sec. Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  36. PBS/MPI (5) • -np $NP • run on NP procs as computed earlier in script • equals “nodes x ppn” from PBS -l option • PBS_JOBID=$PBS_JOBID • seems redundant redundant • do it anyway • myprog • run the darn code already!

  37. Profiling Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  38. Portland Group • Portland Group Compiler flag • function level -Mprof=func • line level -Mprof=lines • much larger file • creates pgprof.out file in working directory Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  39. PG (2) • At unix prompt, type pgprof command • will pop up window with bar chart of timing results • can take file name argument in case you’ve renamed the pgprof.out file pgprof pgprof.lines Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  40. PG (3) • option to specify source directory pgprof -Isourcedir pgprof.lines • can specify multiple directories with multiple -I flags • also can use GUI menu • Options Source Directory... Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  41. PG (4)

  42. PG (5) • Calls - number of times routine was called • Time - time spent in specified routine • Cost - time spent in specified routine plus time spent in called routines Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

  43. PG (6) • Lines profiling • with optimization, may not be able to identify many (most?) lines in source code • reports results for blocks of code, e.g., loops • without optimization, doesn’t measure what you really want • initial screen looks like “func” screen • double-click function/subroutine name to get line-level listing

  44. PG (7)

  45. Questions/Comments • Feel free to contact us directly with questions about the cluster or parallelization/optimization issues Doug Sondak sondak@bu.edu Kadin Tseng kadin@bu.edu Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

More Related