1 / 30

Clusters in Molecular Sciences Applications

Clusters in Molecular Sciences Applications. Serguei Patchkovskii @# , Rochus Schmid @ , Tom Ziegler @ , Siu Pang Chan # , Andrew McCormack # , Roger Rousseau # , Ian Skanes #. @ Department of Chemistry, University of Calgary, 2500 University Dr. NW, Calgary, Alberta, T2N 1N4 Canada

denis
Download Presentation

Clusters in Molecular Sciences Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Clusters in Molecular Sciences Applications Serguei Patchkovskii@#, Rochus Schmid@, Tom Ziegler@, Siu Pang Chan#, Andrew McCormack#, Roger Rousseau#, Ian Skanes# @Department of Chemistry, University of Calgary, 2500 University Dr. NW, Calgary, Alberta, T2N 1N4 Canada #Theory and Computation Group, SIMS, NRC, 100 Sussex Dr., Ottawa, Ontario, K1A 0R6 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 1

  2. Overview • Beowulf-style clusters entered mainstream • Are clusters a lasting, efficient investment? • Odysseus: an internal cluster at the SIMS theory group • Clusters in molecular science applications: software availability and performance • Three war stories, and a cautionary message • Summary and conclusions “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 2

  3. Shared, Academic Clusters in Canada “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 3

  4. Canadian top-500 facilities Cluster “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 4

  5. Internal, “workhorse” clusters “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 5

  6. Clusters became a mainstream research tool – useful, but not automatically worthy of a separate mention Clusters are everywhere Lemma 1: A computationally-intensive research group in Canada can be in one of the three states: • It owns a cluster, or • It builds a cluster, or • It plans building a cluster RSN “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 6

  7. World 100BaseTx (half-duplex) Node 93 Node 1 93x100BaseTx Switch Cobalt Master 2x100BaseTx 128Mb memory 18Gbytes RAID-1 (4 spindles) Cobalt: Hardware Computers on benches all linked together “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 7

  8. Cobalt: Nodes and Network Digital/Compaq Personal Workstation 500au. 4 x 3COM SuperStack II 3300 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 8

  9. Cobalt: Software OS, communications, and cluster management: Base OS: Tru64, using DMS, NIS, and NFS Compilers: Digital/Compaq C, C++, Fortran Communications: PVM, MPICH Batch queuing: DQS Application software: ADF: Amsterdam Density Functional (PVM) PAW: Projector-Augmented Wave (MPI) “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 9

  10. Investment: Dollars Payback: Research Articles Cobalt: Return on the Investment ROI: 1 publication / $4,250 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 10

  11. Odysseus: Low-tech solution for high-tech problems1 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 11

  12. Nodes (16+1) ABIT VP6 motherboard 2xPIII-933, 133MHz FSB 4x256Mbytes RAM 3COM 3C905C 36Gb 7200rpm IDE … plus, on the front end: Intel PRO/1000 Adaptec AHA-2940UW 60Gb 7200rpm IDE Odysseus: Low-tech solution for high-tech problems2 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 12

  13. Network: SCI + 100Mbit Dolphin D339 (2D SCI) H ring V ring HP Procurve 2524 + 1Gig Odysseus: Low-tech solution for high-tech problems3 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 13

  14. Backup unit: VXAtape (www.ecrix.com) 35Gbytes/cartridge (physical) TreeFrog autoloader (www.spectralogic.com) 16 cartridge capacity UPS Unit: Powerware 5119 2880VA Odysseus: Low-tech solution for high-tech problems4 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 14

  15. Four little wheels Odysseus: Low-tech solution for high-tech problems5 “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 15

  16. Odysseus: cost overview “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 16

  17. Gaussian Turbomole GAMESS NWChem GROMOS ADF PAW CPMD AMBER VASP PWSCF ABINIT Clusters in molecular science – software availability “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 17

  18. Speedup ideal Observed Number of Cobalt nodes Software: ADF ADF – Amsterdam Density Functional (www.scm.com) Example: Cr(N)Porph Full geometry optimization 38 atoms 580 basis functions C4v symmetry 45Mbytes of memory Serial time: 683 minutes “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 18

  19. Speedup ideal Observed Cobalt Nodes Software: PAW PAW – “Projector-Augmented Wave” (www.pt.tu-clausthal.de/~ptpb/PAW/pawmain.html) Example: SN2 reaction CH3I + [Rh(CO)2I2]- 11Å unit cell Serial time per step: 83 seconds Memory: 231Mbytes “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 19

  20. Software: CPMD CPMD – Car-Parinello Molecular Dynamic (www.mpi-stuttgart.mpg.de/parinello/) Example: H in Si64 65 atoms, periodic 40Ryd cut-off Geometry opt (2 steps) + free MD (70 steps) odysseus “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 20

  21. Software: AMBER AMBER – “Assisted Model Building with Energy Refinement” (www.amber.ucsf.edu/amber/) Example: 22-residue polypeptide+4K+ +2500 H2O 1ns MD Time (hour) Ncpu “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 21

  22. Software: VASP VASP – Vienna Ab-initio Simulation Package (cms.mpi.univie.ac.at/vasp/) Example: Li198 1000GPa 300 eV cutoff 9 K-points 10 WF optimization steps + stress tensor odysseus “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 22

  23. Software: PWSCF PWSCF and PHONON – Plane wave pseudopotential codes, optimized for phonon spectra calculations (www.pwscf.org/) Example: MgB2 solid Geometry opt. 40 Ryd cut-off 60 K-points odysseus “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 23

  24. Software: ABINIT ABINIT (www.mapr.ucl.ac.be/ABINIT/) Example: SiO2 (stishovite) 70Ryd cut-off 6 K-points 12 SCF iterations “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 24

  25. War Story #1 Odysseus hardware maintenance log, Oct 19, 2001: • Overnight, node 6 had a kernel OOPS … it responds to network pings and keyboard, but no new processes can be started … Reason: • Heat sink on CPU#1 became loose, resulting • in overheating under heavy load. Resolution: • Reinstall the heat sink Detected by: • Elevated temperature readings for the • CPU#1 (lm_sensors) Downtime: • 20 minutes (the affected node) “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 25

  26. War Story #2 Odysseus hardware maintenance log, Nov 12, 2001: • A large, 16-CPU VASP job fails with “LAPACK: Routine ZPOTRF failed”, or random total energy Reason: • DIMM in bank #0 on node 17 developed a single- • bit failure at the address 0xfd9f0c Resolution: • Replace memory module in bank #0 Detected by: • Rerunning failing job with different sets of nodes, • followed by the memory diagnostic on the affected • node (memtest32) Downtime: • 1 day (the whole cluster) + 2 days (the affected node) “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 26

  27. War Story #3 Odysseus hardware maintenance log, Dec 10, 2001: • Apparently random application failures are observed Reason: • Multiple single-bit memory • failures, on the nodes (bank #): • 6 (#2), 7 (#2,#3), 8 (#0), • 10 (#0), 11 (#0) Resolution: • Replace memory modules Detected by: • Cluster-wide memory diagnostic (memtest32) Downtime: • 3 days (the whole cluster) “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 27

  28. Cautionary Note • Using inexpensive, consumer-grade hardware potentially exposes you to low-quality components • Never use components which have no built-in hardware monitoring and error detection capability • Always configure your clusters to report corrected errors and out-of-range hardware sensors readings. • Act on the early warnings • Otherwise, you run a risk of producing garbage science, and never knowing it “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 28

  29. Hardware Monitoring with Linux #http://www2.lm-sensors.nu/~lm78/ $http://www.linux-ide.org/smart.html %http://csl.cse.ucsc.edu/smart.shtml ^http://www.anime.net/~goemon/linux-ecc/ (2.2 kernels only) “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 29

  30. Summary and Conclusions • Clusters are no longer a techno-geek’s toy, and will remain the primary workhorse of many research groups, at least for a while • Clusters give an impressive return on the investment, and may remain useful longer than expected • Many (most?) useful research codes in molecular sciences are readily available on clusters • Configuring and operating PC clusters can be tricky. Consider a reputable system integrator with Beowulf hardware and software experience “Clusters in Molecular Sciences Applications”, 2nd Annual iHPC Cluster Workshop, Ottawa Jan 11, 2002. p. 30

More Related