1 / 18

Managing Scientific Computing Projects

Managing Scientific Computing Projects. Erik Deumens QTP and HPC Center. Overview. What is a scientific computing project? Procedures to manage scientific computing projects. Commodity computing. E-mail Web access Writing: papers, letters, thesis, presentations, web content

opa
Download Presentation

Managing Scientific Computing Projects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Managing Scientific Computing Projects Erik Deumens QTP and HPC Center Scientific Computing

  2. Overview • What is a scientific computing project? • Procedures to manage scientific computing projects Scientific Computing

  3. Commodity computing • E-mail • Web access • Writing: papers, letters, thesis, presentations, web content • Drawing: graphs, figures, plots • Calculating: spreadsheets, Mathematica, Maple, SAS, Matlab Scientific Computing

  4. Science and Engineering • Computing with software • Physics: VASP, WIEN • Chemistry: Gaussian, Q-Chem • Engineering: ANSYS • Developing software • Programming • Prototyping • Debugging • Performance analysis Scientific Computing

  5. Scientific Computing Project • Significant human effort • Many steps with dependencies • Takes a long time on one computer or many computers to complete • Involves a lot of data • Input given to be processed • Intermediate data for the computation • Output produced to be analyzed Scientific Computing

  6. Example SCP • Test a set of model parameters • Given basic parameters Bn • Compute dependent values Dj • Compare to test values Tj • If the number of dependent and test value sets is large, say 1,000 • And each computation takes time, say 1 h • Then this is a project Scientific Computing

  7. Recognizing SCP • Act from early stages as if it is SCP • Then procedures are tested and reliable by the time • the science of the project becomes harder • and requires all attention Scientific Computing

  8. Reliability of modern computers • Computers, networks and software are • Very stable • Very powerful • Leads to wide spread belief that they are • Infinitely stable • Infinitely powerful • Probability of failure • Small chance times lots of work = big chance Scientific Computing

  9. Overview • What is a scientific computing project? • Procedures to manage scientific computing projects Scientific Computing

  10. Manage a SCP • Project analysis • Data • Computation • Develop strategy • Organize the computation • Manage the data • Automation • Avoid human errors • Protect against disasters Scientific Computing

  11. Project analysis • Often a project starts small • Once you decide the project is worthwhile, perform a project analysis • Data: before, during, after • Computation: how many, how long • Precautions: minimize effect of disasters Scientific Computing

  12. Develop strategy • Organize the computation • Choose computer system • Study scheduling system • Match the project computation flow onto the scheduling policies • Manage the data • Input files generated by hand? By machine? • Space for large intermediate files • Space for output files Scientific Computing

  13. Automation • Extra tools needed to manage the project? • Generate input files from a database? • Write scripts? Use a tool already developed? • Generate scheduler command files? • Does a tool exist? Some tools are very complex. Is it easier to write scripts than to learn the tool? • Collect data from output files into a database? • Write scripts? Write a compiled program? Scientific Computing

  14. Automation • Computation and data monitoring • Check status of each run • Submit the job again if it failed • Check correctness and integrity of output data • Even if the job finished • it may have generated an error message • there may be no result • or the result may be invalid or incorrect Scientific Computing

  15. Precautions • Prepare for some disasters • Some or all computed data is lost or corrupted? • Make sure all files created manually are on disks that are backed up • at least, you can run computations again • Some output has been processed • Make sure partial results are on disks that are backed up Scientific Computing

  16. Growing projects • Often projects start small • Procedures are developed and used • They work well for 1,000 cases • Then the scope is increased • After partial success • Procedures are used unchanged • They do not work for 1,000,000 cases! • Must perform new analysis when scope changes Scientific Computing

  17. Tool choices • Small operations • Scripts are easy to write and change • Run fast for small numbers • Large operations • Running a script 10,000,000 times may be very slow and cause unexpected side effects • Investigate better tools • Program in compiled language • Use database instead of simple files Scientific Computing

  18. Conclusion • A little bit of thought, can save you from a lot of trouble and extra work • Every scientific computation project that is worth doing • is worth a little bit of thought about how to do it. Scientific Computing

More Related