1 / 12

NGS computation services: APIs and Parallel Jobs

NGS computation services: APIs and Parallel Jobs. Richard Hopkins rph@nesc.ac.uk NGS Induction, National e-Science Centre, Edinburgh, Sept 29 th 2005. Overview. The C and Java API’s to the low-level tools Using multiple processors. Job submission so far. Command-line interfaces.

elizabeth-savage
Download Presentation

NGS computation services: APIs and Parallel Jobs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NGS computation services: APIs and Parallel Jobs Richard Hopkinsrph@nesc.ac.uk NGS Induction, National e-Science Centre, Edinburgh, Sept 29th 2005

  2. Overview • The C and Java API’s to the low-level tools • Using multiple processors NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  3. Job submission so far Command-line interfaces GLOBUS, etc. GLOBUS, etc. User’s Interface to the grid NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  4. Application-specific tools Application Specificand / or Higher generic tools e.g. BRIDGES Command-line interfaces GLOBUS, etc. User’s Interface to the grid NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  5. Application-specific tools • API’s: • Java • C • … Application Specificand / or Higher generic tools e.g. BRIDGES Command-line interfaces GLOBUS, etc. User’s Interface to the grid NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  6. Available API’s • C http://www.globus.org/developer/api-reference.html • “Community Grid” CoG http://www.cogkit.org/ • Java, Python, Matlab • (very limited functionality on Windows – no GSI) NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  7. Non-communicating Processes Globus_job_submit UI Internet Head processors of clusters Worker processors of clusters Processes run without any communication between them NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  8. Communicating Processes Globus_job_submit UI Internet Head processors of clusters Worker processors of clusters Processes send messages to each other – Must run on same cluster NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  9. Communicating Processes UI Internet Head processors of clusters MPI Worker processors of clusters Processes send messages to each other – Must run on same cluster NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  10. Modes of Parallelism • Communicating processes: on NGS, you run one globus-job-submit command – but need to code and build program so it is parallelised - MPI • Communicating processes: on NGS, multiple executables run from a script on the UI The NGS nodes open these routes to you – but you have to do a bit of work! (Grid is not magic!...) NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  11. MPI notes • How could the task be split into sub-tasks? • By functions that could run in parallel??! • By sending different subsets of data to different processes? More usual ! Overheads of scatter and gather • Need to design and code carefully: be alert to • sequential parts of your program (if half your runtime is sequential, speedup will never be more than 2) • how load can be balanced (64 processes with 65 tasks will achieve no speedup over 33 processes) • Deadlock! • MPI functions are usually invoked from C, Fortran programs, but also Java • Several example patterns are given in the practical.Many MPI tutorials are on the Web! NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

  12. Practical • C API Example • Java API usage • Concurrent processing • MPI example • Follow link from http://homepages.nesc.ac.uk/~gcw/NGS/ NGS Induction, Edinburgh, Sept 2005 – Computation Services – Richard Hopkins

More Related