Community Software Development with the Astrophysics Simulation Collaboratory

1. Community Software Development with the Astrophysics Simulation Collaboratory Gabrielle Allen, Greg Daues, Jason Novotny, Edward Seidel Authors: Gregor von Laszewski, Michael Russell, Ian Foster, John Shalf, Presenter: Javier Munoz Agnostic: Ana Rodriguez

2. Outline • Introduce the ASC • Cactus: Architecture • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

3. Outline • Introduce the ASC • Cactus: Architecture • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

4. NSF Award Abstract -#9979985 KDI: An Astrophysics Simulation Collaboratory: Enabling Large Scale Simulations in Relativistic Astrophysics • NSF Org PHY Latest Amendment Date September 22, 2003 Award Number 9979985 Award Instrument Standard Grant Program Manager Beverly K. BergerPHY DIVISION OF PHYSICSMPS DIRECT FOR MATHEMATICAL & PHYSICAL SCIEN Start Date September 15, 1999 Expires August 31, 2004 (Estimated) Expected • Total Amount $2,200,000.00 (Estimated) • Investigator Wai-Mo Suen wms@wugrav.wustl.edu (Principal Investigator current)Ian Foster (Co-Principal Investigator current)Edward Seidel (Co-Principal Investigator current)Michael L. Norman (Co-Principal Investigator current)Manish Parashar (Co-Principal Investigator current) • Sponsor Washington UniversityONE BROOKINGS DRIVE, CAMPUS BOXSAINT LOUIS, MO 631304899 314/889-5100 NSF Program 8877 KDI-COMPETITION

5. Astrophysics Simulation Collaboratory (ASC) • Astrophysics • General Theory of Relativity • Simulation • Numerical solution of Partial Differential Equations • Collaboratory • Infrastructure to support efforts to solve large complex problems by geographicallydistributed participants.

6. Tired of Middleware? • The ASC is a complete Application • BUT we’ll talk middleware anyway…

7. Focus Application (70%) (30%) Middleware

8. ASC: Purpose • Community (VO) • Domain-specific components • Transparent access • Deployment services • Collaboration during execution • Steering of simulations • Multimedia streams

9. ASC: Technologies Used • Cactus Framework • Application Server • Grid Tools

10. Cactus Simulations in the ASCportal (Agnostic 4) www.ascportal.org

11. Outline • Introduce the ASC • Cactus Architecture (Agnostic 1) • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

12. Cactus • 1995 Original version: • Paul Walker • Joan Masso • Edward Seidel • John Shalf • 1999 Cactus 4.0 Beta 1 • Tom Goodale • Joan Masso • Gabrielle Allen • Gerd Lanfermann • John Shalf.

13. Why Cactus? • Parallelization Model • Easy to Grid-enable • Flexible • C and Fortran

14. Cactus • Modularity • New Equations (physics) • Efficient PDE solution (Numerical Analyst) • Improved distributed algorithm (CS)

15. Cactus • Building Blocks: • Schedule • Driver • Flesh • Thorns • Arrangements • Toolkit www.cactuscode.org

16. Scheduling (workflow) • Flesh invokes a driver to process the schedule ww.cactuscode.org

17. Driver • Parallelizes the execution • Management of grid variables • storage • distribution • communication • Distributed memory model.  • section of the global grid, • Boundaries: Physical or Internal • Each thorn is presented with a standard interface, independent of the driver.

18. Driver • PUGH (Parallel Unigrid Grid Hierarchy) • MPI • Uniform mesh spacing • Non-adaptive • Automatic grid decomposition • Manual decomposition • Number of processors in each direction • Number of grid points on each processor

19. Flesh • In general, thorns overload or register their capabilities with the Flesh, agreeing to provide a function with the correct interface

20. Thorn: Anatomy • Cactus Configuration Language • Parameter Files (Input) • Configuration File • Interface • Schedule • Application Code • C • Fortran • Miscellaneous • Documentation • Make ww.cactuscode.org

21. Thorn: Files • Param.ccl: • What are the parameters for my thorn? • What are their ranges? Are they steerable? • What parameters do I need from other thorns? • Which of my parameters should be available for other thorns? • Interface.ccl: • What does my thorn “do” • What are my thorns grid variables? • What variables do I need from other thorns? • What variables am I going to make available to other thorns? • Timelevels • Ghostzones • Schedule.ccl: • When and how should my thorns routines be run? • How do my routines fit in with routines from other thorns? • Which variables should be synchronized on exit?

22. Objectives • Introduce the ASC • Cactus: Architecture • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

23. Finite Differencing:Infinitesimal vs Small Delta http://homepage.univie.ac.at/franz.vesely/cp_tut/nol2h/applets/HO.html

24. Finite Differencing: Ghost Zones • The grid on each processor has an extra layer of grid-points (in blue) which are copies of those on the neighbor in that direction • After the calculation step the processors exchange these ghost zones to synchronize their states. Cactus 4.0 User’s Guide

25. Finite Differencing: Time Levels • Similar to Ghost Zones for the time dimension • Cactus managed leads to optimization • Numerical differentiation algorithm dependent • Typically three

26. Finite Differencing: Synchronization • Cost of parallelization • Network characteristics important • Transfer of 12MBytes per iteration • BUT…there is room for optimization

27. Objectives • Introduce the ASC • Cactus: Architecture • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

28. Cactus at Work • Members of the Cactus and Globus projects after winning a Gordon Bell Prizes in high-performance computing for the work described in their paper: Supporting Efficient Execution in Heterogeneous Distributed Computing Environments with Cactus and Globus

29. What did they do? • Scaled Out • Grid enabled four supercomputers • 249 GFlops  • Efficiently • Scaling efficiency: • 88% with 1140 CPU’s • 63% with 1500 CPU’s

30. Scaling Out Finite Differencing Solutions to PDE’s • Problem: • Nodes with different types of Processors, Memory sizes • Heterogeneous Communications among processors. • Multiprocessors • LAN • WAN • Bandwidth, TCP, and Latency

31. Scaling Out in a Computational Grid • Strategies • Irregular data distribution • Grid-aware communication schedules • Redundant computation • Protocol tuning

32. Where are they now? (Agnostic 5,8) • Gabrielle Allen Assistant Director for Computing Applications, Center for Computation & Technology Associate Professor, Department of Computer Science Louisiana State University • Edward Seidel Director, Center for Computation & Technology Floating Systems Professor, Departments of Physics and Computer Science Louisiana State University Visualization of Katrina developed at CCT Application Frameworks for High Performance and Grid Computing, G. Allen, E. Seidel, 2006. http://www.cct.lsu.edu/~gallen/Preprints/CS_Allen06b.pre.pdf

33. Outline • Introduce the ASC • Cactus: Architecture • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

34. GridSphere 1st Grid Middleware Congress www.GridSphere.org

35. GridSphere (Agnostic 3) • Developed by the EU GridLab project • About 100,000 lines of code • Version 2.0 • Framework based on: • Grid Portal Development Kit (GPDK) • ASC Web Portal • Open source project • http://www.gridsphere.org • Framework for portlet development • Portlet Container

36. Ongoing Collaborations (Agnostic 10) • Cactus portal at Albert Einstein Institute • Interface to Cactus numerical relativity application / provide physicists with interface for launching jobs & viewing results • Grid Portal at Canadian National Research Council • Provide controlled remote access to NMR spectroscopy instruments • GEON earth sciences portal / CHRONOS portal • Manage/visualize/analyze vast amount of geosciences data and large scale databases • Pgrade portal at SZTAKI Hungary & U. Westminster UK • Creation, execution and monitoring of complex workflows

37. GridSphere • Core Portlets • Login/Logout • Role Based Access Control (RBAC) separating users into guests, users, admins, and super users • Account Request • Account Management • User Management • Portlet Subscription • Local File Manager • Notepad • Text Messaging

38. Action Portlets • Hides branching logic • Action and view methods to invoke for events • Provides default actionPerformed

39. Personalized Environment "GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason Novotny GridSphere and Portlets workshop, March 2005, e-Science Institute

40. Single Sign-On Capabilities "GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason Novotny GridSphere and Portlets workshop, March 2005, e-Science Institute

41. Perform File Transfers (Agnostic 2) "GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason Novotny GridSphere and Portlets workshop, March 2005, e-Science Institute

42. GridSphere "GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason Novotny GridSphere and Portlets workshop, March 2005, e-Science Institute

43. Submit Jobs "GridSphere’s Grid Portlets A Grid Portal Development Framework" Jason Novotny GridSphere and Portlets workshop, March 2005, e-Science Institute

44. GridSphere https://portal.cct.lsu.edu/gridsphere/gridsphere?cid=home

45. Outline • Introduce the ASC • Cactus: Architecture • Cactus: Math • Cactus: Scaling Out • Gridsphere • Agnostic Questions

46. Agnostic Questions • The ASC application server uses a relational database to maintain the state of sessions; could it be implemented in any other way? Explain. • Sure, SQL is a de-facto standard; The Gridsphere Container provides a Persistence Manager that uses open-source Castor libraries from Exolab which provides mechanisms for mapping objects to SQL and an object query language (OQL) • Using Castor, mappings from Java to SQL can be generated automatically

47. Agnostic Questions • Can you expand on the MDS browser developed by the Java CoG Kit? • Currently MDS4 uses XPATH instead of LDAP for query language. • Registration is performed via a Web service • Container built-in MDS-Index service • Aggregator services are dynamic • A Service can become Grid wide service index. • Source: http://globus.org/toolkit/docs/4.0/key/GT4_Primer_0.6.pdf

48. Agnostic Questions • Could the collaboratory framework be implemented using technologies other than Java? If so, could it still be used in the same way? What would be the pros and cons of using Java technologies vs. other alternative technologies? • Cactus uses C and FORTRAN, but Web Portals are mainly being developed using Java: • GridPort • GCE-RG • GPDK • GridSphere

49. Questions? • Fallacy? • “Discipline scientists are typically not experts in distributed computing.” • Cactus Developers have a background in Mathematics and Physics.