GridLab Project

1. GridLab Project • Funded by the EU (5+ M€), January 2002 – December 2004 • Application and Testbed oriented • Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled • Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools...: • resource management (GRMS), • data management, • monitoring, • adaptive components, • mobile user support, • security services, • portals, ... and test them on a real testbed with real applications SC 2003 Demo, NCSA booth

2. GridLab Members • PSNC (Poznan) - coordination • AEI (Potsdam) • ZIB (Berlin) • Univ. of Lecce • Cardiff University • Vrije Univ. (Amsterdam) • SZTAKI (Budapest) • Masaryk Univ. (Brno) • NTUA (Athens) • Sun Microsystems • Compaq (HP) • ANL (Chicago, I. Foster) • ISI (LA, C.Kesselman) • UoWisconsin (M. Livny) • collaborating with: • Users! • EU Astrophysics Network, • DFN TiKSL/GriKSL • NSF ASC Project • other Grid projects • Globus, Condor, • GrADS, • PROGRESS, • GriPhyn/iVDGL, • Most of the other European Grid Projects (GRIDSTART) • GWEN SC 2003 Demo, NCSA booth

3. GridLab Aims • Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community), all the other potential grid apps • Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations • Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources. SC 2003 Demo, NCSA booth

4. What GridLab isn’t • We are not developing low level Grid infrastructure, • Addressing Grids and P2P • We do not want to repeat work which has already been done (want to incorporate and assimilate it …) • Globus APIs, • OGSA, • ASC Portal (GridSphere/Orbiter), • GPDK, • GridPort, • DataGrid, • GriPhyn, • ... SC 2003 Demo, NCSA booth

5. GridLab End User Requirements • Application oriented environment, • Applications on one or more virtual organizations, • Today we are able to run jobs between GridLab and Progress VOs • Flexible, easy-to-use, simple interfaces • resources, jobs, and data (including compiling, tracking jobs, cataloguing data), • Means to make efficient and effective use of resources, • Robustness • smart adaptivity, complete control and fail safety areavailable on all levels, • The ability to work in a disconnected environment, • Mobile working, SC 2003 Demo, NCSA booth

6. Larger computational resources Memory/CPU Faster throughput Cleverer scheduling, configurable scheduling, co-scheduling, exploitation of un-used cycles Easier use of resources Portals, grid application frameworks, information services, mobile devices Remote interaction with simulations and data Notification, steering, visualization, data management Collaborative tools Notification, visualization, video conferencing, portals Dynamic applications, New scenarios Grid application frameworks connecting to services What do our users want? SC 2003 Demo, NCSA booth

7. GridLab end user requirements • From laptops to fully deployed Virtual Organisations, • Complexity hidden as much as possible, • Collaborative infrastructure, • The infrastructure for all classes of applications • The infrastructure must provide capabilities to customise choice of service implementation (e.g.using efficiency, reliability, first succeeding, all) SC 2003 Demo, NCSA booth

8. Dynamic Staging move to faster/cheaper/bigger machine Multiple Universe create clone to investigate steered parameter Automatic Convergence Testing from initial data or initiated during simulation Look Ahead spawn off and run coarser resolution to predict likely future Spawn Independent/Asynchronous Tasks send to cheaper machine, main simulation carries on Application Profiling best machine/queue choose resolution parameters based on queue Dynamic Load Balancing inhomogeneous loads multiple grids Portal User/VO interface to the grid Intelligent Parameter Surveys farm out to different machines Make use of Running with management tools such as Condor, Entropia, etc. Scripting thorns (management, launching new jobs, etc) Dynamic use of eg. MDS for finding available resources Application Scenarios SC 2003 Demo, NCSA booth

9. Motivation for GAT Why do applications need a framework for using the Grid? Our application developers need a layer between applications and grid infrastructure: • Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs • Insulate against rapid evolution of grid infrastructure • Choose between different grid infrastructures • Make it possible for grid developers to develop new infrastructures • Make it possible for application developers to use and develop for the grid independent of the state of deployment of the grid infrastructure SC 2003 Demo, NCSA booth

10. End Users GAT-API Developers GAT Tool Developers Grid Infrastructure Developers Solution... • GAT – a layer between apps and emerging grid technologies • GridLab testbed/VO • Close cooperation between developers and deployers SC 2003 Demo, NCSA booth

11. GridLab Architecture SC 2003 Demo, NCSA booth

12. GridLab services • Software environment for Grid-enablingscientific applications • GridLab services, third party services and various core-grid services will be supported by GAT • In the advent of the Open Grid Service Architecture(OGSA), GridLab's architecture will revolve around the notion of services, • all the GridLab services will be OGSA compliant • currently all the services are Web Services based • roadmap for Web Services to OGSA transformation is being prepared (3-6 months from now) SC 2003 Demo, NCSA booth

13. GridLab services • A primary aim of this project is to produce a GridLab GAT containing a set of high quality services which provide a complete environment for Grid-enabling generic applications • GridLab services: • implement common (strict) security • use common service conneciton protocols (WSDL/OGSA) • are built primarily for Globus infrastructure SC 2003 Demo, NCSA booth

14. What are the GL services? • Authorisation Service (WP6) • Adaptive Services (WP7) • Data Management Services (WP8) • Resource Management System (GRMS) (WP9) • Information Services (WP10) • Monitoring Services (WP11) • Mobile User Support (WP12) SC 2003 Demo, NCSA booth

15. Security (WP6) • Security WP focuses right now on the Authorization Service (AS) • The main requirement is flexibility • The AS is about to provide universal way of defining security policy for the whole Grid, independent of technologies used at lower levels • It should be able to implement most security models for Gridsand use many different scenarios atthe same time SC 2003 Demo, NCSA booth

16. Adaptive Components (WP7) • Adaptive Components Service (ACS) and the Local Adaptive Components (LAC). • ACS provides an interface to query the adaptive system. It currently supports calls to: • rank resources • estimate transfer time • estimate usage (of some given metric) • LAC uses the monitoring system (shown in blue), to continuously collect data about the resource and applications running on it (load information, queue lengths, network bandwidth to other machines, etc.). SC 2003 Demo, NCSA booth

17. Data Management Services(WP8) • replica catalog prototype was ready at Zakopane meeting • data movement/copy service also since Zakopane meeting • supports reliable gridftp file transfer • is gsi enabled with authentication and delegation • scalable and fault-tolerant replica catalog in M12, based on ongoing research at ZIB Soap Soap • automatic load-balance, fail-over between replica catalogs • external access via SOAP and OGSA • internal communication via more efficient protocol (Corba) Host A Host C Soap Host B Soap Host D SC 2003 Demo, NCSA booth

18. GRMS - the plan Information Services Data Management Authorization System Adaptive Resource Discovery File Transfer Unit Jobs Queue BROKER Job Receiver Execution Unit Monitoring SLA Negotiation Scheduler Workflow Manager Resource Reservation Prediction Unit GRMS GLOBUS, other Local Resources (Managers) SC 2003 Demo, NCSA booth

19. GridLab and Condor SC 2003 Demo, NCSA booth

20. GridLab and GriPhyN SC 2003 Demo, NCSA booth

21. MDS Information Services Information Service (OGSA) Users Services Software Firewall GSI-SASL Client V.O. C. A. SOAP over GSI Cluster Job Queues SC 2003 Demo, NCSA booth

22. Mercure Monitoring System • Implements GGF’s GMA architecture • Fast and robust • Small resource usage • Can monitor hosts and jobs • Can deliver event notifications SC 2003 Demo, NCSA booth

23. Mobile User Support (WP12) User Mobile device Applications Portal Grid Services Network Environment / Grid SC 2003 Demo, NCSA booth

24. The Grid is complex … Application “Is there a better resource I could be using?” SOAP WSDL Corba OGSA Other Monitoring Profiling Information Logging Security Notification Resource Management Application Manager Migration Data Management GLOBUS Other Grid Infrastructure? UNICORE SC 2003 Demo, NCSA booth

25. …need to make it easier to use Application “Is there a better resource I could be using?” GAT_FindResource( ) GAT The Grid SC 2003 Demo, NCSA booth

26. Grid Application Toolkit • The GATprovides functionality through a carefully constructed set of generic high-level APIs, through which an application will be able to call the underlying gridservices, • Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled applications (open source!) • Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …) SC 2003 Demo, NCSA booth

27. GAT: AIM • Abstract Grid capabilities (services) from the application developer. • Application developer concentrates on the functionality as needed by the application. • Hide complexity. • Provides a layer (buffer zone) between applications and the Grid. SC 2003 Demo, NCSA booth

28. GAT Goals • The GAT provides an API and an associated setof tools which enable end-users and applicationdevelopers to make easy and flexible use of theGrid, • The infrastructure, and in particular the GAT,must allow developers to develop theirapplications independently of the deployment ofgrid services, • Users must be able to make use of suchapplications in the absence of a fully-deployedinfrastructure. SC 2003 Demo, NCSA booth

29. Cactus/GAT Integration Cactus Flesh Thorn Thorn GAT Library CGAT Thorn Thorn Thorn Thorn Cactus GAT wrappers Additional functionality Build system Physics and Computational Infrastructure Modules GridLab Service GridLab Service SC 2003 Demo, NCSA booth

30. GAT Adaptor • Interface between GAT Engine and one or more capabilities • Translates user requests to appropriate interface syntax for a capability provider • Active adaptors change dynamically • Includes “security context” • Return appropriate error codes • Examples • OGSA adaptor (provides many capabilities) • Globus adaptor (directly talk to gatekeepers) • Adaptors for each GridLab service provider • “Local” adaptors (GAT_MoveFile => “cp”, GATFindResource => “localhost”) SC 2003 Demo, NCSA booth

31. The Same Application … Laptop Super Computer The Grid Application Application Application GAT GAT GAT Firewall issues! No network! SC 2003 Demo, NCSA booth

32. Philosophy • Application makes GAT API calls for operations whichmay be Grid-related. • Application links against the GAT Engine • Application runs irrespective of actual underlyinginfrastructure deployment • Engine loads adaptors which are valid in the environment extantwhen the application starts • Adaptors try to do Grid operations on request, on failure anotheradaptor provided function may be called. • Application can thus be compiled, linked and testedwithout any Grid services • Same application executable can run in a full Gridenviroment. SC 2003 Demo, NCSA booth

33. Philosophy • The GAT uses whatever underlying Grid infrastructurethere is and that people have developed adaptors for, • GAT is not about replacing already developedinfrastructure, but instead to provide a simple, clearinterface which can be used with many differentinfrastructures. • Different versions of Globus • Condor • Unicore • ... SC 2003 Demo, NCSA booth

34. The GAT Architecture GAT: Grid Application Toolkit • API and Toolkit for developing portable Gridapplications independently of the underlying Gridinfrastructure and available services • Implements the GAT-API • Used by applications (different languages) • GAT Adaptors • Connect to capabilities/services • GAT Engine • Provides the function bindings for the GAT-API SC 2003 Demo, NCSA booth

35. API Goals • The GAT must support applications written in anylanguage which people write Grid Applications in: • C, C++, Fortran, Java, Perl, Python, ... • The use of the GAT API should be as natural aspossible for users of these languages. • It must also not require a steep learning curve tomove from the API in one language to the API inanother language • APIs in different languages should be assimilar as possible SC 2003 Demo, NCSA booth

36. Migration Scenario • Application migrates beacause of bad performance • The Goal: Involve all the WPs! • GAT application • Portal • GRMS • Adaptive • Monitoring • GIS • Mobile user support • Security • Data mgmt • Testbed SC 2003 Demo, NCSA booth

37. Migration Scenario SC 2003 Demo, NCSA booth

38. Migration Scenario SC 2003 Demo, NCSA booth

39. Migration Scenario SC 2003 Demo, NCSA booth

40. Migration Scenario SC 2003 Demo, NCSA booth

41. The „behind the scenes” goal of GL „Let’s make the most advanced grid in the world” Michael Russell, AEI ...and that’s what we do ...  SC 2003 Demo, NCSA booth

42. More info / summary • www.GridLab.org • gridlab@gridlab.org, news@gridlab.org • office@gridlab.org • Check the GridLab tutorials available at the Web • Bring your application and test it with the GAT and our services! SC 2003 Demo, NCSA booth