GridSphere’s Grid Portlets A Grid Portal Development Framework

1. GridSphere’s Grid PortletsA Grid Portal Development Framework Jason Novotny Lawrence Berkeley National Laboratory Jason Novotny, Michael Russell, Oliver Wehrens The GridSphere Project www.gridsphere.org 2004

2. The Purpose Of This Talk • The purpose of this talk is to provide a brief overview of the Grid Portlets web application. • We will describe the basic ideas behind Grid Portlets, what functionality it currently provides and development plans 2004

3. GridSphere’s Grid Portlets • The GridSphere portlet container is designed to be web application independent. Indeed, one of the key advantages of the Portlet API is the reuse of web applications. • Thus, the GridSphere portlet container does not contain any support for using Grid technologies. • Instead, GridSphere’s Grid related functionality is contained in a web application we call Grid Portlets. • Grid Portlets, together with the GridSphere portlet container, offers a generic Grid portal and can be used to develop application-specific Grid portal applications. 2004

4. The GridLab Project • Funded by the EU (5+ M€), January 2002 – December 2004 • Application and Testbed oriented • Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled • Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools...: • resource management (GRMS), • data management, • monitoring, • adaptive components, • mobile user support, • security services, • portals, ... and test them on a real testbed with real applications 2004

5. GridLab Is An Architecture 2004

6. And A Global Effort • PSNC (Poznan) - coordination • AEI (Potsdam) • ZIB (Berlin) • Univ. of Lecce • Cardiff University • Vrije Univ. (Amsterdam) • SZTAKI (Budapest) • Masaryk Univ. (Brno) • NTUA (Athens) • Sun Microsystems • Compaq (HP) • ANL (Chicago, I. Foster) • ISI (LA, C.Kesselman) • UoWisconsin (M. Livny) • collaborating with: • Users! • EU Astrophysics Network, • DFN TiKSL/GriKSL • NSF ASC Project • other Grid projects • Globus, Condor, • GrADS, • PROGRESS, • GriPhyn/iVDGL, • Most of the other European Grid Projects (GRIDSTART) • GWEN 2004

7. GridLab Testbed Snapshot 2004

8. GridLab Goals • Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community), all the other potential grid apps • Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations • Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources. 2004

9. What Do Our Users Need? • Application oriented environment • Flexible, easy-to-use, simple interfaces • Efficient and effective use of resources • Robustness, fail-safety, adapability • The ability to work in distributed teams • Support for mobile working environments 2004

10. Larger computational resources Memory/CPU Faster throughput Cleverer scheduling, configurable scheduling, co-scheduling, exploitation of un-used cycles Easier use of resources Portals, grid application frameworks, information services, mobile devices Remote interaction with simulations and data Notification, steering, visualization, data management Collaborative tools Notification, visualization, video conferencing, portals Dynamic applications, New scenarios Grid application frameworks connecting to services What Do Our Users Want? 2004

11. Dynamic Staging move to faster/cheaper/bigger machine Multiple Universe create clone to investigate steered parameter Automatic Convergence Testing from initial data or initiated during simulation Look Ahead spawn off and run coarser resolution to predict likely future Spawn Independent/Asynchronous Tasks send to cheaper machine, main simulation carries on Application Profiling best machine/queue choose resolution parameters based on queue Dynamic Load Balancing inhomogeneous loads multiple grids Portal User/virtual organisation interface to the grid. Intelligent Parameter Surveys farm out to different machines Make use of Running with management tools such as Condor, Entropia, etc. Scripting thorns (management, launching new jobs, etc) Dynamic use of eg MDS for finding available resources Many Application Scenarios! 2004

12. Where Grid Portals Fit In • GridLab needs an effective way to deliver its tools and services to its target users. • We want to avoid having to develop new client-side technologies and methods for deploying them. • Given the ubiquitous Web browser and the evolution of Grid standards and technologies, a Grid portal provides an obvious solution to this problem. 2004

13. Developing Grid Portals • Grid web application development still remains a tedious task with little in the way of reusable components, forcing developers to constantly “re-invent” the wheel. • Often difficult and hard to maintain glue code must be written connecting the portal to Grid services, due to lack of/evolving standards. • Lack of real usability has made it difficult to test and evaluate user interfaces. • A portal is only as good as the underlying deployed infrastructure…. portal development often involves debugging underlying middleware 2004

14. What is a Grid Portal? • “A portal is a web based application that commonly provides personalization, single sign on, content aggregation from different sources and hosts the presentation layer of Information Systems”(JSR 168) • Grid Portals build upon the familiar Web portal model, such as Yahoo or Amazon, to deliver the benefits of Grid computing to virtual communities of users, providing a single access point to Grid services and resources. 2004

15. Early Grid Portal Projects • Grid-Port: • Perl based framework developed by Mary Thomas and Steve Mock at San-Diego Supercomputing Center (SDSC) • Grid Portal Development Toolkit (GPDK): • Developed by Jason Novotny at Lawrence Berkeley National Laboratories (LBNL) • Astrophysics Simulation Collaboratory (ASC): • Developed by Michael Russell at University of Chicago 2004

16. Grid Portals Work Package • Within the GridLab Project, the Grid Portals Work Package (WP4) is responsible for developing a Grid portal for the GridLab Project. • The GridSphere Project was created by WP4 as it saw a need in the general community for a Grid portal framework. • WP4 is both developing and deploying GridSphere to serve target user communities in the GridLab Virtual Organization. 2004

17. GridSphere 2.0 2004

18. Personalized Environment 2004

19. Single Sign-On Capabilities 2004

20. Submit Jobs 2004

21. Perform File Transfers 2004

22. Manage Resources 2004

23. Grid Portal Design Requirements • We require a portal that is generic and not tied to any particular set of Grid technologies. The GridLab Testbed, for example, is Globus 2.X based although integartion with OGSA well under way • Develop a set of portlets and portlet services that provide a higher level model of Grid concepts.The Grid portlet services should provide a generic API to which the Grid portlets are written. 2004

24. Grid Portlets API • Our Grid Portlets API attempts to provide a high-level model of Grid concepts that • Abstracts the developer from particular Grid technologies and infrastructure. • Is extensible, can build upon its concepts to provide more complex services, resources and tasks. • Supports multiple implementations of the API. • Configurable at run-time via administrative interfaces • The Grid Portlets API builds from basic concepts to more advanced ones… 2004

25. Basic Concepts • Users • Users should get a personalized a view of the Grid based upon their access rights to resources and their preferences. • Credentials • Credentials are required to authenticate users to resources and authorize their rights to utilize them. • Resources • Resources are anything that can be “utilized” (by users, portlet services and other resources). • Tasks • Many operations are quite naturally modeled as tasks. Workflow is not provided yet, but is desired in future releases. 2004

26. More Advanced Concepts • Jobs • Job submission to LSF, PBS, etc… • Job migration from one resource to another • Files & Related Tasks • Physical file browsing (list, transfer, etc.) • Logical and replica file browsing • Information Resources • LDAP/MDS information providers • Web service based providers • Other types… 2004

27. Simple Resource Diagram 2004

28. Hardware Resource Model • Hardware resources are utilized for performing computations, storing files, etc…. • Hardware resources contain… • Service resources (such as a Grid FTP resource) • Software resources (such as Globus client tools) • Hardware accounts (allowing people to login and perform operations in a shell). 2004

29. Job Resource Model • Job resources (such as Globus Gatekeepers) are service resources that enable the remote clients to submit jobs on computing resources. • Job resources provide access to job schedulers (such as PBS). • Job schedulers contain queues with different resource capabilities and usage policies. 2004

30. Some Job Resources • In Grid Portlets, we’ve modeled two types of service resources as job resources, both of which have different capabilities for supporting job submission. • GTX Gatekeepers provide the ability to submit jobs directly to job managers running on hardware resources using the GRAM protocol. • GridLab Resource Manager Services (GRMS)s are web services that provide resource brokering capabilities and support job submission with GRAM as well as Condor. 2004

31. Simple Task Diagram 2004

32. How Portlets Are Developed • In Grid Portlets, the presentation layer is made up of portlets which contain reusable UI components. • The business logic layer is implemented as a collection of portlet services which provide some high-level capabilities and make use of external resources. 2004

33. Job Submission Portlet • For example, the job submission portlet uses a number of UI components, including file browser components (i.e. for specifying locations of an executable). • These JSP components utilize a number of portlet services which provide access to job resources, file resources and so on… 2004

34. Current Status • Grid Portlets is still in prototype development mode. • Most of what is their works, but API is still evolving. • Documentation is still poor… Mike promises to improve it • Source code is available via anonymous CVS. • Will move towards a regular development and release cycle in the coming months. • Mike in Brazil ;-) but plans on continuing to enhance grid portlets 2004

35. Current Functionality • Credential Retrieval & Management • Support for retrieval & storing with MyProxy • Resource Registry & Management • Resources can be described in XML and online • Resource Information Provider • Resource discovery and update with MDS2 • Job Submission & Migration • GRAM job submission with Java CoG 1.1 • GRMS 1.9 job submission and migration • File Browsing & Transferring • Support for GridFTP with Java CoG 1.1 2004

36. What’s Next? • Enhancements to UI component model… These enhancements may be incorporated into GridSphere portlet container, but we are still reviewing JSF. • Many enhancements planned to existing portlets, including better integration among portlets and reuse of UI components. • Progress already underway to make JSR 168 compliant • Please submit feature requests and additions to bugtracker and mail list 2004

37. Planned Development • Support for logical files & replica management • Support for GridLab Data Management Services • Support for Globus Replica Catalogue • Enhancements to information gathering • Portlets for managing use of information providers • More resource information providers • Support for GridLab Adaptive Information Service • Support for Network Weather Service 2004

38. How To Obtain Grid Portlets • Grid Portlets can be obtained from CVS…. • Required Software: • GridSphere 2.0.2 or higher. • Tomcat 4.1.29 or higher (Tomcat 5.X supported) • OGSA 3.0.2 (Plan to move OGSA 3.2) • MyProxy cvs -d :pserver:anonymous@portal.aei.mpg.de:/home/repository co gridportlets 2004

39. Configuring Grid Portlets • Configuration amounts to specifying what resources Grid Portlets makes available to users. • The first thing you should do is specify what MyProxy resource your portal will use for retrieving / storing Grid credentials. • If want to use our MDS2 information provider, then you should specify what GIIS resource you want to use. • Otherwise, you should make sure to specify the hardware and service resources that make up your Grid… 2004

40. How To Specify Resources • There are 2 ways to add resources to Grid Portlets • Online with the Resource Registry Portlet • Offline with Resources.xml 2004

41. How To Get Involved • Join one of gridsphere mail lists. Ask questions, provide feedback / comments. • Submit bugs and feature requests to http://bugs.gridsphere.org. • Because we are a small group, getting involved as a developer means demonstrating some knowledge / commitment to the project. Best way is to email ideas / patches to the list. If you’re loud enough, we’ll probably give you commit access and a few tasks! 2004

42. Projects Using Grid Portlets • AEI Numerical Relativity Portal • CCT Numerical Relativity Portal • NRC Portal • Unversity of Melborne, Australia • SDSC… 2004

43. Conclusion • Grid Portlets contains the Grid functionality in the GridSphere Project. It provides a framework for developing Grid portals. • It is still in prototype development mode, but we are already using it in the GridLab Project and in several other projects with which we are affiliated. • We will make announcements on our website at www.gridsphere.org as the status of Grid Portlets changes. • Feel free to try it out and send us comments. 2004