Developing Grid Portals Using Portlets

1. Developing Grid Portals Using Portlets Marlon Pierce Community Grids Lab Indiana University

2. Overview of Material • General remarks on portals and portlets. • General remarks to provide context for the remainder of the session. • OGCE Portal capabilities. • Summary of things we do in OGCE. • JSR 168 Review • Developing OGCE portlets • How to develop OGCE compatible portlets. • With launching points for Dennis and Gregor. • Useful third party testing tool: HttpUnit. • A brief JSR 168 tutorial. • At the risk of a low tutorial rating, I will cover higher level issues. Slides on nuts and bolts details are provided for homework.

3. Towards A Common Grid Client Hosting Environment Grid portal background and emerging common frameworks

4. What Is a Grid Computing Portal? • Browser based user interface for accessing grid and other services • “Live” dynamic pages available to authenticated, authorized users. • Use(d) Java/Perl/Python COGs • Manage credentials, launch jobs, manage files, etc. • Hide Grid complexities like RSL • Can run from anywhere • Unlike user desktop clients, connections go through portal server, so overcome firewall/NAT issues • Combine “Science Grid” with traditional web portal capabilities • Get web pages for news feeds • Post and share documents • Search engine interfaces, calendars, etc. • Enabled by portlets, as we will see. • Customizable interfaces and user roles/views

5. What a Grid Portal Is/Is Not • It is • A tool for aggregating and managing web content • A user customizable view of these Web content pieces. • You see what you want/can see. • But you must log in. • Implemented on top of standard services • Like login, authorization, customization. • May include collaboration, etc, that depend on login. • A way to accomplish Grid tasks through browsers: • Launch, monitor jobs • Move files • Run science applications based on these services. • Compatible with emerging standards and best practices (such as portlets, JSR 168 and WSRP). • It is not (just) • A web page • A collection of links • An applet

6. Which Is the Portal?

7. Which Is the Computing Portal? • In fairness, the screenshots are not large enough to see, but you have to log in to the one on the right. • Point is that they are superficially similar to browser users, but have many differences under the hood. • The screen shot on the left is of the NASA JPL QuakeSim project page. • http://quakesim.jpl.nasa.gov/ • The screen shot on the right is the NASA JPL QuakeSim portal. • http://www.complexity.ucs.indiana.edu:8282 • Go here to run QuakeSim earthquake simulation codes, access earthquake databases, etc.

8. Let 10,000 Flowers Bloom • Many portal projects have been launched since late ’90s. • HotPage from SDSC, NCSA efforts, DOD, DOE Portals, NASA IPG • 2002 Special Issue of Concurrency and Computation: Practice and Experience. • Continue to be important component of many large projects • NEESGrid, DOE SciDAC projects, NASA, NSF, many international efforts • Global Grid Forum’s Grid Computing Environments (GCE) Research Group • Community forum

9. Three-tiered architecture is accepted standard for accessing Grid and other services Three-Tiered Architecture Grid and Web Protocols JDBC, Local, or Remote Connection Portal Client Stub Database Service Database Portal User Interface Portal Client Stub Grid Resource Broker Service HPC or Compute Cluster Portal Client Stub Information and Data Services Grid Information Services, SRB

10. Problem with Portals • GCE revealed two things • Everyone was doing the same thing • Not quite, but significant • Everyone builds secure logins, remote file manipulation, command execution, access to info servers. • Everyone would at least like support for multiple user roles (administrators, users) and customization • No one could share components with other groups • No well defined way of sharing UI components or making services interoperate. • No well defined interfaces to portal services. • A research opportunity! • Two levels of integration: user interfaces and services • Our challenges • Stop reinventing things and provide ways for groups to reuse components. • Provide a portal marketplace for competing (advanced) services. • Provide APIs for service integration

11. A Solution based on components • A software component is object defined by • A precise public interface • A semantics that includes a set of “standard” behaviors. • A Software component architecture is: • A a set of rules for component behavior & • A framework in which components can be easily installed and interoperate. • The component architecture of choice for the Portal community is the one based on portlets • (Java) components that generate content, make local and remote connections to services. • Portlet containers manage portlet lifecycles • We have now many, many components. • So don’t start from scratch.

12. Things to Hate About Portals • If you involved in portal efforts, be aware of the following: • Browsers have limited interactivity. • Desktop GUIs provide much better interactivity but have other problems. • Applets are a solution, but they don’t interact with other parts of the browser very well. • Solution: Service Oriented portals let you use services through both portals and grid desktops. • Developing really useful user interfaces to your set of services is a time consuming, non-scaling process. • Get users involved early in design. • Browsers notoriously have incompatible features. • Things don’t work the same on IE, Mozilla, Konqueror, etc. • Same browsers on Macs, Windows don’t work the same. • No substitute for lots of testing. • Grid portals need grids. • Setting up grids is a fragile process. • Plenty of dumb mistakes like setting up CA signing policies that even experts make. • You will be blamed when things don’t go smoothly.

13. OGCE: Open Grid Computing Environments Marlon Pierce Indiana University

14. NSF NMI Project for Reusable Portal Components: Who We Are • University of Chicago • Gregor von Laszewski • Indiana University • Marlon Pierce, Dennis Gannon, Geoffrey Fox, and Beth Plale • University of Michigan • Charles Severance, Joseph Hardin • NCSA/UIUC • Jay Alameda, Joe Futrelle • Texas Advanced Computing Center/San Diego State University • Mary Thomas

15. What Is OGCE’s Release 1 • The OGCE Portal release is based on CHEF/Jetspeed 1.4 • Available for download and installation from http://www.collab-ogce.org. • It comes with many pre-configured capabilities if you want a grid portal “out of the box”. • Except for the mysql jar. • You must still set up Grid services (MyProxy servers, Globus, etc). • Globus version compatibility through the Java CoG. • Apache Ant-based installation procedure: • Edit one properties file, run ant, and away you go.

16. User Portlets

17. More User Portlets and Services

18. Grid Portlet Examples • We’ll next overview several portal capabilities. • Jetspeed/CHEF acts as a clearing house for portal capabilities • User interface components can be added in well defined ways. • First level of integration • All Grid access goes through the Java COG.

19. Example Capability: Portals for Users User “Beth” • The user contacts the portal server and asks it to do “grid” things on behalf of the user. • To make this possible the server needs a “Proxy Certificate” • The user has previously stored a proxy cert in a secure MyProxy Server stored with a temporary password. • User give the portal server the password and the portal server contacts the proxy server and loads the proxy. • The portal server will hold the proxy for the user for a “short amount of time” in the user’s session state. 1. Load my Proxy Certificate! Portal Server MyProxy Portlet 2. Give me Beth’s proxy certificate COG 3. I am Beth’s Proxy MyProxy Server

20. Example Capability: Grid Context Service • User’s want to be able to use the portal to keep track of lots of things • Application and experiment records • File metadata, execution parameters, workflow scripts • “Favorite” services • Useful directory services, indexes, links to important resources • Notes and annotations • “Scientific Notebooks”

21. Portal Server Portlet Interfaces to Grid Context • A Remote Service Directory Interface • Holds references and metadata about application services. • User selects interface to application service from the directory browser. • Examples: (near completion) • Select a link to a Dagman document and invoke the Condor service on the script. • Same for GridAnt/Ogre or BPEL workflow script. • Factory services for any grid apps that have interactive user interfaces. Remote Service Directory Service Remote Grid Application Service

23. Example Capability: Topic Based Messaging Systems • XML metadata system based on messages. • Newsgroups • Topic based message posting and administration • Citation/reference browsers • Topic based, export/import bibtex • Portlets sit atop JMS-based message system • NaradaBrokering, used in JMS mode.

24. User Privileges for Group Channels • Users request access to specific topics/channels. • Granted by administrator for that topic • Can request • Read/write by browser • Read/write by email (newsgroups) • Receive/don’t receive attachments. • Topic admin can edit these requests. • Super admins can manage administrators to topics

25. Load - aggregated CPU Downtime data for a machine Jobs: aggregated queue MOTD Nodes: job usage for each machine node NWS: based on VO and Click model Grid Monitoring Based on TACC GMS System Custom providers Plans to include MDS3.0 and INCA data uderway Expanding to include: queuing system application profiles performance data Application profiles Doc links Model allows generic inclusion of any XML data from any recognized source Need schema Need query GPIR Data

26. Grid Portal Information Repository (GPIR 1.1)

27. GPIR Components • Web Services Ingestor • Web Services Ingestor and clients • XML Schemas - can be changed • Data Repository • Local Cache • Archival --> PostgreSQL • Web Service Query • retrieve data – XML Queries • Retrieving current snapshot and archived data • Clients • GridPort services • Portal/Web Interface (Portlets, servlets, JSP) • Command line • Any that speak web services

28. What’s Next for the OGCE? • JSR 168 Compatible Grid portlet release at Supercomputing. • Basic capabilities: MyProxy, GridFTP, GRAM, GPIR, based on CoG 4. • Working in uPortal, GridSphere, Jetspeed2, …. • Join (all of) us at SC2004 for a portals BOF. • Solutions JSR 168 limitations that work in multiple JSR 168 containers. • Grid-compatible WSRP portlets. • Compatible with Sakai/CHEF 1.2 capabilities. • Backward compatibility bridges to previous portlets. • Fancy new capabilities available in separate downloads.

29. A JSR 168 Overview A review of the latest portlet developments.

30. What is JSR 168? • Defines a standard for vendor container-independent portlet components. • Many implementations: • Gridsphere, uPortal, WebSphere, Jetspeed2, …. • From the portlet development point of view, it is really very simple: • You write a java class that extends GenericPortlet. • You override/implement several methods inherited from GenericPortlet. • You use some supporting classes/interfaces • Many are analogous to their servlet equivalents • Some (portletsession) actually seem to be trivial wrappers around servlet equivalents in Pluto.

31. Some Terminology

32. Some Generic Portlet Methods

33. Supporting Classes/Interfaces

34. The Infamous Big Picture • As a portlet developer, the previous set of classes are all you normally touch. • The portlet container (such as Pluto or Gridsphere) is responsible for running your portlets. • Init, invoke methods, destroy. • Portlets have a very limited way of interacting with the container. • It is a black box. • The API is basically one-way.

35. Deploying Portlet Applications • The portlet container (i.e. uPortal) runs as a distinct web application. • That is, it has its own directory in tomcat. • Moreover, it runs as a separate context, with its own classloader, session management, etc. • Portlet applications are deployed as distinct war files/web applications. • You go through the container webapp to get to the portlet webapp. • Portlets in the same application share jars, classes, and runtime stuff like request and session variables. • Portlets in different portlet apps do not share anything.

36. A Critique of JSR 168 • There is no way to share data/objects between portlet applications. • So all grid portlets would have to be in the same portlet app. • There is no way to extend the portlet API to add such services. • There is a lack of general purpose portlets. • Right now, you make specific extensions to GenericPortlet for each portlet you develop. • JSR 168’s MVC approach is incompatible with Turbine, Struts, …. • The issue is managing <form action=“”> and href URLs. • No fundamental problem, but this is a gap that must be filled. • JSF Portlets are the exception. • No defined way to integrate with portal-specific services (i.e. logins). • No inter-portlet communication. • Despite these problems, JSR168 (and WSRP) are the best available standards. • They are compatible. • WSRP overcomes many of the JSR 168 limitations. • But more work needs to be done to make WSRP ready for Grid portals (Web Service security).

37. Writing Basic OGCE Portlet Templates Developing Portlets for OGCE Release 1

38. Writing a Velocity Portlet • Imagine the following simplistic scenario: • I have a web form that helps a user submit a job to a supercomputer. • I only need one input form to generate the input file, pick a host computer, etc. • The output file is written back to the user’s home directory, so he or she can gridftp it later. • Before we can do this, we first need a bit of background on programming portlets.

39. Velocity Portlet Example • Before handling the Grid execution, we will first construct a dummy form. • We will use Velocity templates. • Jetspeed has better support and documentation for Velocity. • Most of the portlets in the release use this. • We will examine a JSP example later.

40. Open a new file in nmi/WEB-INF/templates/vm/portlets/html. Give it a name, myTestApp.vm Write the Velocity template Write an xreg file (see next slide). Restart Tomcat. Customize your display to show the new portal. So far, so good. <h3> Enter the command you want to execute</h3> <form method="post"> <table> <tr> <td> Command Name:</td> <td> <input type="text" name="runCommnad" value=""> </td> </tr> </table> </form> Create the Template

41. Sample JSP Portlet Template <?xml version="1.0" encoding="UTF-8"?> <registry> <portlet-entry name="myScienceApp" hidden="false" type="ref" parent="CustomizerVelocity" application="false"> <meta-info> <title>Sample Proxy Form</title> <description>Sample Proxy Form</description> </meta-info> <classname>org.apache.jetspeed.portal.portlets.VelocityPortlet</classname> <parameter name="template" value="myScienceApp" hidden="true" cachedOnName="true" cachedOnValue="true"/> <parameter name="action" value="portlets.myScienceAppAction" hidden="true" cachedOnName="true" cachedOnValue="true"/> <media-type ref="html"/> <url cachedOnURL="true"/> </portlet-entry> </registry>

42. The parts in red (template and action) point to things that you must write. The line below is used to name the VM template in the XREG. Points to myScienceApp.vm <parameter name="template" value="myScienceApp" hidden="true" cachedOnName="true” cachedOnValue="true"/> Specifying The Template

43. Actions in Templates • Note our velocity template is just HTML (at this point) with a form action. • The action implementation is specified in the XREG file. • MyScienceAppAction.java is the code that does the work when you click the button. • Jetspeed action managers are responsible for calling your actions. • You just need to write the java code, put it in the right place, and connect it to a Velocity template in the XREG file. • Jetspeed will take care of the invocation.

44. Writing An Action • A portlet action is just a java class. • It should extend VelocityPortletAction • You should implement the buildNormalContext() method. • This method is called by default when you invoke an HTML form action. • This can do anything you want (i.e. make calls to Grid services through the Java COG). • You can also implement other action methods.

45. Let’s give our simple portlet an action. To do this, we first modify the Don’t forget to shutdown tomcat first. We then write the action and compile it into nmi/WEB-INF/classes. See next slide Set classpath correctly! Restart the server. <parameter name="action" value="portlets.myScienceApAction" hidden="true" cachedOnName="true" cachedOnValue="true"/> Getting Started

46. A Minimal Action: myScienceAppAction.java package org.apache.jetspeed.modules.actions.portlets; //Import Turbine packages. import org.apache.turbine.util.RunData; //Import Velocity packages import org.apache.velocity.context.Context; //Import Jetspeed packages. import org.apache.jetspeed.modules.actions.portlets.VelocityPortletAction; import org.apache.jetspeed.portal.Portlet; public class myScienceAppAction extends VelocityPortletAction { public void buildNormalContext(VelocityPortlet portlet, Context acontext, RunData rundata) { //Real action code goes here. System.out.println("buildNormalContext called"); } }

47. Some Miscellaneous Notes • This action is automatically called whenever the JSP template’s form action is invoked. • In the portal release, the chef-1.0.7/modules/nmi-lib directory contains all the jars needed to compile this code. • You can use our ant build scripts as templates for writing your own. • The portlet actions’s system.out.println() is written to catalina.out. • If you prefer, you can use Jetspeed’s logger, which writes to nmi/WEB-INF/log.

48. RunData, Requests, and Sessions • The method buildNormalContext includes an object called RunData in its arg list. • RunData is your access point to all HTTP request, response, and session data. • HttpSession session=rundata.getSession(); • HttpServletRequest req=rundata.getRequest(); • From here, you can do standard java.net.* development. • I am not planning to cover this, but we can discuss.

49. Connecting Multiple Templates • In reality, a single web form is not enough to set up a complicated input file, select a host and execute a job. • These may be spread over multiple linked forms. • Form actions in templates must be handled a bit differently. • <form action=“SomeOtherPage”> • This can’t point to a template, since it is not directly accessible (in WEB-INF). • Jetspeed actions handle this.

50. Redirecting to Another Page • Setting up and running applications from a portal typically requires many successive HTML forms. • So we need to see how to do this with Jetspeed. • Let’s call this myScienceApp2.vm and place it in the same place as myScienceApp.vm. myScienceApp myScienceApp2 Response Request myScienceAppAction