Scripting Web Services: Harnessing HPSearch & NaradaBrokering

1. HPSearch Design & Development via Scripting Harshawardhan Gadgil Dr. Geoffrey Fox, Dr. Marlon Pierce

2. Why Script Web Services • Scripting Advantages • Rapid prototyping • Scripting Web Services • Manipulate data streams, files and web-services using scripting language. Equivalent to shell and Perl in UNIX. • Allows us to control the invocation and execution of disparate services using scripting. Equivalent to programming a workflow • Setting up data-flow among components by dynamically joining disparate data streams • Rapidly deploy applications (under development) • We have developed a tool for binding URIs to a scripting language that manipulates the local / remote resource identified by the URI - HPSearch

3. HPSearch • Binds URI to a scripting language • We use Mozilla Rhino (A Javascript implementation, Refer: http://www.mozilla.org/rhino), but the principles may be applied to any scripting language such as Perl, Python etc… • Every Resource may be identified by a URI and HPSearch allows us to manipulate the resource using the URI. • For e.g. Read from a web address and write it to a local file x = “http://trex.ucs.indiana.edu/data.txt”; y = “file:///u/hgadgil/data.txt”; Resource r = new Resource(“Copier”); r.port[0].subscribeFrom(x); // read from r.port[0].publishTo(y); // write to f = new Flow(); f.addStartActivities(r); f.start(“1”);

4. HPSearch (contd.) • Currently provide bindings for the following • file:// • socket://ip:port • http://, ftp:// • topic:// • jdbc: • Host-objects to do specific tasks • WSDL – invoke web-services using SOAP • PerfMetrics – Bind NaradaBrokering performance metrics. Store published metrics and allow querying • Resource – Every data source / filter / sink is a resource. • Flow – To create a data flow between resources. Useful for streaming applications. • For more information, visit • http://www.hpsearch.org

5. NaradaBrokering • Event- brokering system designed to run on a large network of co-operating brokers. • Guarantees delivery in presence of failures • Implements high-performance protocols (message transit time < 1 ms per broker) • Order-preserving optimized message transport • Interface with reliable storage for persistent events • Fault tolerant data transport • Support for different underlying transport implementations such as TCP, UDP, Multicast, SSL, RTP

6. HPSearch + NaradaBrokering • HPSearch uses NaradaBrokering to route data streams • Consider the script as follows Resource a = new Resource(“ResourceA”); a.port[0].subscribeFrom(“file:///u/hgadgil/data.txt”); a.port[0].publishTo(“topic:///x”); // Optional Resource b = new Resource(“ResourceB”); b.port[0].publishTo(“file:///tmp/rawData.dat”); b.port[0].subscribeFrom(a.port[0].publish); f = new Flow(); f.addComponents(b); f.addStartActivities(a); f.start(“1”); WFE WFE Resource B Resource A HPSearch Shell

7. HPSearch - WSProxy HPSearch Shell WSProxyClient can control the WSProxy using simple Web-service calls Request Handler Flow Handler WFE Pipe DB Connector WFE WSProxyClient Broker Network WSProxy WSProxy is a simple web-service that listens for requests from the client (WSProxyClient) and steers the service Service The WSProxy connects to a broker and handles all the data communication on behalf of the service WFE

8. Flood Modeling Example using Web Service scripting: Components • Rainfall Data • Inches of rainfall gathered by the sensor network at specific intervals (Simulated to be in intervals of 30 min in our example) • WebMap Server • OpenGIS based terrain data repository • Generates Watershed model • Compute Service – Runoff Model • Computes the runoff model using the Hydrostatic watershed model and Hydrodynamic rainfall data • Visualization Service • Shows the visual representation of the runoff model for the terrain in question • Each of these components run as a separate Web service (Either direct or wrapped as a web service) • Scripting to manage the chain of these disparate services into one complete application • Each of these components is wrapped using WSProxy which is then controlled by the Workflow Engine

9. Crisis Grid - Flood Modeling Map Service • Compute Service is the basic control • On start up, MapService sends the spatial information for the terrain under consideration to the Compute service. • Compute Service requests the Rainfall publisher to send the next rainfall reading (pulls the next reading). • The Compute service then operates the run off model for flood analysis. • The computed results are then sent to a Visualization service which displays the computed predicted model results in form of an animation Rainfall Publisher Compute Service Visualization

10. HPSearch Java Script Code for creating and initializing the WMS Component maps = "/CGL/FLOODMODEL/MAPS"; rain = "/CGL/FLOODMODEL/RAINFALL"; visu = "/CGL/FLOODMODEL/VISUAL"; mapServer = "org.hpsearch.demo.floodModel.MapService"; mapServerLoc = "http://156.56.104.170:9090/axis/services/WSSConnector?wsdl"; mapSource = new WebServiceHandler(mapServer); mapSource.setEndPointURI(mapServerLoc); mapSource.setParameter("MAPSERVERTOPIC", maps); rainFallPublisher = "org.hpsearch.demo.floodModel.RainfallPublisher"; rainFallPublisherLoc = "http://156.56.104.170:6060/axis/services/WSSConnector?wsdl"; rainFall = new WebServiceHandler(rainFallPublisher); rainFall.setEndPointURI(rainFallPublisherLoc); rainFall.setParameter("RAINFALLTOPIC", rain); rainFall.setParameter("VISUALIZATIONTOPIC", visu); computeService = "org.hpsearch.demo.floodModel.ComputeService"; computeServiceLoc = "http://156.56.104.170:7070/axis/services/WSSConnector?wsdl"; compute = new WebServiceHandler(computeService); compute.setEndPointURI(computeServiceLoc); compute.setParameter("MAPSERVERTOPIC", maps); compute.setParameter("RAINFALLTOPIC", rain); compute.setParameter("VISUALIZATIONTOPIC", visu); crisisGridFlow = new Flow(); crisisGridFlow.addComponents(compute); crisisGridFlow.addStartActivities(mapSource, rainFall); crisisGridFlow.start("1"); Code for creating and initializing the Rainfall data Component Code for registering the components of the flow and starting it Code for creating and initializing the Compute service

11. How to initialize… • First Define the service to be invoked mapServer = "org.hpsearch.demo.floodModel.MapService"; • The location of the service. This should be normally done via a registry lookup. For For E.g. NBDiscover.discover(“MapService”, specs); Its hard-coded as of now mapServerLoc = "http://156.56.104.170:9090/axis/services/WSSConnector?wsdl"; • Initialize aWebService Handler. This object invokes a WSProxyClient that interacts with the WSProxy which is a wrapper on the actual service mapSource = new WebServiceHandler(mapServer); • Set the End-point URI mapSource.setEndPointURI(mapServerLoc); • Set the service initialization properties. This is optional, but allows for more customization. This could include QoS properties, security tokens and any other service specific parameters mapSource.setParameter("MAPSERVERTOPIC", maps);

12. How to Initialize (contd.) • Finally we create a new Flow Object. This object helps to serialize the requests for setting up a new Flow. The FlowHandler component of the WorkflowEngine then distributes these requests to other engines and the flow is finally started. crisisGridFlow = new Flow(); • We add the components of the flow. All components which generate the data to be processed are startActivities and are started after the other components are started. This enables the components to start processing the data as soon as the startActivities publish the data crisisGridFlow.addComponents(compute); crisisGridFlow.addStartActivities(mapSource, rainFall); • Finally we start the flow. The ”1” is for debugging purposes. Ignore it for now crisisGridFlow.start("1");

13. In Short…(w.r.t. Flood model) • Handle file-based data. • Read from static files (archived data), databases, For E.g. The terrain data is static • Support streaming data sources. For E.g. Rainfall data read in real-time in a stream. • In an actual system we may expect the frequency of observation (and hence data generation) to be about 5 to 20 minutes. • In our system, we assume just one source for rainfall data but in an actual scenario, multiple data sources possible

14. Wrapping a Simple Service • Two implementations of ProxyWebService • RunnableProxyWebService • Provides operations START, STOP, SUSPEND, RESUME. • Service is provided the required Input and Output streams from where it will read and write. • Good for applications which work on a chunk of data at a time • Easiest way to create your first service • WrapperProxyWebService • Provides just the 4 operations as above. • The application handles its own communication • Example (next slide)

15. Coding a simple string reversal service package mypackage; public class StringProcessingService extends RunnableProxyWebService { BufferedReader input; BufferedWriter output; public void initService(InputStream[] in, OutputStream[] out) { input = new BufferedReader(new InputStreamReader(in[0])); output = new BufferedWriter(new OutputStreamWriter(out[0])); } public String[] serviceExceptions() { String[] exceptions = { "java.io.EOFException", "java.io.IOException“}; return exceptions; } MUST initialize the service in this fashion Enables the WSProxyClient to add error specific handlers. You may return an empty array also.

16. Coding a simple string reversal service public void process() throws Exception { String s, newS; try { s = input.readLine(); if (s == null) { serviceFinished(); return; } } catch (IOException e) { throw e; } newS = reverseOf(s); output.write(newS); output.newLine(); output.flush(); } public String reverseOf(String s) { … } } Your code that processes the data

17. Deploy using AXIS • Create a deployment descriptor for AXIS • Or use one which already exists • Deploy the service • Finally run the script to start using the Filter

18. Invoking our simple service service = “mypackage.StringProcessingService”; serviceLoc = “http://...path to WSDL”; stringproc = new WebServiceHandler(service); stringproc.setEndpointURI(serviceLoc); stringproc.setInput(“input URI”); stringproc.setOutput(“output URI”); f = new Flow(); f.addStartActivities(stringproc); f.start(“1”);

19. Useful links for additional information • WSProxy • http://www.hpsearch.org/notes/wsproxy • Technical Report on the flood model • http://www.hpsearch.org/documents/crisisGridFlow.pdf • Timing Analysis etc… (incomplete as of now) • http://www.hpsearch.org/notes/notes/9-timing.html

20. Credits… • Dr. Sunghoon Ko (CGL) and Dr. Jin-Yong Choi (Purdue Univ.) for the Flood Demo • NB – team @ CGL, Dr. Pierce, Dr. Fox