1 / 17

Scripting based architecture for Management of Streams and Services in Real-time Grid Applications

This presentation introduces the HPSearch architecture, a scripting-based runtime for managing streams and services in real-time grid applications. It explores the system goals, provides an example of RDAHMM, and discusses related and future work.

jwarren
Download Presentation

Scripting based architecture for Management of Streams and Services in Real-time Grid Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Scripting based architecture for Management of Streams and Services in Real-time Grid Applications Authors Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon Pierce Community Grids Lab, Indiana University, Bloomington Robert Granat NASA, Jet Propulsion Lab, Pasadena Presented By Harshawardhan Gadgil hgadgil@cs.indiana.edu

  2. Talk Outline • Introduction to HPSearch Architecture • A quick view of NaradaBrokering middleware • System Goals • RDAHMM Example • Related and Future Work Community Grids Lab Presentation for CCGrid 2005

  3. HPSearch • A JavaScript based scripting runtime serving as an administration console • Currently uses Rhino (http://www.mozilla.org/rhino) implementation of JavaScript. • Management viewed as • Setting up the distributed application (Involves setting up the broker network, initializing system components) • Querying Run-time System Metadata • For logging purposes • Monitor metadata to help dynamically rewire system to improve performance Community Grids Lab Presentation for CCGrid 2005

  4. HPSearch ArchitectureComponent Summary • Binds URIs to a scripting language • Allows us to manage (manipulate) the Resource identified by the URI • E.g. Read / Write to files, sockets, topics • Read from data base, data from FTP/HTTP resource • This data can then be streamed to distributed application OR data can be read from a stream and processed / stored • “Host-objects” allow us to dynamically access the host system. • Useful for constructing objects that monitor system meta-data, perform management tasks • PerfMetrics gathers system performance data and allows us to query it at run-time. Community Grids Lab Presentation for CCGrid 2005

  5. HPSearch Kernel HPSearch Kernel WSProxy WSProxy WSProxy Service Service Service Request Handler Request Handler JavaScript Shell URIHandler Task Scheduler Flow Handler Broker Network DBHandler Web Service EP WSDLHandler WSProxyHandler Other Objects HPSearchArchitecture Diagram Files Sockets Topics Network Protocol DataBase JDBC Web Service SOAP/HTTP SOAP/HTTP HPSearch Control Events usingPUB/SUB on predefined topic Data buffers sent / received as Narada Events . . . Community Grids Lab Presentation for CCGrid 2005 HPSearch Kernel

  6. NaradaBrokering • NaradaBrokering • Messaging infrastructure for collaboration, peer-to-peer and Grid applications • Implements high-performance protocols (message transit time of 1 to 2 ms per hop) • Order-preserving, optimized message transport with QoS and security profiles for sent and received messages • Support for different underlying protocols such as TCP, UDP, Multicast, RTP • Discovery Service to locate nearest brokers Community Grids Lab Presentation for CCGrid 2005

  7. HPSearch + NaradaBrokeringManaging Streams • HPSearch uses NaradaBrokering to route data streams • Each stream is represented by a topic name • Components subscribe / publish to specified topic • The WSProxy component automatically maps topics to Input / Output streams • Each write (byte[] buffer) and byte[] read() call is mapped to a NaradaBrokering event Community Grids Lab Presentation for CCGrid 2005

  8. System Goals • Investigate the use of HPSearch as a management console to deploy system and application components using scripting • Use HPSearch in scientific / grid applications which can use streaming data • Data filtering is essential in most cases and we do not want to transfer entire data set when only a small percentage of data would be used, • E.g. Choosing data satisfying certain input constraints. • Data re-ordering might be required for formatting input data to match the requirements of executable, • E.g. Converting GML to remove XML elements • Use publish-subscribe methodologies to connect components. Community Grids Lab Presentation for CCGrid 2005

  9. ExampleRDAHMM • GPS time series have modes caused by unknown underlying physical processes • RDAHMM allows us to identify these modes and time periods where physical processes dominated the sequence without any a-priori knowledge of these processes • Help to determine • The actual physical causes • When is the system entering a new mode (perhaps a hint of some important seismic event) Community Grids Lab Presentation for CCGrid 2005

  10. ExampleApplication Components • GPS database (Surface displacement time series collected by SCIGN, http://www.scign.org) • Data filter (filters data, removes unwanted components, reorders data as required by the actual RDAHMM executable) • RDAHMM executable, performs the time series analysis. This data is transferred to the Graph plotting application. • Matlab based Graph plotting application Community Grids Lab Presentation for CCGrid 2005

  11. HPSearch Engine GPS Data Sensor Source Applications Streaming Data Filtering HPSearch Handler Handler Data Filter Filters the input data to get only the estimate and error values Handler Each Handler controls operation of 1 service RDAHMM Analyze the data Matlab Plotting Script Output Graph Community Grids Lab Presentation for CCGrid 2005

  12. Sample OutputClaremont, CA Community Grids Lab Presentation for CCGrid 2005

  13. Related Work • Scripting languages have been very popular and successful for Rapid Application Deployment (RAD) • Sash by IBM, allow RAD and handlers for various tasks such as reading from databases, LDAP registries, invoking Web Services and providing GUI • Jython and Matlab are popular in many scientific communities. E.g. GeoDISE • XCAT project at IU, Extreme Lab uses Jython for scripting to deploy distributed components • Karajan (Ant Like scripting) is used for deploying applications over grid • WSRF::Lite uses perl to implement Web Services Resource Framework and host grid services Community Grids Lab Presentation for CCGrid 2005

  14. Future Work • Investigate system and management scaling with increasing number of components. • Use of security for streams • Negotiation of optimal (high performance) transport • More handlers for different aspects of NaradaBrokering’s management (broker / topic discovery, security, replay etc…) • Investigate interaction with WS Management and WS - Distributed Management based systems Community Grids Lab Presentation for CCGrid 2005

  15. To conclude… • Presented a scripting based architecture for management of data streams and distributed services • Shown how we can use publish subscribe methodologies to connect components that process data in a stream • Create and deploy quick data filtering applications Community Grids Lab Presentation for CCGrid 2005

  16. More Information • HPSearch Project Website http://www.hpsearch.org • NaradaBrokering Project @ IU http://www.naradabrokering.org • CGL Publications http://grids.ucs.indiana.edu/ptliupages/publications/ Community Grids Lab Presentation for CCGrid 2005

  17. Questions / Comments • Any Questions / Comments ? THANKS for attending the presentation Community Grids Lab Presentation for CCGrid 2005

More Related