290 likes | 305 Views
Grids for Real-time and Streaming Applications. I-Light Symposium IUPUI September 22 2005 Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 http://grids.ucs.indiana.edu/ptliupages/presentations/
E N D
Grids for Real-time and Streaming Applications I-Light Symposium IUPUI September 22 2005 Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 http://grids.ucs.indiana.edu/ptliupages/presentations/ gcf@indiana.eduhttp://www.infomall.org
Four Data Streaming Application Areas • Data Assimilation applied to link the data deluge (satellites, sensors, seismometers) in real time to large scale parallel simulations • Use in Earthquake Science • Department of Defense (and Homeland Security) have built the Global Information Grid with a target architecture NCOW (Network Centric Operations and warfare) • They submit no jobs; rather stream data to brokers from they are filtered and distributed • Includes their rather dated distributed simulation HLA • Audio-Video Conferencing implemented with services and Grid messaging • Hand-held Grid linking PDA/cell-phones to Grids
Key Concepts • Grid of Grids (System of Systems) allows “library” approach to composing Grids • Service Oriented architectures (Web or Grid services) are attractive for many/most distributed systems • There are many applications that are NOT best considered as jobs and files (classic Grid) but rather as streams and filters (services) • Services exchanging messages becomesServices exchanging streams (sets of messages) • Publish-Subscribe messaging gives better QoS and management than point to point messaging with negligible performance loss
Typical use of Grid Messaging in NASA Sensor Grid GIS Grid Grid Eventing Datamining Grid
GIS and Sensor Grids • OGC has defined a suite of data structures and services to support Geographical Information Systems and Sensors • GML Geography Markup language defines specification of geo-referenced data • SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors • Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information • Grid workflow links services that are designed to support streaming input and output messages • We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid
Grid Eventing via NaradaBrokering • The Scripps Orbit and Permanent Array Center (SOPAC) GPS station network data published in RYO format is converted to ASCII and GML
Grid Messaging with Grid of Grids Architecture Filter or Datamining Grid DoD Principles Sensor Grid Post afterProcessing Post beforeProcessing Web Feature Service NaradaBrokering Notify WFS (GIS data) Database Archives Subscribe HPSearch Manages GIS Grid WS-Context Stores dynamic data GeographicalInformation System
WMS uses WFS that uses data sources <gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2 </segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineStringsrsName="null"> <gml:coordinates> -118.72,34.243 -118.591,34.176 </gml:coordinates> </gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>
Google Map Client Archived Real Time Databases withSERVOGrid Faults Sensor Grid Google Central HTTP WFS2 WFS1 Google Map Client Helper Services SOAP DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards! UDDI
Real Time GPS and Google Maps Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients. Select and zoom to GPS station location, click icons for more information.
Integrating Archived Web Feature Services and Google Maps Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.
Google plus GIS Grid Integrated with Los Alamos Critical Infrastructure Simulations for DHS Natural Gas Layer Energy Power Layer
Mean transit delay for message samples in NaradaBrokering: Different communication hops 9 hop-2 hop-3 8 hop-5 7 hop-7 6 5 Transit Delay (Milliseconds) 4 3 2 1 0 100 1000 Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux Message Payload Size (Bytes)
Average Video Delays for one broker – Performance scales proportional to number of brokers Multiple sessions One session Latency ms 30 frames/sec # Receivers
Consequences of Rule of the Millisecond • Useful to remember critical time scales • 1) 0.000001 ms – CPU does a calculation • 2a) 0.001 to 0.01 ms – Parallel Computing MPI latency • 2b) 0.001 to 0.01 ms – Overhead of a Method Call • 3) 1 ms – wake-up a thread or process (do simple things on a PC) • 4) 10 to 1000 ms – Internet delay • 2a), 4) implies geographically distributed metacomputing can’t in general compete with parallel systems • 3) << 4) implies a software overlay network is possible without significant overhead • We need to explain why it adds value of course! • 2b) versus 3) and 4) describes regions where method and message based programming paradigms important
Databases and/or Sensors Data Data Filter Filter Filter Data Filter Data OGC or OGSA-DAIGrid Services AnalysisControl Visualize Grid Data Filter This Type of Grid integrates with Parallel computing Multiple HPC facilities but only use one at a time Many simultaneous data sources and sinks HPC Simulation Grid Data Assimilation Other Gridand Web Services Distributed Filters massage data For simulation SERVOGrid (Complexity) Computing Model
Gateway Gateway Gateway Gateway XGSP Media Service WS-Context Collaboration Grid NaradaBroker Audio Mixer HPSearch Video Mixer UDDI NaradaBroker Transcoder Thumbnail WS-Security Replay NaradaBroker Record Annotate SharedWS SharedDisplay WhiteBoard
GIS TV GlobalMMCS SWT Client Chat Video Mixer Webcam
PDA Download video (using 4-way video mixer service) Desktop PDA
NB Features Released 2005-2006 • Production implementations of WS-Eventing, WS-RM and WS-Reliability. • WS-Notification when specification agreed • SOAP message supportand NaradaBrokers viewed as SOAP Intermediaries • Active replay support: Pause and Replay live streams. • Stream Linkage: can link permanently multiple streams – using in annotating real-time video streams • Replicated storage support for fault tolerance and resiliency to storage failures. • Management: HPSearch Scripting Interface to streams and brokers (uses WS-Management) • Broker Topics and Message Discovery: Locate appropriate • Integration with Axis2 Web Service Container (?) • Support of IBM MQSeries functionality and Legacy MQSeries Systems as a Grid of Grids gateway • Better Security tracking endless changes of WS-Security • High Performance Transport supporting SOAP Infoset
What to Remember • Grids are Services exchanging Messages • Developing messaging paradigm for Grids using Message Oriented Middleware or Software Overlay Network or Grid Service Bus • Web Service container replaces computer • Service replaces process • A stream is an ordered set of messages • NaradaBrokering replaces MPI with different applications and different requirements • Service Internet replaces Internet: messages replace packets • (Sub)Grids replace Libraries in Grid of Grids Architecture • Use W3C OASIS OGC GGF standards
Location of software for Grid Projects in Community Grids Laboratory • htpp://www.naradabrokering.org provides Web service (and JMS) compliant distributed publish-subscribe messaging (software overlay network) • htpp://www.globlmmcs.org is a service oriented (Grid) collaboration environment (audio-video conferencing) • http://www.crisisgrid.org is an OGC (open geospatial consortium) Geographical Information System (GIS) compliant GIS and Sensor Grid (with POLIS center) • http://www.opengrids.org has WS-Context, Extended UDDI etc. • The work is still in progress but NaradaBrokering is quite mature • All software is open source and freely available