1 / 22

Real-time Streaming Applications in Grids: Enhancing Data Management and Collaboration

Explore the applications of data streaming in Grid systems, from data assimilation to global information grids. Learn about the use of service-oriented architectures for efficient message exchange and the integration of GIS and sensor data through Grid messaging. Discover how standards and messaging protocols optimize quality of service in real-time data processing. Dive into case studies such as NASA Sensor Grid, GIS Grid, and NaradaBrokering in linking services and data streams. Get insights into workflow design for streaming input and output messages using Web services. Find out how Grid technologies elevate collaboration and data management efficiency. Stay updated on current trends and approaches in leveraging Grid systems for real-time and streaming applications.

kempf
Download Presentation

Real-time Streaming Applications in Grids: Enhancing Data Management and Collaboration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grids for Real-time and Streaming Applications GCC2005 Beijing China December 1 2005 Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 http://grids.ucs.indiana.edu/ptliupages/presentations/ gcf@indiana.eduhttp://www.infomall.org

  2. Four Data Streaming Application Areas • Data Assimilation applied to link the data deluge (satellites, sensors, seismometers) in real time to small and large scale parallel simulations • Use in Earthquake Science • Department of Defense (and Homeland Security) have built the Global Information Grid with a target architecture NCOW (Network Centric Operations and warfare) • They submit no jobs; rather stream data to brokers from which they are filtered and distributed • Includes their rather dated distributed simulation HLA • Audio-Video Conferencing implemented with services and Grid messaging • Hand-held Grid linking PDA/cell-phones to Grids

  3. SS Database SS SS SS SS SS SS SS Raw Data  Data  Information  Knowledge  Wisdom AnotherGrid Decisions AnotherGrid SS SS SS SS FS FS OS MD MD FS Portal OS OS FS OS SOAP Messages OS FS FS FS AnotherService FS FS MD MD OS MD OS OS FS Other Service FS FS FS FS MD OS OS OS FS FS FS MD MD FS Filter Service OS AnotherGrid FS MetaData FS FS FS MD Sensor Service SS SS SS SS SS SS SS SS SS SS AnotherService

  4. Key Concepts • Grid of Grids (System of Systems) allows “library” approach to composing Grids • Service Oriented architectures (Web or Grid services) are attractive for many/most distributed systems • There are many applications that are NOT best considered as jobs and files (classic Grid) but rather as streams and filters (services) • Services exchanging messages becomesServices exchanging streams (sets of messages) • Publish-Subscribe messaging gives better QoS and management than point to point messaging with negligible performance loss • Always use standards including those for GIS

  5. Typical use of Grid Messaging in NASA Sensor Grid GIS Grid Grid Eventing Datamining Grid

  6. GIS and Sensor Grids • OGC has defined a suite of data structures and services to support Geographical Information Systems and Sensors • GML Geography Markup language defines specification of geo-referenced data • SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors • Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information • Grid workflow links services that are designed to support streaming input and output messages • We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid

  7. Grid Eventing via NaradaBrokering • The Scripps Orbit and Permanent Array Center (SOPAC) GPS station network data published in RYO format is converted to ASCII and GML

  8. Grid Messaging with Grid of Grids Architecture Filter or Datamining Grid DoD Principles Sensor Grid Post afterProcessing Post beforeProcessing Web Feature Service NaradaBrokering Notify WFS (GIS data) Database Archives Subscribe HPSearch Manages GIS Grid WS-Context Stores dynamic data GeographicalInformation System

  9. Earthquake Faults from the WMS Client

  10. WMS uses WFS that uses data sources <gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2 </segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineStringsrsName="null"> <gml:coordinates> -118.72,34.243 -118.591,34.176 </gml:coordinates> </gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>

  11. Google Map Client Archived Real Time Databases withSERVOGrid Faults Sensor Grid Google Central HTTP WFS2 WFS1 Google Map Client Helper Services SOAP DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards! UDDI

  12. Real Time GPS and Google Maps Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients. Select and zoom to GPS station location, click icons for more information.

  13. Integrating Archived Web Feature Services and Google Maps Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.

  14. Google Maps as Service accessed from our WMS Client

  15. Google plus GIS Grid Integrated with Los Alamos Critical Infrastructure Simulations for DHS Natural Gas Layer Energy Power Layer

  16. NaradaBrokering 2003-2006 • Messaging infrastructure for collaboration, peer-to-peer and Grids Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop) • Order-preserving message transport with QoS and security profiles • Support for different underlying transport such as TCP, UDP, Multicast, RTP • SOAP message supportand WS-Eventing, WS-RM and WS-Reliability. • WS-Notification when specification agreed • Active replay support: Pause and Replay live streams. • Stream Linkage: can link permanently multiple streams – using in annotation of real-time video streams • Replicated storage support for fault tolerance and resiliency to storage failures. • Management: HPSearch Scripting Interface to streams and brokers (uses WS-Management) • Broker Topics and Message Discovery: Locate appropriate • Integration with Axis2 Web Service Container (?) • High Performance Transport supporting SOAP Infoset

  17. Mean transit delay for message samples in NaradaBrokering: Different communication hops 9 hop-2 hop-3 8 hop-5 7 hop-7 6 5 Transit Delay (Milliseconds) 4 3 2 1 0 100 1000 Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux Message Payload Size (Bytes)

  18. Average Video Delays for one broker – Performance scales proportional to number of brokers Multiple sessions One session Latency ms 30 frames/sec # Receivers

  19. Gateway Gateway Gateway Gateway XGSP Media Service WS-Context Collaboration GridServices Audio Mixer NaradaBroker HPSearch Video Mixer UDDI Transcoder NaradaBroker Thumbnail WS-Security Replay NaradaBroker Record Annotate SharedWS SharedDisplay WhiteBoard

  20. GIS TV GlobalMMCS SWT Client Chat Video Mixer Webcam

  21. e-Annotation Archived Stream Annotated e-Annotation Player Player Stream Player Whiteboard Archived stream Annotation / WB e - Annotation e - Annotation player player Whiteboard Player Archived Real Time Real TimeStream List Stream List Player Real time Real time stream Archieved stream list player stream list

  22. Location of software for Grid Projects in Community Grids Laboratory • htpp://www.naradabrokering.org provides Web service (and JMS) compliant distributed publish-subscribe messaging (software overlay network) • htpp://www.globlmmcs.org is a service oriented (Grid) collaboration environment (audio-video conferencing) • http://www.crisisgrid.org is an OGC (open geospatial consortium) Geographical Information System (GIS) compliant GIS and Sensor Grid (with POLIS center) • http://www.opengrids.org has WS-Context, Extended UDDI etc. • The work is still in progress but NaradaBrokering is quite mature • All software is open source and freely available

More Related