Integrating Geographical Information Systems and Grid Applications

1. Integrating Geographical Information Systems and Grid Applications Marlon Pierce Contributions: Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawardhan Gadgil Community Grids Lab Indiana University

2. Acknowledgements • The real work was done by (in alphabetical order). • Mehmet Aktas • Galip Aydin • Harshawardhan Gadgil • Ahmet Sayar • Project web site: • http;//www.crisisgrid.org • This work was supported by NASA AIST as part of “SERVOGrid: Complexity Computational Environment”

3. Geographical Information Systems and Grid Applications • Pattern Informatics • Earthquake forecasting code developed by Prof. John Rundle (UC Davis) and collaborators. • Uses seismic archives. • Regularized Dynamic Annealing Hidden Markov Method (RDAHMM) • Time series analysis code by Dr. Robert Granat (JPL). • Can be applied to GPS and seismic archives. • Can be applied to real-time data. • Interdependent Energy Infrastructure Simulation System (IEISS) • GeoFEST • Finite element method code developed by Dr. Jay Parker (JPL) and Prof. Greg Lyzenga (JPL/Harvey Mudd College) • Uses fault models as input. • Virtual California • Prof. Rundle’s UC-Davis group • Used for forecasting • Uses fault and fault friction input

4. GIS Data Grid Work at CGL • We decided that the Data Grid components of SERVO is best implemented using standard GIS services. • Use Open Geospatial Consortium standards • Provide downloadable GIS software to the community as a side effect of SERVO research. • We implemented two cornerstone standards as Web Services (WS-I+ approach) • Web Feature Service (WFS): data service for storing abstract map features • Supports queries • Faults, GPS, seismic records • Web Map Service (WMS): generate interactive maps from WFS’s and other WMS’s. • Can be used to set up problems by extracting features (faults, seismic events, etc) from user GUIs to drive problems such as the PI code and (in near future) GeoFEST, VC. • We also built a GIS compatible UDDI and WS-Context • Browse capabilities files. • We are currently working on these steps • Improving WFS performance • Integrating WMS with video streaming technologies. • Implementing Sensor Web Enablement for streaming, real-time data.

5. GIS and Sensor Grids • OGC has defined a suite of data structures and services to support Geographical Information Systems and Sensors • GML Geography Markup language defines specification of geo-referenced data • SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors • Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information • Grid workflow links services that are designed to support streaming input and output messages • We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid

6. A Screen Shot From the WMS Client

7. WMS uses WFS that uses data sources <gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2 </segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineStringsrsName="null"> <gml:coordinates> -118.72,34.243 -118.591,34.176 </gml:coordinates> </gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>

8. Automating Pattern Informatics

9. Pattern Informatics (PI) • PI is a technique developed at University of California, Davis for analyzing earthquake seismic records to forecast regions with high future seismic activity. • They have correctly forecasted the locations of 15 of last 16 earthquakes with magnitude > 5.0 in California. • See Tiampo, K. F., Rundle, J. B., McGinnis, S. A., & Klein, W. Pattern dynamics and forecast methods in seismically active regions. Pure Ap. Geophys. 159, 2429-2467 (2002). • http://citebase.eprints.org/cgi-bin/fulltext?format=application/pdf&identifier=oai%3AarXiv.org%3Acond-mat%2F0102032 • PI is being applied other regions of the world, and John has gotten a lot of press. • Google “John Rundle UC Davis Pattern Informatics”

10. Pattern Informatics in a Grid Environment • PI in a Grid environment: • Hotspot forecasts are made using publicly available seismic records. • Southern California Earthquake Data Center • Advanced National Seismic System (ANSS) catalogs • Code location is unimportant, can be a service through remote execution • Results need to be stored, shared, modified • Grid/Web Services can provide these capabilities • Problems: • How do we provide programming interfaces (not just user interfaces) to the above catalogs? • How do we connect remote data sources directly to the PI code. • How do we automate this for the entire planet? • Solutions: • Use GIS services to provide the input data, plot the output data • Web Feature Service for data archives • Web Map Service for generating maps • Use HPSearch tool to tie together and manage the distributed data sources and code.

12. Example of Data Mining and GIS Grid Data Mining Grid Databases with NASA, USGS features SERVOGrid Faults NASA WMS WFS3 WFS1 WFS2 WMS handling Client requests UDDI SOAP HTTP WMS Client WMS Client

13. Web Map Client WSDL Aggregating WMS Stubs Stubs HTTP SOAP WSDL WSDL “REST” WFS + Seismic Rec. WFS + State Bounds … WMS + OnEarth

14. WMS uses WFS that uses data sources <gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2 </segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineStringsrsName="null"> <gml:coordinates> -118.72,34.243 -118.591,34.176 </gml:coordinates> </gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>

15. GIS Behind the Scenes • The web features are served up by a Web Feature Service. • Web Map Service aggregates maps • NASA OnEarth + our own renderings. • We re-implement Open Geospatial Consortium standards using Web Service Standards. • SOAP messages, WSDL service definitions. • Will allow us to separate messages from HTTP transport layer in future. • More WMS Info: • http://grids.ucs.indiana.edu/ptliupages/publications/acm-gis-sayar.pdf. • http://grids.ucs.indiana.edu/ptliupages/publications/Geoinformatics05_asayar.pdf. • More WFS Info: • http://grids.ucs.indiana.edu/ptliupages/publications/gwpap243.pdf • More general info, software, demos: http://www.crisisgrid.org

16. Tying It All Together: HPSearch • HPSearch is an engine for orchestrating distributed Web Service interactions • It uses an event system and supports both file transfers and data streams. • Legacy name • HPSearch flows can be scripted with JavaScript • HPSearch engine binds the flow to a particular set of remote services and executes the script. • HPSearch engines are Web Services, can be distributed interoperate for load balancing. • Boss/Worker model • ProxyWebService: a wrapper class that adds notification and streaming support to a Web Service. • More info: http://www.hpsearch.org

17. Filter PI Data Mining Filter WS-Context WFS3 GIS Grid Databases with NASA,USGS features SERVOGrid Faults Data Mining Grid from Grid of Grids WFS4 SOAP Pipeline UDDI HPSearch“Workflow” Traditional Execution Grid NaradaBrokering System Services

18. HPSearch (TRex) HPSearch (Danube) Actual Data flow HPSearch controls the Web services Final Output pulled by the WMS HPSearch Engines communicate using NB Messaging infrastructure Data can be stored and retrieved from the 3rd part repository (Context Service) WS Context (Tambora) WFS (Gridfarm001) NaradaBroker network: Used by HPSearch engines as well as for data transfer WMS Data Filter (Danube) Virtual Data flow WMS submits script execution request (URI of script, parameters) HPSearch hosts an AXIS service for remote deployment of scripts • PI Code Runner • (Danube) • Accumulate Data • Run PI Code • Create Graph • Convert RAW -> GML GML (Danube)

19. IEISS GUI FOR OVERLAYING OUTAGE AREA ON A MAP

20. IEISS Summary • IEISS simulates power outages resulting from damage to electrical and natural gas grids. • GIS Grid integration is similar to earlier PI application. • Primary differences: • Better support for dynamic GIS service discovery. • Better integration of distributed state monitoring (WS-Context). • Google map clients as well as modified PI clients.

21. WFS and WMS publish their WSDL URL to the UDDI Registry 1-2-3 - WMS Client -> WMS Server -> UDDI -> WFS 4-5 - WFS publishes the results as GML FeatureCollection document into a topic (“/NISAC/WFS”) in a pub/sub based messaging system. WFS -> WMS Server (creates a map overlay) and IEISS receive this GML document. WMS Server -> WMS Client (displays it) 6 - User invokes IEISS through WMS Client interface for the obtained geospatial features, and WMS Client starts a workflow session in the Context Service.

22. 7 - On receiving invocation message, IEISS updates the shared state data to be “IEISS_IS_IN_PROGRES”. IEISS runs and produces an ESRI Shape file and then invokes shp2gml tool to convert produced Shape file to GML format. After the conversion IEISS updates shared session state to be “IEISS_COMPLETED”. As the state changes, the Context Service notifies all interested workflow entities such as WMS Client.

23. 9-10 - WFS-L publishes the IEISS output as a GML FeatureCollection document to NB topic ‘NISAC/WFS-L’. WMS Server is subscribed to this topic and receives the GML file then converts it to map overlay,and the Client displays the new model on the map. 8 – On receiving the notification, WMS Client makes a request to the WFS-L for the IEISS output

24. Electric Power and Natural Gas data Zoom-in Zoom-out FeatureInfo mode Measure distance mode Clear Distance Drag and Drop mode Refresh to initial map

25. Overlaid Outage Area - I • Basic Steps: • Select Energy Power AND Natural Gas Data and Update Layer List rendered on the map • Click on “Overlay Outage” button • See the outage area on the map

26. Overlaid Outage Area - II • Basic Steps: • Select Energy Power Data and Update Layer List rendered on the map • Click on “Overlay Outage” button • Use zoom-in mapping tool below to get same outage area in more detail • See the outage area on the map

27. Overlaid Outage Area - III • Basic Steps: • Select Energy Power and Natural Gas Data and Update Layer List rendered on the map • Select St. Petersburg from the “Area of Interest” dropdown list. • Click on “Overlay Outage” button. • See the outage area on the map

28. Getting Info about specific EP Data by clicking on the map • Basic Steps: • Select Energy Power Data and Update Layer List rendered on the map • Select (i) from the mapping tools below. • Click on any feature data on the map. • See the information for selected feature in pop-up window

29. Google Hybrid Map and Feature Information call to WMS Natural Gas Layer Electric Power Layer

30. Google Map Client Archived Real Time Databases withSERVOGrid Faults Sensor Grid Google Central HTTP WFS2 WFS1 Google Map Client Helper Services SOAP DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards! UDDI

31. Real Time GPS and Google Maps Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients. Select and zoom to GPS station location, click icons for more information.

32. Integrating Archived Web Feature Services and Google Maps Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.

33. Google Maps as Service accessed from our WMS Client

34. Support for Real Time Applications

35. RDAHMM: GPS Time Series SegmentationSlide Courtesy of Robert Granat, JPL GPS displacement (3D) length two years.Divided automatically by HMM into 7 classes. • Complex data with subtle signals is difficult for humans to analyze, leading to gaps in analysis • HMM segmentation provides an automatic way to focus attention on the most interesting parts of the time series • Features: • Dip due to aquifer drainage (days 120-250) • Hector Mine earthquake (day 626) • Noisy period at end of time series

36. Towards Real-Time RDAHMM • A real-time version of RDHAMM could potentially be used to detect state change events in live data from a GPS station. • SCIGN maintains 125+ GPS stations, so trivially parallel RDAHHM clones can monitor state changes in the entire network. • HPSearch can help • But first we must get the data to RDAHMM.

37. NaradaBrokering: Message Transport for Distributed Services • NB is a distributed messaging software system. • http://www.naradabrokering.org • NB system virtualizes transport links between components. • Supports TCP/IP, parallel TCP/IP, UDP, SSL. • See e.g. http://grids.ucs.indiana.edu/ptliupages/publications/AllHands2005NB-Paper.pdf for trans-Atlantic parallel tcp/ip timings.

38. Traditional NaradaBrokering Features

39. Mean transit delay for message samples in NaradaBrokering: Different communication hops 9 hop-2 hop-3 8 hop-5 7 hop-7 6 5 Transit Delay (Milliseconds) 4 3 2 1 0 100 1000 Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux Message Payload Size (Bytes)

41. Typical use of Grid Messaging Filter or Datamining Sensor Grid Post afterProcessing Post beforeProcessing Web Feature Service NaradaBrokering Notify WFS (GIS data) Database Archives Subscribe HPSearch Manages GIS Grid WS-Context Stores dynamic data GeographicalInformation System

42. Raw to GML via NaradaBrokering • The Scripps Orbit and Permanent Array Center (SOPAC) GPS station network data published in RYO format is converted to ASCII and GML

43. Typical use of Grid Messaging in NASA Sensor Grid GIS Grid Grid Eventing Datamining Grid

44. GIS and Collaboration Tools

45. GIS and Collaboration • The previous slide illustrates an initial interface for capturing, annotating, and storing/replaying video streams. • Still images can be captured and annotated on shared white board. • Annotations are stored along with rest of system.

46. Challenges for Geographical Information System Grids • Must address performance issues. • Related workshop at GGF 15. • HTTP is not an adequate transport mechanism for moving data around. • XML representations, compression, etc. • Well established techniques from real-time collaboration can be applied to sensors • Stream archiving and playback, session management, software multicasting. • Applies to both data streams (GPS) and maps (streaming video).