1 / 32

Grid-Enabled Geospatial Systems

OGF-24. Grid-Enabled Geospatial Systems. Craig Lee lee@aero.org . Mark Reichardt mreichardt@opengeospatial.org. Geospatial Data: Immense Applicability. Estimates vary, but ~80-90% of all data collected or produced by the human race is geospatially referenced.

yazid
Download Presentation

Grid-Enabled Geospatial Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OGF-24 Grid-Enabled Geospatial Systems Craig Lee lee@aero.org Mark Reichardt mreichardt@opengeospatial.org

  2. Geospatial Data: Immense Applicability Estimates vary, but ~80-90% of all data collected or produced by the human race is geospatially referenced Image: E. Gennai, C. Terbough, ESRI • Natural exploration, e.g., oil & gas • Public administration • Civic planning and engineering • Weather and aviation • Environmental planning • Disaster management • Satellite ground systems 2

  3. An Example: GEO Grid Global Earth Observation Grid S. Sekiguchi, AIST 3

  4. The OGC-OGF Collaboration • Promote best practices and international standardization for distributed geospatial data processing capabilities that is: • Transparent -- users is not aware of the infrastructure • Interoperable -- the resources work together • Scalable -- small local, to massive distributed platforms Directly Relevent to Satellite Ground Systems 4

  5. Two Issues • What geospatial and computing technologies must be integrated? • How do we manage the process to make progress on doing so? 5

  6. Some Key OGC Standards • Web Map Service (WMS) • Display of registered and superimposed map-like views of information that come simultaneously from multiple remote and heterogeneous sources. • Web Feature Service (WFS) • Retrieval and update of digital representations of real-world objects referenced to the Earth’s surface. • Web Coverage Service (WCS) • Access to a grid coverage, usually encoded in a binary format, and offered by a server. • Catalogue Service for the Web (CSW) • Common interfaces to publish, discover, browse, and query metadata about data, services, and other resources 6

  7. Key OGC Standards • Sensor Observation Service (SOS) • Access observations for a sensor or sensor constellation, whether in-situ or dynamic (e.g, water monitoring or satellite imaging). Optionally access associated sensor and platform data • Sensor Planning Service (SPS) • An interface to task sensors or models. Using SPS, sensors can be reprogrammed or reconfigured, sensor missions can be started or changed, simulation models executed and controlled. • Sensor Alert Service (SAS) • Defines an interface for publishing and subscribing to alerts from sensors. If an event occurs the SAS will notify all clients subscribed to this event type. • Observations and Measurements Encoding Standard (O&M) • Defines an abstract model and an XML schema encoding for observations and it provides support for common sampling strategies. It also provides a general framework for systems that deal in technical measurements in science and engineering. 7

  8. Key OGC Standards • Transducer Markup Language (TML) • TML defines the conceptual model and XML Schema for describing transducers and supporting real-time streaming of data to and from sensor systems • Web Processing Service (WPS) • WPS defines a standardized interface to facilitate the publishing of geospatial processes, and the discovery of an binding to those processes by clients 8

  9. OGC Sensor Web Enablement Framework (SWE) Registered sensor and observation metadata Metadata for a sensor and observations, and a URL Metadata for a sensor observations, and a URL Metadata for a sensor observations, and a URL Metadata for a sensor observations, and a URL Metadata for a sensor observations, and a URL ……… CSW SPS Mission Control System  WNS • SWE Architecture Encodings: • Observations & Measurements • SensorML • Transducer Markup Language (TML) SOS IEE1451  SAS ! 9

  10. OGC Web Services Combined 10

  11. Key OGF Standards • HPC-Basic Profile • Cluster scheduling interoperability layer supported by Microsoft, Altair Computing, Platform Computing, and others • Simple API for Grid Applications (SAGA) • Complete distributed programming environment with C++ and Java bindings • GridRPC • Grid-enabled Remote Procedure Call • Data Access and Integration specs (WS-DAI-*) • Set of specs for accessing remote files, databases, XML documents • Web Services Resource Framework (WSRF) • Allow remote data and services to be independently managed 11

  12. An Example of a Global Grid: EGEE Bob Jones, EGEE Project Director 12

  13. Coalescing Service Categories to Specification Areas • Specification Areas • Catalogues & Registries • Discovery • Applications • Execution Services • Workflow/Transactions • Data Management • State Management • Messaging, Routing, Addressing • Event Notification • Metadata Schemas & Ontologies • Portals and User Interfaces • Security • Policy & Agreement • System Architecture • System Management • Infrastructure Service Categories • Catalogues & Registries • Discovery • Applications • Job Submission/Execution Svcs • Workflow Management • Data Product Management • Storage & Archiving • Messaging • Event Notification • User Management • Information Assurance/Security • Monitor & Control • Enterprise Svc Mgmt/Software Mgmt • APIs/Ops Systems • Financial Accounting 13

  14. Populating the Specification Areas Updated from Fox, Ho, Pierce – U. Indiana Green: On DISR Baseline 07-3.0 14

  15. Populating the Specification Areas Updated from Fox, Ho, Pierce – U. Indiana Green: On DISR Baseline 07-3.0 15

  16. How Do We Manage Progress for Distributed Geospatial Systems? • Targeted Projects on Decisive Issues • Build Critical Mass of Key Stakeholders • Continual polling and coordination across the community • They must agree on: • Clear Goals • Clear Schedule (“time-box” the process) • Clear Responsibilities • Properly Provisioning the Effort 16

  17. A General Process Model Task A Concept Development Task B RFQ/CFP* Development Task C Selection & Kick-off Task E Task D Deploy & Persist Develop & Test *RFQ/CFP = Request for Quotation/Call for Participation Clear Schedule, Deliverables and Project Responsibilities 17 17

  18. Return on Investment • What are the “carrots” to build the critical mass of stakeholders? • Get early influence in specification development, early skills building, visibility, and opportunity for early market deployment of standards, but just as important… • Return on Investment (ROI) • Investment • Time, Money & People • Both Monetary and In-Kind (labor & materials) • Timely Connection to Concrete Results • Stakeholders benefit from collaboration • Get more than they put in 18

  19. A Case Study: HPC Basic Profile • 2006 • Key stakeholders decided to demonstrate interoperability between their existing job submission infrastructures • November 2006 (SC06) • Prototype implementations demonstrated • 28-August-2007 • HPC Basic Profile, Version 1.0, published • http://www.ogf.org/documents/GFD.114.pdf • November 2007 (SC07) • Interoperability demonstrated by Altair, Microsoft, Platform, OMII-UK, OMII-Europe, EGEE, UVa • 21-February-2008 • Interoperability Experiences with HPCBP, Version 1.0, published http://www.ogf.org/documents/GFD.124.pdf • Commercial adoption plans by Altair, Microsoft & Platform 19

  20. OGC Web Service Testbed • OWS is an annual process where sponsors identify specific demonstration targets • Larger number of participants supply in-kind resources • OWS-5 recently finished (March 28, 2008) • OWS-5 Sponsors • BAE Systems - National Security Solutions • Federal Geographic Data Committee (FGDC/USGS) • Google • Lockheed Martin Integrated Systems & Solutions (Lead Org.) • Northrop Grumman • US National Aeronautic and Space Administration (NASA) • US National Geospatial-Intelligence Agency (NGA) • ~35 Participating Organizations • 3x-4x Return on Investment • Strength of Collaboration 20

  21. OWS-5 • Five Threads • Sensor Web Enablement (SWE) • Geo Processing Workflow (GPW) • Information Communities and Semantics (ICS) • Agile Geography • Compliance Testing (CITE) • Outbrief Videos • NASA Sensor Web 2.0 Experiments (7:43) • Data Architecture Views (10:37) • Conflation Processing (5:07) • Web Coverage Processing Service (5:58) 21

  22. Still from OWS-5 NASA Sensor Web Video http://www.opengeospatial.org/pub/www/ows5/index.html Satellite EO-1 tasked to collect imagery on Northern San Diego County Wildfires that was integrated with UAV track data (red lines) in Google Earth. Demo drove issues around sensor networks, data interoperability, and command & control. 22

  23. OWS-6: RFQ/CFP Topics • Sensor Web Enablement (SWE) • Georeferenced sensors, event notification, security • Aviation Information • Aeronautical Information Exchange Model (AIXM) for next generation flight control system (FAA and EuroControl) • GeoProcessing Workflows (GPW) • Web Processing Service (WPS), Workflow and Grids (“WPS to Grid”) • Distributed resource management, e.g., data, services, hosts, networks, for geospatial applications • Decision Support Services • 3D Visualization with Fly Through • Open Location Services • Integrated Client for multiple OWS services • Compliance, Interoperability & Test Evaluation (CITE) • Suite of tests and reference implementations for all approved standards 23

  24. Possible Geospatial/Grid Topics • Integration, interoperability of geospatial and computing resource • Information models (CIM/GLUE) • Distributed and federated catalogs (ebRIM, MDS, etc.) • Discovery services • Implementation of WPS on various grid tools • SAGA, GridRPC, HPC-Basic Profile • Integration of WS-Eventing, WS-Notification, INFOD • Sensor Web Enablement • Integration of configuration and lifecycle management tools • Configuration Description, Deployment and Lifecycle Management (CDDLM) • Integration of WPS with workflow management tools • BPEL, Kepler, Taverna, Triana, Pegasus, etc. • Workflow design tools, execution engines, planning (data virtualization), data provenance • Integration of WPS with grid security models • SAML, XACML, VOMS, GSI • Support for virtual organizations • Integration of geospatial data with rule-based, data management policy engines • iRODS 24

  25. Digital Repositories/Libraries • Digital Repositories/Libraries represent a growing part of the “infomass” • Federated catalogues and federated storage • OGF-Europe Workshop at OGF-23 • 4 sessions, ~70 participants from industry, academia and European Commission • Proposed OGF-Europe collaboration with Global Research Library 2020 • Proposed OGF-Europe involvement in GENESI-DR Digital Repositories Seminar in Autumn 2008, Frascati • OGF-Europe PC membership for the DReSNet special DR session at IEEE e-Science conference, Dec 2008 • Eventual development of an OGF Working Group dedicated to issues surrounding DRs 25

  26. Bill St. Arnaud Another Example: Green Grids Managing Energy Policy • Extensive use of virtualization and fiber optics to put the computing where the power is • Enable the workload to “follow the wind” or “follow the sun” Source: www.Bastionhost.com 26

  27. Bill St. Arnaud Another Example: Green Grids Managing Energy Policy • Extensive use of virtualization and fiber optics to put the computing where the power is • Enable the workload to “follow the wind” or “follow the sun” Source: www.Bastionhost.com 27

  28. Bill St. Arnaud Another Example: Green Grids Managing Energy Policy • Extensive use of virtualization and fiber optics to put the computing where the power is • Enable the workload to “follow the wind” or “follow the sun” • A geospatial sensor network Source: www.Bastionhost.com 28

  29. Summary • Significant work already underway for standard, distributed geospatial computing infrastructures • Highly relevant to many application domains • Opportunity to leverage the work of key participants through collaboration • How to engage application groups and industries needing distributed geospatial processing? 29

  30. Back-Ups andOther Draft Material

  31. Data Collection Data Collection reduce reduce reduce map map map map map map map map ….. A “Cloudy” Solution Space Vendors of high-level application configuration tools are potential “cloud framework providers” OS Virtualization Amazon S3/EC2 RightScale, GigaSpaces, Elastra, 3Tera Cohesive Hadoop over EC2 Sun’s Caroline Hadoop MS Astoria Mesh MapReduce, GFS, BigTable Google AppEngine Parallel Frameworks Software as a Service Courtesy of Dennis Gannon, www.extreme.indiana.edu/~gannon/science-data-center.pdf 31

  32. A Host of Cloud Issues • Data access and interoperability • Will have to be approached at the application domain level, by the domain users involved • Security • Can cloud providers provide sufficient security for sensitive applications? • Reliability • Can cloud providers provide sufficient reliability for critical applications? • Frameworks • How to manage sets of VMs and VOs? • These are essentially “cloud frameworks” and call for "cloud framework providers" • Performance management • Clouds tend to abstract away location • Managing the compute-data locality (affinity) can be important • Costing models • How to compare your own infrastructure costs with a cloud computer? • How to compare two clouds? 32

More Related