1 / 67

A flexible and generic web-service for the delivery of geophysical data

EuroSeismic Meeting, Paris, November, 2004. A flexible and generic web-service for the delivery of geophysical data. Experiences from 2 years of Intrepid’s JetStream System in Australia. Presented by Philip McInerney. Topics – Web Data-Delivery. Introduction Example - Geoscience Australia

marnin
Download Presentation

A flexible and generic web-service for the delivery of geophysical data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EuroSeismic Meeting, Paris, November, 2004 A flexible and generic web-servicefor the delivery of geophysical data Experiences from 2 years of Intrepid’s JetStream System in Australia Presented by Philip McInerney

  2. Topics – Web Data-Delivery • Introduction • Example - Geoscience Australia • JetStream Architecture / Design • System built on standard protocols of the web • Design elements: Catalog and Geospatial Intelligence • JetStream Implementation • Consultation, Customisation, Integration • The Australian Experience • Future Vision • Distributed Data Management and Delivery

  3. Web Data-Delivery Example Geoscience Australia’s Geophysical Archive Data Delivery System (GADDS)

  4. GADDS Example Geoscience Australia’s GADDS • GADDS is an implementation of Intrepid JetStream and delivers all of GA’s 50Gb archive of airborne magnetic/radiometric survey data, and the gravity database

  5. GADDS Example Geoscience Australia - GADDS • April, 2003 – web data-delivery pilot • Intrepid’s JetStream system installed and successfully delivers a small sample of survey datasets + continental-scale grids • The study is rapidly expanded with the goal of delivering all of GA’s survey datasets • November, 2003 – GADDS launched • The Geophysical Archive Data Delivery System (GADDS) is formally launched • GA’s 50Gb archive of magnetic and gravity data are freely available across the web (across the globe!) … with minimal administrative overhead

  6. GADDS Example Geoscience Australia - GADDS • June, 2004 – Expansion • Upgraded to deliver 256-channel radio-metric line datasets, and multi-band grids (K, U, Th & Total Count) • November, 2004 – Vic, Qld data added • Datasets from state government surveys were added to pool of data being served • Data from most states will be included by mid-2005

  7. GADDS Example Define Area of Interest

  8. GADDS Example Define Area of Interest

  9. GADDS Example Define Area of Interest

  10. GADDS Example Define Area of Interest

  11. GADDS Example Define Area of Interest

  12. GADDS Example Select DataType

  13. GADDS Example Search Results ...

  14. GADDS Example Metadata Review

  15. GADDS Example Metadata Review

  16. GADDS Example Metadata Review

  17. GADDS Example Metadata Review

  18. GADDS Example Dataset Selection

  19. GADDS Example Dataset Selection

  20. GADDS Example Submit Request

  21. GADDS Example Submit Request

  22. GADDS Example Email: Data Ready … Download

  23. GADDS Example Dataset Download

  24. GADDS Example GADDS Example - Summary • We used a standard browser • We viewed standard web-pages • We made some simple choices • Area of interest • Type of data: Vector or Grid • Theme: Magnetics, Gravity, Radiometrics, … • We chose to download one survey dataset • Selected fields of the dataset • Chose the Datum, Projection, and file format • We received an email when data were ready • We used a standard web ‘download to file’

  25. JetStream Architecture/Design Schematic Diagram Design: Catalog Design: Geospatial Intelligence

  26. HTTP TCP/IP Architecture / Design JetStream Client-Server Architecture Web Server JetStream Server Web Client Apache Tomcat(or equivalent) JetStream Processes JetCat Catalog Explorer Netscape OpenDAP Drivers Binary Datasets Intrepid Processes Data Administrator Firewall

  27. Architecture / Design JetStream Server Architecture Web Server JetStream Server Apache Tomcat(or equivalent; Servlets Container) Client queries Get additional information Queue (Process) Management - Dataset extract & process - Zipping the requested data - Email ‘data-ready’ service Download management JetCat Catalog OpenDAP Drivers Interface to a web-browser Client Binary Datasets - Dataset extract & process - Zipping the requested data - Email ‘data-ready’ service Intrepid Processes Data Administrator (Acquire_Catalog, …) In a distributed system the data and processes would be located on ‘remote’ servers

  28. Architecture / Design Design: Catalog Web Server JetStream Server Apache Tomcat(or equivalent; Servlets Container) Client queries Get additional information Queue (Process) Management - Dataset extract & process - Zipping the requested data - Email ‘data-ready’ service Download management JetCat Catalogue JetCat Catalog OpenDAP Drivers Interface to a web-browser Client Binary Datasets - Dataset extract & process - Zipping the requested data - Email ‘data-ready’ service Intrepid Processes Data Administrator (Acquire_Catalog, …) Data Administrator (Acquire_Catalog, …)

  29. Architecture / Design Design: Catalog • The catalog … • is at the heart of the JetStream System; all interactive client-queries interrogate the catalog rather than the binary datasets • is a very simple data structure … • a flat table; can be ASCII, Access, Oracle table, … • one record per dataset • a small number of essential fields … • Lat/Long limits of the dataset • URL address of the dataset • Data-type and ‘theme’ • additional user-defined fields can contain any other metadata that might be pertinent to the application

  30. Architecture / Design Design: Catalog • The catalog … • must be ‘maintained’; Maintaining the catalog is the essential data-administrative task … in order for the system to ‘know’ about new datasets that have been added to the system • is easily maintained! Because … • it is inherently a very simple data structure! • the automated ‘Acquire_Catalog’ administrative tool ‘harvests’ metadata from the data-files themselves … and refreshes the current catalog • an intelligent ‘Data Manager’ tool facilitates additional manual maintenance of the catalog

  31. Architecture / Design Design: Geospatial Intelligence Web Server JetStream Server Apache Tomcat(or equivalent; Servlets Container) Client queries Get additional information Queue (Process) Management - Dataset extract & process - Zipping the requested data - Email ‘data-ready’ service Download management JetCat Catalogue OpenDAP Drivers Interface to a web-browser Client Binary Datasets - Dataset extract & process - Zipping the requested data - Email ‘data-ready’ service Intrepid Processes Intrepid Processes Data Administrator (Acquire_Catalog, …)

  32. Architecture / Design Design: Geospatial Intelligence • JetStream is geospatially intelligent about a wide variety of industry file types; this means that … • it can interpret such files to determine a dataset’s location and extent • it can ‘look into’ such files … and intelligently extract subsets of the contained data; Subsets may be spatial subsets, or some subset of fields of the dataset • This geospatial intelligence is achieved through ‘Intrepid Processes’ – the functions of the Intrepid Geophysical Data Management and Processing System

  33. Architecture / Design Design: Geospatial Intelligence • Geospatial intelligence is used … • to maintain the catalog … • the ‘Acquire_Catalog’ administrative tool intelligently ‘harvests’ metadata from the data-files themselves • the Data Manager tool also uses intelligent analysis of data-files to assist manual administrative tasks. • to present ‘on-the-fly’ previews of the data to the end-user client (e.g. thumbprint image displays) • to extract subsets of data – either spatial or by selected dataset fields – in order to deliver to the client only that subset of data requested; in effect, reducing download time by excluding data not required by the client

  34. File-type Back-Office Preview Web-based Delivery Vector Intrepid DB Y Y Geosoft GDB Y Y Oracle Y (… and other RDB’s) Y ESRI Shape files Y Y SEG-Y (seismic) Soon Soon Grid Files ERMapper Y Y Geosoft Y Y netCDF Y Y Image Files GeoTiffs Y Y Jpeg (with .jgw) Y Y Tiff (with .tfw) Y Y ECW + Algorithms Soon Soon Architecture / Design Geospatial Intelligence – File Types

  35. Architecture / Design Geospatial Intelligence – File Types • Although JetStream can treat many file-types intelligently – and extract subsets of data from such files - it is also possible to configure the system to deliver any file • simply add a file to the catalog … assigning the essential fields of ‘dataset extents’ … and JetStream can deliver that file across the web • Nominate specific file-extensions to be ‘associated’ files; e.g. a “.doc” or “.pdf” might be ‘report’ files associated with a survey dataset; the files can be ‘associated’ by using the same base-file-name; the data file, and it’s associated report file, will be web-delivered together

  36. JetStream Implementation Consultation, Customisation, Integration

  37. Implementation ‘Off-the-Shelf’ Solution ? Yes, but … • JetStream is an ‘off-the-shelf’ web data-delivery solution … which must be integrated into a corporation’s business • In our experience, implementation requires … • Consultation • Integration • with existing data-management systems • into existing web-interface systems • Customisation of the system • JetStream is flexible, with many options, • Balance this with the value of keeping web-pages simple!

  38. Implementation Integration with Legacy Systems • Consultation needs to identify opportunities of using existing data management systems • JetStream can use sources of systematic metadata in various ways … • JetStream’s Catalog can simply be an existing database table … in Access, Oracle, ASCII, … • an existing database table can be used to initially populate a Catalog • if a legacy system is maintained … then the Catalog can be regularly refreshed from that system’s database tables

  39. Implementation Integration into Existing Web-Pages • Again – consultation should explore opportunities to build upon any existing investment in an organisation’s web-systems • For example, PIRSA had developed an Arc-IMS site for map-composition and map-based querying of spatial databases; JetStream was integrated with that system within days of delivery

  40. Implementation Integration – PIRSA Example Existing Arc-IMS investment in web-pages JetStream is added with a single TAB panel

  41. Implementation Customisation – Corporate Image • Despite obvious differences – the GA and PIRSA sites have almost identical JetStream functionality • JetStream (an ‘off-the-shelf’ solution) can be implemented behind a customised web-page environment … customised to corporate needs … to maintain a corporate image, or integrate with an existing system, etc.

  42. Implementation Customisation – Client Service • Despite being a ‘off-the-shelf’ solution, JetStream is very flexible – and there is considerable scope to tailor the web-page interface to clients needs … • the GA interface, for example, provides access to metadata to assist the client’s selection • the PIRSA site provides much less metadata • For a seismic data service, one would want to see seismic line locations at the time of defining an ‘area of interest’ …

  43. Implementation Customisation – Client Service Display seismic line location to assist definition of ‘area of interest’

  44. Implementation Customisation – Client Service Preview SEG-Y image … then request to download the SEG-Y data file

  45. Australian Experience Client Perspective Data Provider Perspective

  46. Australian Experience GADDS – Customer Reaction • Well received by the customer base • Clients have expressed satisfaction with the access via universally available web-browsers • The simple series of web-pages provide sufficient metadata to facilitate effective dataset selection • Clients have found the dataset delivery mechanism – notification by email, with a URL link – is effective and practical

  47. Australian Experience GADDS – Benefits for GA • Assists GA in their charter to ‘make data freely available’ … • Use of the universally available ‘web-browser’, and delivery via standard web-download protocols – ensures ease-of-access for all. (No proprietary software needed by the client). • Clients construct their own queries to find ‘what data are available ?’

  48. Australian Experience GADDS – Benefits for GA • Reduced data administration overheads • The JetStream system facilitates the dataset management, with features to assist the maintenance of the system catalogue • Significant reduction in clerical staff. Tasks, such as answering client queries, taking orders, extracting datasets from archives, arranging delivery … are now automated

  49. Australian Experience PIRSA Experience • JetStream has improved our efficiencies • More time is available to add value to the products available • More and more stakeholders access our potential field data via JetStream. • Data are also being better managed centrally via JetStream. Domenic Calandro, Manager, Geoscience Datasets

  50. Australian Experience PIRSA Experience – Client View • JetStream is being very well received by our stakeholders, particularly international users, and users with broadband internet access • JetStream definitely contributed to our "number 1 status in the world" for delivery of pre-competitive data (Report of the independent Fraser Institute) Fraser Institute Survey: 100% of respondents considered the South Australian geoscience databases to encourage exploration investment Domenic Calandro, Manager, Geoscience Datasets

More Related