JetStream OPeNDAP Geoscience Data Web Delivery

1. JetStream OPeNDAP Geoscience Data Web Delivery

2. Topics • JetStream Overview • Tour of Some Existing sites • Data Management Issues • Downloads, compression, format, … • System Administration / Back Office • Thick Client support • Summary – JetStream Advantages

3. Web Data Delivery Two key components … • Data Provider - with data to serve • Client – using standard web-pages and standard file downloading protocols Goal: Easy access by all / Reliable data transfer

4. Web pages User Input Downloads JetStream – the Intelligent Data Server JetStream Server HTTP Browser Provider’s Data User’s Software (and metadata)

5. Client Side 1: Standard Web Browser

6. Client Side 2: Simple Inputs

7. Client Side 3. Standard File Download!

8. JetStream Summary 1. Prepare Data OPeNDAP Cat Jet- Stream Server HTTP Browser

9. Web pages User Input JetStream Summary 2. Serve WebPages / Client Access OPeNDAP Cat Jet- Stream Server HTTP Browser

10. JetStream Summary 3. Process Request OPeNDAP Cat Jet- Stream Server HTTP Browser

11. Downloads JetStream Summary 4. Deliver Data OPeNDAP Cat Jet- Stream Server HTTP Browser

12. JetStream Summary 5. Analyse OPeNDAP Cat Jet- Stream Server HTTP Browser

13. Client Query based on Metadata • A typical implementation of JetStream’s web-page interface would allow a Client to request datasets which meet criteria … • Spatial extents (Lat/Long) • Primary theme • Depending on the theme, a second level of criteria … e.g. Date>1990, Spacing<300

14. Some Australian JetStream sites • The SARIG Site is hosted on a MS/Windows environment and integrated with their arcIMS application • The GA JetStream Site is hosted on a dual Solaris system and has been integrated with their MapServer based web application – based on ROSA applette

15. The SARIG site

16. Use sophistication of an existing interface

17. Grid data • report metadata • a preview bitmap • choose download?

18. JETSTREAM at GeoSceince Australia

19. The GA site

20. The GA site (Cont)

21. The GA site (Cont)

22. The GA site (Cont)

23. The GA site (Cont)

24. Current Sites (Cont)

25. JetStream - Data Management Issues (1) • JetStream can be integrated with any existing system of data management • Requires a catalogue … which can be generated (and refreshed) by … • Either gathering the relevant metadata from the dataset’s metadata files • Or (with customisation) by drawing on the metadata contained within existing Departmental databases

26. JetStream - Data Management Issues (1a) Metadata stored with the data …

27. JetStream - Data Management Issues (1b) Metadata stored in a database …

28. GDADS

29. Located Data Management • JetStream’s management of ‘located data issues’ occurs on two levels … • Firstly, dataset extents are fields in the catalogue, so the initial search-for-data simply interrogates the catalogue to find datasets located within a defined are of search • Then either whole-datasets or subsets-of- datasets can be delivered. The data-serving organisation may make this choice … or allow their clients the ability to make that choice.

30. System Administration Back Office • Customisable catalogue • Configurable data-delivery – add licence files, disclaimer files, associated report files, etc. • Job queue management … • Priorities • Abort • Report/log – record of what was downloaded • Automatic metadata creation and collection

31. System Administration cont. • Provides User with Time estimates • Reporting of total zip file size • Site Administrator Logging of requests • Site Administrator Queue Management • Uses Tomcat and Apache as well as JSP to create and manage individual sessions.

32. Downloads (1) • JetStream uses the industry-standard file download protocols of the web … • Choice of HTTP or FTP protocols configurable in JetStream • Simply click on a URL … and initiate a routine save-file-to-disk download! • For a more complex request … receive an emailed URL link when data are ready

33. Downloads (2) …Don’t wait … receive an email when ready

34. Downloads (3) …… then a standard file download

35. Downloads • The client … • Can be on any web-enabled computer system (Windows, Linux, UNIX, Mac, …) • Minimal client needs … • any browser • an un-zipper … WinZip, pkunzip, gzip, jar • Download, unzip … and use your preferred spatial analysis software (GIS, Profile Analyst, etc.)

36. Compression Ratios • Choose the subset you need … do not download the entire dataset • Raster: Spatial subset • Vector: Selected fields • WinZip … industry standard • Raster (ERMapper) up to 50% • Vector (ASCII) … up to 94%

37. Export File Formats (1) • Current implementations … • Raster: ERMapper binary file format • Vector: ASCII in ASEG GDF2 format • Potential implementations … • Rapidly implement any of the formats that are currently available in Intrepid … • Working on SWATH & SEGY

38. Export File Formats (2) • Vector … • ASCII ASEG GDF2 format • Binary: Intrepid and Geosoft *.gdb • ASCII: XYZ, CSV • GIS: Arc Shape and MapInfo TAB / MIF • Others … ECS, Geosolutions

39. Export File Formats (3) • Raster (Grid) … • ERMapper binary format • Binary: Geosoft *.grd, NetCDF • ASCII: GXF, Image & Image_XYZ • GIS: ESRI format bil-file, MapInfo GeoTiff • Others … AGSO, ECS, Geopak, GIPSI, LCT, Geosolutions, ZMap

40. Projection Conversion • Two main options … • Compute-on-the-fly by the server data-extraction process – extract (subset), projection conversion, reformat to requested format, zip and place in the ‘to-be-delivered’ area. • Pre-computed • Additional (X, Y) fields pre-computed in vector datasets • A choice of 2 or 3 prepared grid file options

41. Thick Client Support • openDAP compliance allows the possibility of direct access to the Jetstream Catalog • openDAP is an Oceanographic sponsored data standard • The JetStream Engine can support direct and immediate delivery of data into an application.

42. Thick Client Support (Cont) • For example, the Intrepid Project Manager can browse the JetStream catalogs directly • Add facility for a base gif so that the catalog can be viewed visually as well as a table • An Intrepid Data Extraction wizard guides the user through the same kind of dialog as the existing web browser based JetStream sites

43. Thick Client Support (Cont) • The Data Extract can “register” its capabilities so that it can do some of the processing. • Data formatting and Coordinate System conversions for example • JetStream Site administrators decide whether or not this type of service is to be supported for their site.

44. Thick Client Support (Cont)

45. Thick Client Support (Cont)

46. JetStream Advantages (1) • Your clients see your web-interface … which will typically be highly integrated with other elements of your web-interface to the world … … and thus you retain control of the corporate image that your organisation projects to your client-base.

47. JetStream Advantages (2) • System independent … • Server-side: JetStream is currently implemented on Windows 2000, Linux and Sun Solaris servers • Client-side: Your clients can log in to your site from any computer with a web-browser … ensuring the widest possible public access

48. JetStream Advantages (3) • No proprietary software … • Client access does not require proprietary software. Public client access to datasets is achieved by anyone with a web-browser and an un-zipper … … after which the client can proceed to use datasets within their preferred spatial analysis package (GIS, imaging, processing, etc.)

49. JetStream Advantages (4) • Based on Open Standards … … and proven industry standards • JetStream uses an OPeNDAP compatible catalogue • Can serve OPeNDAP compatible datasets • Can serve data from a range of industry-standard file formats (Intrepid, Geosoft) • Can serve data directly from an organisations’ corporate relational databases (Oracle, etc.)

50. JetStream Advantages (5) • Based on Open Standards … … and proven industry standards • Can deliver data in a wide range of industry-standard file formats … • Uses the universally available zip for compression • Allows a wide range of Datum/Projection options