1 / 26

Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and Repositories

Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and Repositories. eResearch Australasia 2011, ½ Day Morning Workshop, Thursday 10 th November 2011. GENERAL INFORMATION. This is a half-day workshop (9am to 12:30pm) 9:00am Introductions and Participants Goals

kostya
Download Presentation

Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and Repositories

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and Repositories eResearch Australasia 2011, ½ Day Morning Workshop, Thursday 10th November 2011

  2. GENERAL INFORMATION • This is a half-day workshop (9am to 12:30pm) • 9:00am Introductions and Participants Goals • 9:15am Session 1: Discovering OPeNDAP data access services • 10:00am Session 2: Applicable use cases of OPeNDAP data services • 10:30am Tea Break for 15 minutes • 11:00am Session 3: OPeNDAP service protocols and features • 11:45am Session 4: Accessing complementary features and services • 12:30pm End of Workshop 10:00am Session 2: Applicable use cases of OPeNDAP data services

  3. Session 2 • Applicable use cases of OPeNDAP data services for data cataloging and data access using a variety of applications and tools. • A short tutorial exploring data access using an OPeNDAP-enabled tool within a scripting language such as python. • 45 minutes in length + 15 minute Tea Break at 10:30am

  4. Spectrum of Use Cases

  5. Workshop Use-Cases

  6. Use Case limitations • Time to access data is dependent on the following factors: • Hardware and network performance • Selection of variables and dimensions • Number of data requests to be issued • Latency inherent in the data request • Number of concurrent accesses to the server

  7. Performance limitations to data delivery *Time to transfer a 500 MB data object

  8. Performance limitations to data delivery *Time to transfer a data object on an ADSL2 modem = 14 Mbps

  9. DAP-enabled client tools/applications • OPeNDAP Clients (partial list) • http://opendap.org/whatClients • To be demo’ed today. • Web browser returning ASCII data • Pydap - is a pure Python library implementation of the DAP2 • NetCDF - is a set of software libraries and self-describing, machine-independent data formats with interfaces to Python, FORTRAN, C/C++, and Java languages • NCO – comprises a dozen standalone, command-line programs that take netCDF files as input • MATLAB – session 3 • Panoply – session 4

  10. Web Browser demo • “.ascii” tells the OPeNDAP service to return the data in ASCII format. • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon • Try accessing multiple variables such as latitude • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon,lat • What other variables are available in the file? • Try accessing “sst” and download to ascii

  11. Tutorial - Subsetting • Modify the variable indice (C array syntax 0..n-1 ), and request ascii data in the web browser • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:1:50] • What happens if the middle index number is changed from “1” to “2”? • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:2:50]

  12. Tutorial – subsetting continued • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:1439] • Add a new variable to the above URL, separated by a comma, and make a request for ascii data in the web browser: • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:50],lat[0:1:30] • Now do the same thing in the form and modify the indice range • watch out for large indice ranges returning large amounts of data

  13. Tutorial: .dods response • Try the binary response “.dods” • “.dods” tells the OPeNDAP service to return the data in binary format • http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.dods?lon • This is two part binary DAP data object which contains 1) meta data, and 2) binary data structure. • This is the typical response for OPeNDAP enabled client applications.

  14. Pydap • Pydap is a pure Python library implementing the Data Access Protocol, also known as DODS or OPeNDAP. You can use Pydap as a client or server. • http://pydap.org/ • To install Pydap on Windows… see the next slide for Windows • To install Pydap on Mac OS X… see the slide for Mac OS X • To install Pydap on Linux… see the slide for Mac OS X

  15. Pydap installation for Windows • To install Pydap on Windows … • Install python onto Windows • Install easy_install: ez_setup.py • Install Pydap: easy_install Pydap

  16. Pydap installation for Mac OS X • To install Pydap on Mac OS X… • Python is install on Mac OS 10.5 and 10.6 by default • Install easy_install: ez_setup.py • Install Pydap: easy_install Pydap

  17. Test Pydap client installation • >>> frompydap.clientimport open_url • >>> dataset = open_url('http://test.opendap.org/dap/data/nc/coads_climatology.nc') • >>> var = dataset['SST'] • >>> var.shape • (12, 90, 180) • >>> var.type • <class 'pydap.model.Float32'> • >>> print var[0,10:14,10:14] # this will download data from the server • <class 'pydap.model.GridType'> • with data • [[ -1.26285708e+00 -9.99999979e+33 -9.99999979e+33 -9.99999979e+33] • [ -7.69166648e-01 -7.79999971e-01 -6.75454497e-01 -5.95714271e-01] • [ 1.28333330e-01 -5.00000156e-02 -6.36363626e-02 -1.41666666e-01] • [ 6.38000011e-01 8.95384610e-01 7.21666634e-01 8.10000002e-01]] • and axes • 366.0 • [-69. -67. -65. -63.] • [ 41. 43. 45. 47.]

  18. More Pydap client features • See Pydap client: http://pydap.org/client.html

  19. NetCDF API and Tools • NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. • http://www.unidata.ucar.edu/software/netcdf/ • To install, go to • http://www.unidata.ucar.edu/downloads/netcdf/index.jsp • To use with python, build netCDF4 and its python module, or … • easy_install netCDF4

  20. NetCDF demo • >>> import netCDF4 • >>> url = 'http://test.opendap.org/dap/data/nc/coads_climatology.nc’ • >>> dataset = netCDF4.Dataset(url) • >>> var = dataset.variables['SST'] • >>> var.shape • (12, 90, 180) • >>> print var[0,10:14,10:14] # this will download data from the server • <class 'pydap.model.GridType'> • with data • [[-1.26285707951 -- -- --] • [-0.769166648388 -0.77999997139 -0.675454497337 -0.595714271069] • [0.128333330154 -0.0500000156462 -0.0636363625526 -0.141666665673] • [0.638000011444 0.895384609699 0.721666634083 0.810000002384]] • >>> print var • <type 'netCDF4.Variable'> • float32 SST('TIME', 'COADSY', 'COADSX') • …

  21. NetCDF demo • Get metadata information about the following data set: • ncdump -h http://opendap.bom.gov.au:8080/thredds/dodsC/nmoc/oceanmaps2_ofam_fc/latest/ocean_fc_20111108_000_surface.nc

  22. NCO Tools • The netCDF Operators (NCO) comprise a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids manipulation and analysis of gridded scientific data. • http://nco.sourceforge.net/ • To install NCO tools, go to • http://nco.sourceforge.net/#Binaries

  23. NCO tool demo • Download the initial conditions for regional ocean model • ncks -O -F -d xt_ocean,648,979 -d yt_ocean,467,798 http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_analysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc • View the original file metadata: • ncdump –h http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_analysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc • View the subsetted file’s metadata • ncdump –h ocean_temp_2010_03_12.nc

  24. NCO demo • Download the initial conditions for regional ocean model using longitude and latitudes ranges for the dimensions • ncks -O -F -d xt_ocean,143.55,176.66 -d yt_ocean,-28.35,4.75 http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_analysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp2_2010_03_12.nc • Are the files the same (dimensions and lon/lat range)? • ncdump –v xt_ocean ocean_temp_2010_03_12.nc • ncdump –v xt_ocean ocean_temp2_2010_03_12.nc

  25. Tutorial: Pick a demo to try • Please select from pydap, NetCDF, and NCO demos • Install the software on your machine • Run a test case and see if the software is installed correctly • Access a different file from a TDS or Hyrax data service • Get the metadata information • Get the coordinate axes data • Get a subset of data from an array

  26. Thank you • Authors: • Tim F. Pugh1, James Gallagher2, Dave Fulker3 • 1Australian Bureau of Meteorology, Melbourne, Australia, t.pugh@bom.gov.au2 OPeNDAP, Butte, Montana, USA, jgallagher@opendap.org • 3 OPeNDAP, Boulder, Colorado, USA, dfulker@opendap.org

More Related