1 / 12

UWG 2013 Meeting Publish-Subscribe ( Datacasting )

UWG 2013 Meeting Publish-Subscribe ( Datacasting ). Goal. Goal PO.DAAC to publish availability of our data Why? So users can subscribe and no longer have to crawl FTP. High Level Functions: Publishing. Data Providers. Information Providers. Consumers. Information In. Data In.

cale
Download Presentation

UWG 2013 Meeting Publish-Subscribe ( Datacasting )

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UWG 2013 MeetingPublish-Subscribe (Datacasting)

  2. Goal • Goal • PO.DAAC to publish availability of our data • Why? • So users can subscribe and no longer have to crawl FTP

  3. High Level Functions: Publishing Data Providers Information Providers Consumers Information In Data In Information Out Data Out High-Level Access Tools Ingest Web Portal Visualization Web Services & Publishing Direct Data Access Inventory Archive

  4. Solution • Datacasting • Extension of RSS • Work funded by NASA ACCESS 2005, 2009 • Data providers publishes availability of new data granules via XML feed • Data consumers subscribes to feed to learn about recently available data granules

  5. By the way… • Datacasting enables • Down select items in feed based on multiple criteria • Temporal • Spatial • Other metadata • Automatically download selected data files • Mash up across different types of feeds • Combine newspaper articles, scientific reports, event analyses, data, etc. in one place • And much more

  6. Publishing DatacastingRSS Data Providers Information Providers Consumers Information In Data In Information Out Data Out DatacastingRSS High-Level Access Tools Ingest Web Portal Visualization Web Services & Publishing Direct Data Access Inventory Archive

  7. Publish-Subscribe Sequence

  8. DatacastingRSS <rssxmlns:datacasting="http://datacasting.jpl.nasa.gov/datacasting"xmlns:georss="http://www.georss.org/georss"xmlns:gml="http://www.opengis.net/gml" version="2.0"><channel><title>MetOp-A ASCAT Level 2 25.0 km Ocean Surface Wind Vectors</title><link>http://podaac-test.jpl.nasa.gov/dataset/ASCAT-L2-25km</link><description>This dataset contains operational near-real-time Level 2 ocean surface wind vector retrievals...</description><pubDate>Wed, 27 Feb 2013 07:49:44 GMT</pubDate> ...<datacasting:channelUID>PODAAC-ASOP2-25X01</datacasting:channelUID><datacasting:dataSource>METOP-A/ASCAT</datacasting:dataSource><item><title>ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc</title><link>http://podaac-test.jpl.nasa.gov/dataset/ASCAT-L2-25km</link><description>ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc</description><enclosure length="847714" type="application/x-netcdf" url="ftp://podaac-ftp.jpl.nasa.gov/allData/ascat/preview/L2/25km/2013/057/ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc.gz"/><pubDate>Wed, 27 Feb 2013 07:49:44 GMT</pubDate> ...<georss:where><gml:Envelope><gml:lowerCorner>-89.37 0.02</gml:lowerCorner><gml:upperCorner>89.26 359.99</gml:upperCorner></gml:Envelope></georss:where><datacasting:acquisitionStartDate>Tue, 26 Feb 2013 19:00:01 GMT</datacasting:acquisitionStartDate><datacasting:acquisitionEndDate>Tue, 26 Feb 2013 20:41:59 GMT</datacasting:acquisitionEndDate></item> ...<item> ...</item></channel></rss>

  9. Link to DatacastingRSS Feed

  10. Client Tool: Scripts • Scripts in Python, Perl, MATLAB, etc. can be written to download data granules as they are made available • Sample MATLAB script for downloading data granules: xml = xmlread('http://podaac-test.jpl.nasa.gov/ws/metadata/granule/?format=datacasting&datasetId=PODAAC-ASOP2-25X01'); enclosures = xml.getElementsByTagName('enclosure'); fori = 0:enclosures.getLength-1 url = char(enclosures.item(i).getAttribute('url')); [pathstr, name, ext] = fileparts(url); filename = [name, ext]; if exist(filename, 'file') disp(['File exists: ' filename]); else urlwrite(url,filename); disp(['Downloaded: ' filename]); end end exit;

  11. Client Tool:Data::Downloader % dado configinit --filename ./ascat-l2-25km.conf % dado feeds refresh % dado files download Terminal Perl tool for subscribing to feeds and downloading and maintaining files on disk

  12. Client Tool:Datacasting Feed Reader Subscribe to DatacastingRSS Filter feed items Create mash up Download data

More Related