1 / 13

7 +/- 2 Maybe Good Ideas

7 +/- 2 Maybe Good Ideas. John Caron June 2011. (1) . NetCDF -Java (aka CDM) has lots of functionality, but only available in Java NcML Aggregation Access to lots of other file formats Feature types ( eg collections of point data)

kimball
Download Presentation

7 +/- 2 Maybe Good Ideas

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 7 +/- 2 Maybe Good Ideas John Caron June 2011

  2. (1) • NetCDF-Java (aka CDM) has lots of functionality, but only available in Java • NcMLAggregation • Access to lots of other file formats • Feature types (eg collections of point data) • Ironically, some functionality (eg aggregation) already available for remote datasets through opendap • But not for local datasets How can we get the CDM into other languages ? • Replicate in C and maintain two software stacks • Use reverse JNI (call Java from C) • Or …

  3. CdmRemote Server (aka TDS Lite) • Lightweight server for CDM datasets • Zero configuration – use queries to configure • Local filesystem • Cache expensive objects • Allow non-Java applications access to CDM stack • Create virtual datasets: aggregations, logical views • Coordinate space queries • Feature Type subsetting • New API (!)

  4. CdmRemote Server (aka TDS Lite) Python / ? cdmRemote Server C Client cdmRemote CDM Point Feature API CDM Point Feature API Application Coordinate Systems Data Access Data

  5. (2) Ncstream as a netCDF file format • Write-optimized • Append only • Encode the full CDM object model • Uses Google’s protobuf for serialization • Java, C Libraries can read and access through the standard netCDF API • Tools to convert to netcdf-3 and 4 formats

  6. (3) BUFR/GRIB Table registration • Unidata sponsored web service • Registered users can upload BUFR/GRIB tables • Unique id is assigned (MD5 16 byte checksum?) • Convince producers to include the id into the data – unambiguous which table was used • Anyone can download. • GRIB and BUFR Decoding • Using CDM – find bugs ! • Might become (ad-hoc) reference library • Might spur objections from “the experts” • Turn over to WMO if they want it • Survival of Human Race is at stake here

  7. (4)Streaming data / standing queries • The proposal Dennis and I submitted last year • “As soon as it arrives on IDD, send me PrecipTotal from NCEP/ RUC2 model subsetted by lat/lon bounding box in netCDF-4 / CF format” • “As it arrives, send me GTS BUFR data in lat/lon bounding box in CSV”

  8. Push (header) Current IDD data access IDD Data LDM FILE TDS Dataset Dataset Pull requests Dataset Dataset CDM library

  9. Content based filtering(standing requests) IDD Data LDM PIPE • Message Service • Content filtering • Change encoding • Protocol? Push (content) Request Request Request Content Filter Standing request service

  10. (5) Python • Unidata should choose a scripting language to support, and give scientists full access to all of our tools in it • Python wants to be the open-source Matlab • DOE, BADC have bought into Python • Python is a safe choice

  11. (6) NetCDF management tools • Develop consistent set of tools for managing collections of netCDF files • Use existing tools (ncgen, nccopy, ncdump, nco, etc) under the covers • but don’t be constrained by their interfaces • Look at RDBMS management languages • Use a scripting language like Python

  12. (7) Hadoop • Open Source started by Doug Cutting (Lucene) and Yahoo • Based on Google’s Map-Reduce for parallel processing • Lots of industry use, part of new data ecosystems • Objects in distributed, replicated file system • Commodity, shared-nothing hardware nodes • Simple key-value store • Append-only, sequential reading • Scale to arbitrarily large amount of data (batch) • Gather many queries and run them over the data

  13. (8) SciDB • Michael Stonebraker, David DeWitt • “SciDBwill be optimized for data management of big data and for big analytics.  • “The scientists that are participating in our open source project believe that the SciDB database — when completed — will dramatically impact their ability to conduct their experiments faster and more efficiently and further improve the quality of life on our planet by enabling them to run experiments that were previously impossible due to the limitations of existing database systems and infrastructure.” • Getting involved: • Load netcdf/hdf5 into SciDB • “Native mode” – leave data in netcdf/hdf5

More Related