100 likes | 219 Views
GENI I&M Workshop NetCDF and Local Data Manager (LDM) Mike Zink. November 4, 2010 http://geni.cs.umass.edu/vise http://geni.cs.umass.edu/dicloud http://www.geni.net. Overview. What is NetCDF.
E N D
GENI I&M WorkshopNetCDF and Local Data Manager (LDM)Mike Zink November 4, 2010 http://geni.cs.umass.edu/vise http://geni.cs.umass.edu/dicloud http://www.geni.net
What is NetCDF • NetCDF is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. • Uses HDF5 as data storage layer. • Also provides read-only access to some HDF4, HDF5 archives. • Parallel I/O for high performance computing. • Developed and maintained by Unidata at UCAR
Who uses NetCDF? • NetCDF is widely used in University Earth Science community. • Used for Intergovernmental Panel on Climate Change (IPCC) data sets. • Used by NASA and other large data producers. • Used for NEXRAD and CASA radar data (moments)
NetCDF Example float Reflectivity(Radial, Gate) ; Reflectivity:Units = "dBz" ; float Velocity(Radial, Gate) ; Velocity:Units = "MetersPerSecond" ; float SpectralWidth(Radial, Gate) ; SpectralWidth:Units = "MetersPerSecond" ; float DifferentialReflectivity(Radial, Gate) ; DifferentialReflectivity:Units = "dB" ; float DifferentialPhase(Radial, Gate) ; DifferentialPhase:Units = "Degrees" ; float CrossPolCorrelation(Radial, Gate) ; CrossPolCorrelation:Units = "Unitless" ; float NormalizedCoherentPower(Radial, Gate) ; NormalizedCoherentPower:Units = "Unitless" ; float SpecificPhase(Radial, Gate) ; SpecificPhase:Units = "DegreePerKm" ; float HPropagationPhase(Radial, Gate) ; HPropagationPhase:Units = "Radians" ; float VPropagationPhase(Radial, Gate) ; VPropagationPhase:Units = "Radians" ; int GateFlags(Radial, Gate) ; GateFlags:Units = "BitField" ; float CorrectedReflectivity(Radial, Gate) ; CorrectedReflectivity:Units = "dBZ" ; float CorrectedDifferentialReflectivity(Radial, Gate) ; CorrectedDifferentialReflectivity:Units = "dB" ; 5
OpeNDAP Client for NetCDF • OPenDAP (http://www.opendap.org/) is a widely supported protocol for access to remote data • Defined and maintained by the OPenDAP organization • Designed to serve as intermediate format for accessing a wide variety of data sources. • Client is now built into netCDF C library.
Local Data Manager (LDM) • LDM software acquires data and shares them with other nodes • Data product is treated as a opaque unit, thus nearly any data can be relayed • LDM can handle data from: • National Weather Service "NOAAport channel 3" data streams • NEXRAD radar data • lightning data from the National Lightning Detection Network • GOES satellite imagery
LDM Features • User configurable • execute an arbitrary program with the data product as input • Supports distributed processing • Data ingest functions can be separated from storage and use functions • LDM servers may be configured to relay subsets of data among themselves, distributing the storage and processing around the network. • Extensible • New decoders can be added easily • Event-driven • Handles data as it arrives
Demo Data Flow archived netcdf data aggregated multi-radar data Nowcast images for display • Dynamic end-to-end Nowcasting • Mapping Nowcast Workflows onto GENI “raw” live data Radar Nodes Archival Storage Upstream LDM feed Nowcast Processing Post to Web NetCDF over LDM LDM server NetCDF over LDM LDM server NetCDF over LDM LDM server Storage Nowcasting Merge
Generate “raw” live data ViSE/CASA radar nodes http://stb.ece.uprm.edu/current.jsp Ingest mulit-radar data feeds Merge and grid multi-radar data Generate 1min, 5min, and 10min Nowcasts Send results over NLR to Umass Repeat Use proxy to track usage-based spending on Amazon and enforce quotas and limits http://geni.cs.umass.edu/vise/dicloud.php ViSE views steerable radars as shared, virtualized resources http://geni.cs.umass.edu/vise “raw” live data Nowcast images for display DiCloud Archival Service (S3) LDM Data Feed (EC2) Multi-radar NetCDF Data Nowcast Processing