1 / 16

Numbriliste 4D andmete esmane töötlemine

Numbriliste 4D andmete esmane töötlemine. 4D võrguandmed (gridded data) Formaadid: “uurija” formaat GRIB NetCDF. Okeanograafiliste andmete iseloom üldjuhul juhuslikud väljad ψ (x,y,z,t) erijuhud: 1D aegrida ψ (x 0 ,y 0 ,z 0 , t ) 1D sügavusprofiil ψ (x 0 ,y 0 , z ,t 0 )

espitia
Download Presentation

Numbriliste 4D andmete esmane töötlemine

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Numbriliste 4D andmete esmane töötlemine 4D võrguandmed (gridded data) Formaadid: “uurija” formaat GRIB NetCDF

  2. Okeanograafiliste andmete iseloom üldjuhul juhuslikud väljad ψ(x,y,z,t) erijuhud: 1D aegrida ψ(x0 ,y0 ,z0 ,t) 1D sügavusprofiil ψ(x0 ,y0 ,z,t0 ) 1D horisontaalprofiil ψ(x,y0 ,z0 ,t0 ) ψ(x0 ,y,z0 ,t0 ) 2D horisontaalne kaart ψ(x,y,z0 ,t0 ) 2D aeg-sügavuslõige ψ(x0 ,y0 ,z,t) 2D ruumiline lõige ψ(x,y0 ,z,t0 ) ψ(x0 ,y,z,t0 ) Esitus: funktsiooni graafik Esitus: kaart

  3. Võrguandmed (gridded data) x, y, z, t diskreetsete sammudega Erivariandid x, y: tasapinnaline x,y (sh pööratud) Lat, Lon (sh pööratud) ortogonaalne kõverjooneline (Lat, Lon on selle erijuht) Erivariandid z: sügavused/kihid (sageli muutuva kihi paksusega) sigma-koordinaat (jälgib sügavusi võrdsete vahedega) isopükniline (jälgib liikuvaid samatiheduspindasid) hübriidne (eelnevate kombinatsioon) vt ka http://www.msi.ttu.ee/~elken/SOL_07.ppt http://www.msi.ttu.ee/~elken/OceanLim_Notes15.pdf

  4. “uurija” formaat Programmide näited: Uurija kirjutab ise näiteks FORTRANis talle sobivas formaadis ja loeb hiljem samal kujul sisse Kirjutamisel ja lugemisel tuleb üldjuhul kasutada täpselt samasugust andmete kirjeldust Tavaliselt puudub info andmete sisu ja aja kohta (selle saab panna küll faili nimesse) ning füüsilise/geograafilise ruumi kohta Jälgi kas fail on BIG või LITTLE ENDIAN!!! (windows on little-endian) DIMENSION T(200,200,10) !…..arvutused OPEN(20,’andmed.dat’, FORM=‘BINARY’) WRITE(20) T CLOSE(20) END DIMENSION T(200,200,10) OPEN(20,’andmed.dat’, FORM=‘BINARY’) READ(20) T CLOSE(20) !…..arvutused END

  5. GRIB (GRIdded Binary) GRIB (GRIdded Binary) is a mathematically concise data format commonly used in meteorology to store historical and forecasted weather data. It is standardized by the World Meteorological Organization’s Commission for Basic Systems, known under number GRIB FM 92-IX, described in WMO Manual on Codes No.306. Currently there are two versions of GRIB, first edition (current sub-version is 2) is used operationaly world-wide by all meteorological centers, for Numerical Weather Prediction output (NWP). A newer generation was introduced, known as GRIB second edition, but it is used only by few centers and in many cases not for operational broadcast. Some of the second-generation GRIB are used for derived product distributed in Eumetcast of Meteosat Second Generation. GRIB superseded the Aeronautical Data Format (ADF). wikipedia

  6. 2 kirjega GRIB fail PDS GDS BMS BDS PDS GDS BMS BDS ‘GRIB’ ‘7777’ ‘7777’ ‘GRIB’ 1. kirje 2. kirje GRIB Formaat GRIB Fail koosneb erinevatest kirjetest, mis kõik on isekirjelduvad. Iga kirje (nt temperatuuri, veetaseme jne väli) koosneb sektsioonidest. GRIB 1 Section 0: Indicator Section.GRIB kirje alguse tähis ja pikkus Section 1: Product Definition Section (PDS).Standardsed definitsioonid kodeerituna (võrk, parameeter, aeg, väljaandja jne) Section 2: Grid Description Section (GDS) - Optional.Mittestandardsed definitsioonid. Kasutatakse juhul kui andmed ei ole PDS tabelitega kirjeldatavad Section 3: Bit Map Section (BMS) - Optional.Kasutatakse nt maa punktide märkimiseks so märgitakse ära millistes võrgusõlmedes on olemas andmedSection 4: Binary Data Section (BDS).Andmete pikkus ja andmedSection 5: '7777' - ASCII Characters indicating end of GRID record. Märgib kirje lõppu

  7. Koodid PDS-is

  8. GRIB formaadi iseärasusi + • Isekirjeldav, äärmiselt kokku pakitud kiire andmeülekande jaoks. • Väga paindlik. • Operatiivses okeanograafias ja meteoroloogias üldlevinud standard. - • 4D esituse asemel domineerib 2D, kus ühel ajahetkel on 3D väljad kirjutatud 2D väljade vahelduvaks jadaks. • Orienteeritud meteoandmetele. Mereprognooside puhul tehakse muudatused (üledefineerimised), mis lihtsate programmide juures ei toimi. Ka keerukama softi puhul esinevad konverteerimisel sageli tõrked. • Saadaval WINDOWS tarkvara (nt OCENS GRIB Explorer) mereprognooside faile ei ava. • Kaasaegsete tehnoloogiate (nt openDAP) tugi on nõrk

  9. GRIB Software Only decoders/encoders • WGRIB and WGRIB2 • Convsh • CDO • Picogrib • NCEP codes • JGrib • Meteosatlib • PyNIO • degrib • GRIB API • GRIB Java Decoder • GDAL • PyGrib Viewers • GrADS • Xconv • NCAR Command Language (linux) • Ugrib • SmartMet • OpenCPN • Panoply

  10. Okeanograafias kasutatav GRIB tarkvara • WGRIB – Command line based program to manipulate, inventory and decode GRIB files • GrADS – Linux/Unix based desktop application that handles GRIB • Xconv/Convsh – Xconv is a graphical tool for displaying and converting gridded data, and is available for most operating systems. Convsh is the command-line equivalent. • CDO (Climate Data Operators) is an analysis tool for geoscientific data with GRIB support • Panoply netCDF, HDF and GRIB Data Viewer. With Panoply 4 you can do a lot of data manipulations and graphical presentations http://www.giss.nasa.gov/tools/panoply/ Töötab hästi erinevate Läänemere andmete jaoks

  11. NetCDF (Network Common Data Form) NetCDF (Network Common Data Form)is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. The project homepage is hosted by the Unidata program at the University Corporation for Atmospheric Research (UCAR). The format is an open standard. NetCDF Classic and 64-bit Offset Format are an international standard of the Open Geospatial Consortium. The project is actively supported by UCAR. The recently released (2008) version 4.0 greatly enhances the data model by allowing the use of the HDF5 data file format. Version 4.1 (2010) adds support for C and Fortran client access to specified subsets of remote data via OPeNDAP. The format was originally based on the conceptual model of the NASA CDF but has since diverged and is not compatible with it. wikipedia The freely available distribution contains the C/C++/F77/F90 libraries, (plus tools ncgen/ncdump) and can be obtained as source code as a gzipped tar file or compressed tar file, or see our list of precompiled binaries on the netCDF 3.6.2 downloads page. http://www.unidata.ucar.edu/software/netcdf/

  12. NetCDF formaadi iseärasusi + • orienteeritud 4D andmetele • võimalikud suured failid üle 2 GB • palju “lehitsejaid” • OpenDAP tugi • suhteliselt lihtne siduda programmeerimiskeeltega (FORTRAN, C, MATLAB, PYTHON jne) • HDF5 tugi - • Üldlevinud (NetCDF-3) andmete maht (väga) palju suurem kui GRIBil • NetCDF-4/HDF5 andmete maht mõistlik kuid veel ebastabiilne

  13. NetCDF faili päis netcdf dataset-hbmV2PHYS-bal-latestforecast_1352145534043 { dimensions: time = 3 ; lat = 70 ; lon = 107 ; depth = 21 ; variables: int time(time) ; time:standard_name = "time" ; time:long_name = "time" ; time:units = "hours since 2012-11-05 12:00:00 UTC" ; time:axis = "t" ; time:_CoordinateAxisType = "Time" ; time:valid_min = 25 ; time:valid_max = 27 ; float lon(lon) ; lon:standard_name = "longitude" ; lon:long_name = "longitude" ; lon:units = "degrees_east" ; lon:unit_long = "Degrees East" ; lon:axis = "x" ; lon:_CoordinateAxisType = "Lon" ; lon:valid_min = 21.4583f ; lon:valid_max = 30.2917f ; float temp(time, depth, lat, lon) ; temp:_CoordinateAxes = "time depth lat lon" ; temp:standard_name = "sea_water_temperature" ; temp:long_name = "potential temperature" ; temp:units = "degree_Celsius" ; temp:unit_long = "degree Celsius" ; temp:valid_range = -3.00000f, 40.0000f ; temp:_FillValue = -999.000f ; temp:missing_value = -999.000f ; // global attributes: :title = "MyOcean Baltic MFC: Physical forecast" :institution = "Baltic MFC“ :references = "http://ocean.dmi.dk" :source = "HBM_V2 model output converted to NetCDF" :Conventions = "CF-1.0" :time_min = 25.0000 ; :time_max = 27.0000 ; :julian_day_unit = "hours since 2012-11-05 12:00:00 UTC" :z_min = 0.000000 ; :z_max = 100.000 ; :latitude_min = 56.5250 ; :latitude_max = 59.9750 ; :longitude_min = 21.4583 ; :longitude_max = 30.2917 ; } mõõtmed muutujad üldinfo

  14. OpenDAP – the cloud framework for scientific data OPeNDAP is a framework that simplifies all aspects of scientific data networking. OPeNDAP provides software which makes local data accessible to remote locations regardless of local storage format. OPeNDAP also provides tools for transforming existing applications into OPeNDAP clients (i.e., enabling them to remotely access OPeNDAP served data). OPeNDAP software is freely available. http://www.opendap.org/ The THREDDS Data Server (TDS) is a web server that provides metadata and data access for scientific datasets, using OPeNDAP, OGC WMS and WCS, HTTP, and other remote data access protocols. Test openDAP interface at http://test.opendap.org/dap/data/nc/

  15. ODV – the interactive exploration of scientific data Ocean Data View (ODV) is a software package for the interactive exploration, analysis and visualization of oceanographic and other geo-referenced profile, time-series, trajectory or sequence data. The ODV data format allows dense storage and very fast data access. Large data collections with millions of stations can easily be maintained and explored on  inexpensive desktop and notebook computers. Data from ARGO, GTSPP, CCHDO,  World Ocean Database, World Ocean Atlas, World Ocean Circulation Experiment (WOCE), SeaDataNet, and Medar/Medatlas can be directly imported into ODV. ODV also supports the netCDF format and lets you explore and visualize CF, COARDS, GDT and CDC compliant netCDF datasets. This works with netCDF files on your local machine as well as with remote netCDF files served by an OPeNDAP server.  http://odv.awi.de/en/home/

  16. Vabalt allalaaditavaid Läänemere andmeid http://www.smhi.se/en/services/open-data/model-data-hiromb-bs01-1.33361 http://en.ilmatieteenlaitos.fi/open-data-manual-forecast-models http://www.bsh.de/aktdat/modell/stroemungen/stroemungspaketee.htm http://emis.msi.ttu.ee/

More Related