1 / 31

NARR User Issues

Wesley Ebisuzaki CPC/NCEP Wesley.Ebisuzaki@noaa.gov http:wwwt.emc.ncep.noaa.gov/mmb/rreanl/. NARR User Issues. Topics. Files: contents, availability, formats Tools: BUFR, GRIB Obtaining the data: NCEP, NCAR, NCDC Scripting downloads

mab
Download Presentation

NARR User Issues

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Wesley Ebisuzaki CPC/NCEP Wesley.Ebisuzaki@noaa.gov http:wwwt.emc.ncep.noaa.gov/mmb/rreanl/ NARR User Issues

  2. Topics • Files: contents, availability, formats • Tools: BUFR, GRIB • Obtaining the data: NCEP, NCAR, NCDC • Scripting downloads • What is reality? Is reanalysis truth? An insider's viewpoint.

  3. GRIB Files • For grids (horizontal fields) such as 500 mb Z • Lambert conformal grid: 32 km eta grid (#221) • Have grids of lat/lon, display programs • Model grid: not available except at NCEP • No plans to make available, display programs? • Analyses are on-line and forecasts ? • 99% of people will use the GRIB files

  4. BUFR • For all input observations except SST, snow, ice, observed precipitation (GRIB) • Input obs BUFR files have QC marks • Analyses/forecasts at station locations (MOLTS) • Much smaller than GRIB files, limited locations • Distribution (informal) • Includes restricted data • Few users (experience from global reanalysis) • Often other sources of data (in easier formats)

  5. Misc Files • Restart files: huge, full precision, NCEP-only • Statistics: • Fit of obs to NARR, fit of obs to R2 • Small, can be made available on-line • Observation counts: • Small, should be put on-line • PNG (monitoring plots) • Small, should be put on-line

  6. NARR has zero funding. Want more files?Don't expect too much.

  7. Despair? • Global Reanalysis has been running this way for years! • CPC has been running global reanalysis for its own operations. CPC and the outside community have shared interest in maintaining a stable assimilation system. NCEP has an interest in others using its products. • However, limited resources, need to be selective, work with partners

  8. Status • Real-time is running (4 days late) • Started building 1.5-2 year on-line rotating archive of latest analyses at NCEP • 1979-2003 merged files at NCAR, NCDC • NCDC has 5 TB merged files on-line, trying to support the on-line community by NOMADS • EMC people are working on other projects

  9. Tools: BUFR • Situation improving • GrADS can read bufr directly

  10. Tools: GRIB • GrADS: display and computations • grib2ctl.pl: make GrADS control files • wgrib: inventory, decode GRIB files, remove records from a file (save space) • lcgrib: regional subset (save space) • copygb: change grid (save space or neurons) • grbpoint.pl: grib -> point value printout (slow) • Gempak (?), NCL (?)

  11. Example: GrADS • Make control file (analyses) • grib2ctl.pl -rr cpc_rr_mon.197901 >my.ctl • grib2ctl.pl -rr cpc_rr_mon.%y4%m2 >my.ctl • Make index file (analyses) • gribmap -0 -i my.ctl • Run GrADS • grads • open my.ctl

  12. Example: wgrib • Use: inventories, make files smaller, decode data for fortran and C programs • See contents of a grib file • wgrib (options) cpc_mon_197901 • To extract records, decode records, see wgrib home page.

  13. Example: lcgrib • Regional subset of Lambert conformal grid • *** MAKE SMALLER FILES *** • lcgrib in-grib out-grib west south east north • West, south, east and north are in degrees • N and E are +ve, S and W are -ve

  14. Obtaining Data: NCEP • Rotating archive: latest 1.5 – 2 years of analyses • http://nomad7.ncep.noaa.gov/ncep_data • Other files: TBD • NCEP not set up to deal with the public requests • Go through the web • Go through NCDC, NCAR

  15. Obtaining Data: NCAR • Have 1979-2003 merged data • Sure they will get more of the data • Plans, availability?

  16. Obtaining Data: NCDC • NCDC has the merged data 1979-2003 and want 2004 ASAP • NCDC will use NOMADS to distribute the data to the on-line community. • NOMADS is project of many institutions including NCDC and NCEP for distributing data over the internet. • NOMADS-NCDC is taking software from NCEP and COLA to distribute GRIB data.

  17. Data Access at NCDC • Custom plots • ftp4u: access by selected time, field, region • http access to raw data (like FTP) • GDS (GrADS Data Server): OpeNDAP server (formerly DODS), on-line access to data

  18. Difficulties in on-line distribution • Downloading a 5 TB data set is difficult • Need ways to minimize data transfer • Plots (maps) work but limited utility • GDS needs to be optimized for large datasets • ftp4u is a solution

  19. FTP4U • Only download data that you need • Select levels • Select variables • Select geographic regions • Select times

  20. Making FTP4U Work • Don't request more data than needed • Make one big request, not many small requests of the same file; i.e., make one request to get the winds, temp and heights rather than 3 requests. Do not request long time series. • Shared resource (1 job at a time) • Be prepared to script/automate the downloads • May be impossible to download in a few hours, may require days, weeks and multiple jobs/requests

  21. Scripting FTP4U Downloads • FTP4U is normally accessed by a browser • Anything a browser can do, a script can also do • Experience at NCEP is that most plot/ftp4u access is scripted (automated) • NCDC's modifications to the NCEP provided programs make it more difficult to script the download but nothing that a cURL user couldn't handle. NCDC wants to change!

  22. Scripting Downloads cont'd • Use cURL, a unix/windows program, to “emulate” a browser • Each browser page becomes a call to cURL. • With a little RTFM was able get script working • sample program will be put on NARR home page

  23. Scripting Downloads cont'd • Downside: steep learning curve, may have no local experts for help • Upside: can make large data transfers possible, can be used in other projects (ex. getting fcst data automatically, skewing on-line opinion polls) • Given that we get more data off the web, it's a skill that someone in a large group should eventually have.

  24. Obtaining Data: NCDC • It's a learning situation for NCDC • Give feedback (complain), it's the only way to improve the situation. Lessons learned here will help the next time you try getting a different dataset from NCDC's nomad server. • Current issues: full disk (new RAID is will be installed soon), obtaining time series, scripting support

  25. What is Reality? • Some people consider reanalysis to be truth • NARR is new, bound to be warts, already found some.

  26. What is Reality? • Accuracy of reanalysis depends on many factors • Data going into the analyses • Types, counts, quality of observations • Accuracy of previous analysis • Accuracy of atmospheric/land model • Spatial and temporal filtering • NO EASY ANSWER

  27. What is Reality? cont'd • Accuracy • check to the appropriate data counts and biases • Need to add our counts to web • Need to add fit to data to web • check other reanalyses • Fit to observations • Consider sensitivity to modeling assumptions • Sensitivity experiments (data, resolution, model), only available to limited # of people

  28. “Trust but verify” Ronald Reagan What is Reality? cont'd

  29. URLs • copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb • change grid type of grib files • copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb • change grid type of grib files • cURL: http://curl.haxx.se • download URLs, handles post data • GEMPAK: http://my.unidata.ucar.edu/content/software/gempak/index.html • graphics, can handle NARR files with some effort • GrADS: http://grads.iges.org/grads/ • graphics, computations

  30. URLs • grbpoint.pl: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/wgrib.scripts • simple program to extract data from lat/lon • grib2ctl.pl: http://www.cpc.ncep.noaa.gov/products/wesley/grib2ctl.html • makes control files for GrADS • g95: http://www.g95.org/ • f95 compiler for copygb • HIPHOP: http://iacweb.etzh.ch/stall/dominik/hihop/ • grib support for idl • lcgrib: http://www.cpc.ncep.noaa.gov/products/wesley/gribw.html • regional subset of Lambert conformal grib files

  31. URLs • NARR: http://wwwt.emc.ncep.noaa.gov/mmb/rreanl/index.html • home page for NARR • NARR rotating archive: http://nomad7.ncep.noaa.gov/ncep_data • NCAR Command Language (NCL): http://ngwww.ucar.edu/ • graphics, computations package that supports grib • NCDC data: http://nomads.ncdc.noaa.gov/#narr_datasets • on-line supplier of NARR data • wgrib: http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html • inventory, slice & dice grib files

More Related