Cyberinfrastructure to Support Real-time, End-to-End Local Forecasting

1. Cyberinfrastructure to Support Real-time, End-to-End Local Forecasting Mohan Ramamurthy Tom Baltzer, Doug Lindholm, and Ben Domenico Unidata/UCAR AGU Fall Meeting December 16, 2004

2. Local NWP: A Growing Activity • Applied Modeling Inc. (Vietnam) MM5 • Atmospheric and Environmental Research MM5 • Colorado State University RAMS • Florida Division of Forestry MM5 • Geophysical Institute of Peru MM5 • Hong Kong University of Science and Technology MM5 • IMTA/SMN, Mexico MM5 • India's NCMRWF MM5 • Iowa State University MM5 • Jackson State University MM5 • Korea Meteorological Administration MM5 • Maui High Performance Computing Center MM5 • MESO, Inc. MM5 • Mexico / CCA-UNAM MM5 • NASA/MSFC Global Hydrology and Climate Center, Huntsville, AL MM5 • National Observatory of AthensMM5 • Naval Postgraduate School MM5 • Naval Research Laboratory COAMPS • National Taiwan Normal University MM5 • NOAA Air Resources Laboratory RAMS • NOAA Forecast Systems Laboratory LAPS, MM5, RAMS • NCAR/MMM MM5 • North Carolina State University MASS • Environmental Modeling Center of MCNC MM5 MM5 • NSSL MM5 • NWS-BGM MM5 • NWS-BUF (COMET) MM5 • NWS-CTP (Penn State) MM5 • NWS-LBB RAMS • Ohio State University MM5 • Penn State University MM5 • Penn State University MM5 Tropical Prediction System • RED IBERICA MM5 (Consortium of Iberic modelers) MM5 (click on Aplicaciones) • Saint Louis University MASS • State University of New York - Stony Brook MM5 • Taiwan Civil Aeronautics AdministrationMM5 • Texas A\&M UniversityMM5 • Technical University of MadridMM5 • United States Air Force, Air Force Weather Agency MM5 • University of L'Aquila MM5 • University of Alaska MM5 • University of Arizona / NWS-TUS MM5 • University of British Columbia UW-NMS/MC2 • University of California, Santa Barbara MM5 • Universidad de Chile, Department of Geophysics MM5 • University of Hawaii MM5 • University of Hawaii RSM • University of Hawaii MM5 • University of Illinois MM5, workstation Eta, RSM, and WRF • University of Maryland MM5 • University of Northern Iowa Eta • University of Oklahoma/CAPSARPS • University of Utah MM5 • University of WashingtonMM5 36km, 12km, 4km • University of Wisconsin-Madison UW-NMS • University of Wisconsin-Madison MM5 • University of Wisconsin-Milwaukee MM5 • Mesoscale forecast models are being run by universities, in real time, at dozens of sitesaround the country, often in collaboration with local NWS offices • Tremendous value • Leading to the notion of “distributed” NWP • Yet only a few (OU and U of Wash) are actually assimilating local observations – which is one of the fundamental reasons forsuch models!

3. Science Drivers for Local Modeling • Many weather phenomena that affect society and commerce occur on the mesoscale. E.g., squall-lines, snowbands, hurricanes; downslope windstorms, lake-effect snowfall, etc. • Need high-resolution local modelling to accurately resolve and predict these phenomena; • Utilize dense local observations (e.g., Mesonets); • Resolve local topography • Collaboration with local NWS forecast offices; Show examples

4. Technology Trends Enabling A New Generation of Local NWP Activities • Commodity microprocessors & inexpensive but powerful workstations/clusters • High-bandwidth networks (e.g., Internet 2) • Transparent data access and delivery • Community Models (MM5, WRF) • Local observatories (e.g., mesonets) • Community codes for data assimilation (e.g., 3DVAR, ADAS)

5. Numerical Weather Prediction: Key Steps Observations & Previous Model Forecast Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Radar Data Wind Profilers GPS/Met instruments • Analysis/Assimilation • Quality Control • Retrieval of Unobserved • Quantities • Creation of Gridded Fields Prediction PCs to Teraflop Systems • Product Generation, • Visualization, • Dissemination • End Users • NWS • Private Companies • Students

6. Unidata Technologies [That can be Used] in Local Modeling • Local Data Manager – data transport • Data streams: IDD and CONDUIT – Relaying and accessing data • Decoders – Data conversion • NetCDF libraries and tools – Data infrastructure • OPeNDAP – Remote data access (Collaborator) • THREDDS – Cataloging data • GEMPAK and IDV - Visualization • GIS Integration tools (in future)

7. Real-time Data Distribution Model Satellite Radar There are over 150 university sites in North and South America, Europe, and Asia that receive real-time data using the Unidata Local Data Manager; Plus there are over 300+ LDM sites in NWS, NOAA, NASA, KMA, Taiwan, and Spain that are not part of the “open” IDD.

8. LDM in Action SuomiNet WSR 88-D Data LDM is providing a variety of real-time meteorological observations and model output from operational prediction systems for local NWP initialization

9. National Forecast Model Output Real-time Weather Data Today’s Local NWP Process at Many Universities User running local analysis and display tools Meteorological Assimilation System Decoders Decoders Decoders Regional Model Hosted on local hardware Assimilated Data For Initial Conditions

10. National Forecast Model Output Real-time Weather Data Today’s Local NWP Process at Many Universities There is no Data Sharing (other than with local NWS offices) User running local analysis and display tools Meteorological Assimilation System Decoders Decoders Decoders Regional Model Hosted on local hardware Assimilated Data For Initial Conditions

11. OPeNDAP Servers Unidata Motherlode Server Unidata LEAD Testbed There are many OPeNDAP servers for operational and historical data, but none outside of Unidata & LEAD for real-time local NWP output

12. Remote Data Access and Catalogs Developed for real-time WRF predictions from University of Illinois. Courtesy: Brian Jewett

13. Integrated Data Viewer • Unidata’s newest scientific analysis and visualization tool • Provides 2, 3 and 4-D displays of geoscientific data • Stand-alone or networked application, providing client-server data access via multiple protocols • Java-based tools: Runs on Windows, Macs and Unix machines

14. Remote Visualization of Local NWP Output Developed for real-time WRF predictions from University of Illinois. Courtesy: Brian Jewett

15. Some sites convert their forecast output into a format compatible with GEMPAK analysis and visualization tool Enables integration of local model output with other operational data sets GEMPAK Example

16. THREDDS Middleware Thematic Real-time Environmental Distributed Data Services (THREDDS) To make it possible to publish, locate, analyze, visualize, and integrate a variety of environmental data Combines “push” with several forms of “pull” and digital library discovery Connecting People with Documents and Data

17. LEAD: A Large-ITR Effort • Linked Environments for Atmospheric Discovery • Identify, Access, Assimilate, Predict, Manage, Mine, and Visualize a broad array of meteorological data and model output, independent of format and physical location • A range of Grid and Web Services will be developed for dynamic, on-demand, end-to-end weather prediction • Institutions: U. Oklahoma, Unidata, U. Alabama, U. Illinois, U. Indiana, Millersville U., Howard U. and Colorado State U.

18. Web Services • They are self-contained, self-describing, modular applications that can be published, located, and invoked across the Web. • Web Services are emerging as tools for creating next generation distributed systems that are expected to facilitate program-to-program interaction without the user-to-program interaction. • Besides recognizing the heterogeneity as a fundamental ingredient, these web services, independent of platform and environment, can be packaged and published and they can communicate with other systems using the common protocols.

19. Data Service Decoder Service Assimilation Service Regional Model Service Product Generation & Data Mining Service LEAD Vision User Orchestrates Web Services to Create Regional Forecast User running local analysis and display tools