1 / 62

The LEAD Effort at Unidata

The LEAD Effort at Unidata. The Unidata Seminar will start at 1:30 PM MST. The LEAD Effort at Unidata. Tom Baltzer, Brian Kelly, Doug Lindholm, Anne Wilson December 14, 2005. LEAD is funded by the National Science Foundation under the following Cooperative Agreements : ATM-0331594

leanna
Download Presentation

The LEAD Effort at Unidata

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The LEAD Effort at Unidata The Unidata Seminar will start at 1:30 PM MST

  2. The LEAD Effort at Unidata Tom Baltzer, Brian Kelly, Doug Lindholm, Anne Wilson December 14, 2005

  3. LEAD is funded by the National Science Foundation under the following Cooperative Agreements: ATM-0331594 ATM-0331591 ATM-0331574 ATM-0331480 ATM-0331579 ATM-0331586 ATM-0331587 ATM-0331578

  4. Outline • Setting the Stage: Introduction to LEAD and Unidata’s LEAD Efforts: Anne • Application of current technology on the LEAD testbeds: Tom • The LEAD Hardware at Unidata: Brian • The THREDDS Data Repository: Doug

  5. Setting the Stage: Introduction to LEAD and Unidata’s LEAD EffortsAnne Wilson

  6. Current IT Barriers to Mesoscale Weather Research and Education • Data and tools useable mainly by experts • Researchers and educators constrained by hardware limitations • Rigid, brittle technology can’t accommodate mesoscale weather research requirements: • real time, on demand, dynamic data processing and sensor steering

  7. A Solution: Linked Environments for Atmospheric Discovery (LEAD) • Funded by NSF Large Information Technology Research (ITR) award • Produce a web service based, scalable framework for handling meteorological data and model output: • Identifying, accessing, preparing, assimilating, predicting, managing, analyzing, mining, visualizing • Independent of data format and physical location • Dynamically adaptive workflows and steering of sensors

  8. The LEAD Vision • Data access via querying, and browsing • Analysis and forecast tools that can be composed into workflows • Workflows and sensors that respond to the weather • Support users ranging from grade 6 to experienced researchers

  9. LEAD Objectives • Lower the barrier for entry and increase the sophistication of problems that can be addressed by complex end-to-end weather analysis and forecasting/simulation tools • Improve our understanding of and ability to detect, analyze and predict mesoscale atmospheric phenomena by interacting with weather in a dynamically adaptive manner • Result: Paradigm change in how experiments are conceived and performed

  10. LEAD Challenges

  11. Multidisciplinary Effort • Meteorology • Computer Science and Information Technology • Education and Outreach

  12. LEAD Institutions > 100 scientists, students, technical staff

  13. LEAD Thrust Groups • Data* • Orchestration • Portal • Meteorology • Grid and Web Services Test Bed* • Education and Outreach Test Bed *Major Unidata areas

  14. LEAD Portal Query Service Ontology Service Resource Catalog myLEAD Catalog Dictionary LEAD Data Subsystem Public Data (e.g. IDD data) LEAD Data Repository (LDR)

  15. Unidata Technology Used in LEAD • LDM/IDD Data Delivery: near real time data delivery • THREDDS: catalogs of data and their associated metadata • Common Data Model (CDM): single interface to multiple data formats • THREDDS Data Server (TDS): integrated OPeNDAP and http data access • Integrated Data Viewer (IDV): visualization • THREDDS Data Repository (TDR): data storage framework • Decoders

  16. Unidata and LEAD • Unidata also brings: • Experience with atmospheric data • Community of users • Robust, fielded software

  17. Goal: Support both LEAD and our community Recent LEAD-Related Efforts 2. Application of current technology on our LEAD testbed: Tom 3. Structure of the LEAD testbed: Brian 4. THREDDS Data Repository: Doug

  18. Application of Current Technologies on the LEAD Testbed Systems Tom Baltzer

  19. Acronyms for LEAD Tools ADAS - ARPS Data Assimilation System (Center for Advanced Prediction of Storms at OU) ADaM - Algorithm Development and Mining (University of Alabama at Huntsville) IDV – Integrated Data Viewer (Unidata) LDM/IDD – Local Data Manager/Internet Data Distribution (Unidata) OPeNDAP – Open-source Project for a Network Data Access Protocol (OPeNDAP.org) THREDDS – Thematic Real-time Environmental Distributed Data Services TDS - THREDDS Data Server TDR – THREDDS Data Repository (Unidata) WRF – The Weather and Research Forecasting Model (ARW Core - NCAR) Also: WS-Eta – Workstation Eta Model

  20. LEAD Testbed Systems Testbed systems at several LEAD locations to provide: Data Near Real-Time data ingest, storage and access LEAD Data Product storage and access Data Processing High Performance Computing Grid and Web Services Allow each institution to develop methods by which their capabilities fit into LEAD effort Single Web Portal system at Indiana Univ. to bring it all together and provide User Interface

  21. MU CSU HU Unidata UI IU UNC OU UAH LEAD Grid Core Academic Partner Core Academic Partner + Education Test Bed Core Academic Partner + Grid Test Bed + Education Test Bed Core Academic Partner + Grid Test Bed

  22. Data Aspects of LEAD Testbeds

  23. LEAD Testbed Systems UPC Technologies being leveraged to facilitate LEAD needs LDM/IDD THREDDS IDV NetCDF Decoders OPeNDAP (Unidata supported)

  24. Forecast Model Output Typical LEAD Testbed (Current Source Data Configuration) LEAD Grid System Weather station observations Testbed System THREDDS Catalog OPeNDAP IDD Aircraft data Decoders GridFTP Radar data

  25. Forecast Model Output Typical LEAD “Data” Testbed (Future Source Data Configuration) LEAD Grid System Weather station observations Testbed System THREDDS Catalog OPeNDAP IDD TDS & TDR Aircraft data Decoders GridFTP Radar data Note: UPC plans ~ 6 month store

  26. LEAD Processing on the Unidata Testbed System

  27. UPC Processing Testbed (Current Configuration) - WRF being Steered by Chiz’s GEMPAK precipitation locator NCEP NAM (Eta) Forecast Initial and Boundary Conditions Precipitation Locator WRF THREDDS Catalog Center Lat/Lon Regional Forecasts OPeNDAP Access WS-Eta Unidata LEAD Test Bed

  28. CAPS ADAS Assimilation Millersville ADaM Precip Locator Next Steps NCEP NAM (Eta) Forecast Initial Conditions Center Lat/Lon Boundary Conditions Precipitation Locator WRF THREDDS Catalog Regional Forecasts OPeNDAP Access WS-Eta Unidata LEAD Test Bed

  29. IDD Datasets • Radar • Surface & Upper air • Satellite • NCEP NAM Longer Term NCEP NAM (Eta) Forecast Boundary Conditions ADaM ADAS Precipitation Locator WRF Center Lat/Lon THREDDS Catalog Regional Forecasts OPeNDAP Access WS-Eta Unidata LEAD Test Bed

  30. IDD Datasets • Radar • Surface & Upper air • Satellite • NCEP NAM Ultimately LEAD Grid System NCEP NAM (Eta) Forecast Boundary Conditions Web Service ADaM Web Service ADAS Precipitation Locator Web Service WRF Center Lat/Lon THREDDS Catalog Regional Forecasts OPeNDAP Access WS-Eta Unidata LEAD Test Bed

  31. Objectives for UPC Testbed • Testing ground for integration new UPC and LEAD technologies • Determining ways to bring LEAD Technologies to the Unidata Community • “Operational” environment for LEAD • Processing cluster • Data Storage • ~6 months of IDD data • LEAD product data

  32. The LEAD Hardware at Unidata Brian Kelly

  33. Existing LEAD Infrastructure Lead3 HTTP Server THREDDS Server OpenDAP Server LDM Node NFS Server Cluster Node Lead1 GRID Server Development Tools NFS Server Cluster Node Lead4 TDS LDM Node NFS Server Cluster Node Lead2 GRID Server NFS Server Cluster Node Cluster Monitoring LeadStor 8 TB of Disk NFS Server

  34. Portal Servers for Web, TDS, Grid and LDM Services UCAR/Unidata LEAD Infrastructure ~30 GFLOP Processing Cluster 40 TB Storage Cluster

  35. HTTP, TDS and Grid Server LDM Server Test Server Processing Cluster Head Node Storage Cluster Gateway Gigabit Network for NFS Storage Access LEAD Portal Systems

  36. LEAD Processing Cluster Beowulf Cluster Connected by a Gigabit Fibre Network Each Node contains Two Athlon 2400+ CPUs Cluster Uses OSCAR with the MPICH MPD Eight Nodes is ~30 GFLOPs

  37. LEAD Storage Cluster LEAD Storage Head Node LEAD Storage Gigabit Network LEAD Storage Nodes

  38. One (1) Guanghsing GHI-583 5U Case 24 hot swapable SATA trays 1000W 2+2 power supply • One (1) Tyan Thunder K8SD Pro Motherboard Dual Opteron CPUs Four 64-bit 133/100 Mhz PCI-X Slots Two Gigabit Ethernet ports • One (1) AMD Opteron 242 Processor 1.6 Ghz CPU • Three (3) Broadcom RAIDCore BC4853 Eight SATA ports Controller spanning Advanced raid • Twenty-Four (24) Seagate Barracuda ST3400832AS 7200 RPM 400GB SATA Drives LEAD Storage Node

  39. LEAD Storage Node Twenty-Four (24) 400 GB Drives Divided into Two (2) Eleven Column RAID 5 Arrays and Two Hot Spares Form Two (2) 4 TB LUNs Using bcraid Each Node Publishes the Two LUNS over iSCSI

  40. LEAD Storage Gateway • Mounts Each Node's Two (2) 4 TB LUNs Published via iSCSI • Builds Two (2) 20 TB 6 column RAID 5 Meta-devices using mdadm • Divides Each Meta-device into Volume using LVM • Each Volume is Formatted with an XFS Filesystem • Each Filesystem is Published with NFS Result: 40 TB of mid-performance double-redundant storage

  41. THREDDS Data Repository (TDR) Doug Lindholm

  42. Unidata NCSA OU UAH IU LEAD ArchitectureData Storage Perspective LEAD Data Grid

  43. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk LEAD ArchitectureData Storage Perspective Cataloger (myLEAD) LEAD Data Grid “Atomic” Capabilities

  44. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk LEAD ArchitectureData Storage Perspective Forecast Model (WRF) Data Assimilation (ADAS) Data Mining (ADAM) Cataloger (myLEAD) Visualization (IDV) LEAD Data Grid Application Services “Atomic” Capabilities

  45. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk LEAD ArchitectureData Storage Perspective Forecast Model (WRF) Data Assimilation (ADAS) Portal Data Mining (ADAM) Cataloger (myLEAD) Visualization (IDV) User LEAD Data Grid Application Services “Atomic” Capabilities

  46. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk LEAD ArchitectureData Storage Perspective Forecast Model (WRF) Data Assimilation (ADAS) Portal Data Mining (ADAM) Cataloger (myLEAD) Visualization (IDV) User LEAD Data Grid Application Services “Atomic” Capabilities

  47. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk LEAD ArchitectureData Storage Perspective Forecast Model (WRF) Data Assimilation (ADAS) Portal Data Mining (ADAM) Cataloger (myLEAD) Visualization (IDV) User LEAD Data Grid Application Services “Atomic” Capabilities

  48. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk LEAD ArchitectureData Storage Perspective Forecast Model (WRF) Data Assimilation (ADAS) Portal Data Mining (ADAM) Cataloger (myLEAD) Visualization (IDV) User LEAD Data Grid Application Services “Atomic” Capabilities

  49. Storage Locator Data Mover Unidata NCSA ID Generator OU UAH Name Resolver Metadata Generator IU Metadata Crosswalk Cataloger (myLEAD) LEAD ArchitectureData Storage Perspective Forecast Model (WRF) Data Assimilation (ADAS) THREDDS Data Repository Portal Data Mining (ADAM) Visualization (IDV) User LEAD Data Grid Data Repository Application Services “Atomic” Capabilities

  50. THREDDS Data RepositoryComponent Architecture Data Storage Name Resolver Metadata Crosswalk Metadata Generator Data Mover Unique ID Generator Storage Locator Cataloger locate- Storage() move- Data() generate- UniqueID() mapID- ToURL() generate- Metadata() translate- Metadata() catalog- Metadata() THREDDS Data Repository putData() discoverData() getData()

More Related