Enabling Real-time Environmental Forecasting and Management Through Cyberinfrastructure

1. Enabling Real-time Environmental Forecasting and Management Through Cyberinfrastructure Barbara Minsker Associate Professor, Civil and Environmental Engineering Director, Environmental Engineering, Science, and Hydrology Group, NCSA University of Illinois Urbana-Champaign PI and co-Director, CLEANER Project Office minsker@uiuc.edu National Center for Supercomputing Applications

2. Introduction • CLEANER and CUAHSI are jointly proposing the WATERS (WATer and Environmental Research Systems) Network • Each group has developed separate draft science plans, with joint plans to be created over the next 6 months • Emerging CLEANER draft science plan indicates significant interest in real-time environmental forecasting and management to: • Better monitor and understand large-scale, event-based environmental phenomena (e.g., hypoxia, flooding) • Create and test engineered systems (e.g., agricultural systems, flood control systems, water cycle engineering systems), that better prevent and control such phenomena National Center for Supercomputing Applications

3. Corpus Christi Bay Hypoxia Monitoring Source: David Maidment, Univ. of Texas National Center for Supercomputing Applications

4. CC Bay Researchers Currently Cannot Adapt Monitoring to Hypoxia Events • Oxygen data from continuous sondes are only downloaded periodically • Other sensor data are available in near-real-time, but connections with oxygen levels have not been quantified • E.g., wind speed & direction, water surface level, temperature • Manual sampling should be increased when probability of hypoxia is high, but researchers cannot integrate diverse data and models to predict when to mobilize • Cyberinfrastructure can create an information system to enable near-real-time, adaptive monitoring National Center for Supercomputing Applications

5. Daily Fluctuations in CCBay Sonde Data Oxygen Source: Paul Montagna, Univ. of Texas National Center for Supercomputing Applications

6. WATERS Cyberinfrastructure • Two major CI demonstrations are underway for WATERS Network: • CUAHSI Hydrologic Information System project (David Maidment, PI) • NCSA Environmental CI Demonstration (ECID) project (Barbara Minsker and Jim Myers, co-leads) • We have proposed a draft common environmental CI architecture • The 2 projects are working together at CC Bay to demonstrate how observatories can enable near-real-time forecasting and management National Center for Supercomputing Applications

7. Environmental CI Architecture: Research Services Integrated CI ECID Project Focus: Cyberenvironments Supporting Technology Data Services Workflows & Model Services Knowledge Services Meta-Workflows Collaboration Services Digital Library HIS Project Focus Analyze Data &/or Assimilate into Model(s) Link &/or Run Analyses &/or Model(s) Create Hypo-thesis Obtain Data Discuss Results Publish Research Process National Center for Supercomputing Applications

8. What are Cyberenvironments (CEs)? • An end-to-end science and engineering oriented cyberinfrastructure interface/framework: • Supports the full scientific research process including • Bidirectional group community knowledge flow • Access to Sensors/Instruments/Data/Computation • Research and Community Standard Data/Models/Workflows • Designed and built to • Reuse CE design/architecture and technologies • Meet the specific needs and processes of a scientific community • Be persistent, robust, and supported • Evolve as technology changes and scientific understanding increases National Center for Supercomputing Applications

9. ECID Cyberenvironment Components • Collaboration Service: CyberCollaboratory • Enables community sharing and collaboration • Web portal with customized tools & views • Knowledge Service: CI-KNOW • Helps users find resources & people quickly and effectively • Uses social networking technology to make referrals • Meta-workflow Service: CyberIntegrator • Studying complex environmental systems requires coupling analyses or models of different components of the systems, often created by different people • E.g., water flows drive contaminant transport in multiple media • Real-time, automated updating of analyses and modeling can require diverse tools • E.g., spreadsheets, scripts, GIS tools, models • CyberIntegrator enables heterogeneous workflows & models to be readily coupled and run on any available computational resource National Center for Supercomputing Applications

10. ECID Cyberenvironment Components • Metadata Repository • Harvests from all cyberenvironment activities to enable comprehensive knowledge services & event management • Using RDF & Kowari to log “provenance” (source & links) of all objects in “triples” (subject, object, property) • Event Broker (under development) • Mediates among the tools to: • Notify users when an event of interest happens (sensor anomaly, publication of interest, model run finishes, etc) • Launch models when an event happens (conditions indicate hypoxia possible) • Acts on subscriptions from the CyberCollaboratory • Stores and obtains events from Metadata Repository National Center for Supercomputing Applications

11. Proposed ECID Cyberenvironment Service-Oriented Architecture for WATERS HIS Data Services Workflow & Model Services Metadata Repository Meta-Workflow Service: CyberIntegrator Collaboration Service: CyberCollaboratory Event Broker (Under Development) Knowledge Service: CI-KNOW See our poster for details on the technologies in the blue boxes. National Center for Supercomputing Applications

12. How Will This Help the Researchers in Corpus Christi Bay? Consider the following scenarios that define what could be enabled…. National Center for Supercomputing Applications

13. Sensor Anomalies • Sensors are not always reliable (see above wind data), and real-time streaming data can be difficult to check by hand • ECID has developed machine learning anomaly detectors • Can be implemented with data services in CyberIntegrator to automatically detect anomalies & alert data managers National Center for Supercomputing Applications

14. Sensor Anomaly Time • Bob tells the CyberCollaboratory that he wants to: • Run anomaly detection workflow on his sensor • Be notified any time an anomaly is detected Anomaly detector is launched & detects an anomaly Bob receives a notification of the anomaly CyberIntegratror CyberCollab CyberCollab Produce Event Consume Event Subscribe Event Broker National Center for Supercomputing Applications

15. Hypoxia Alert • George Smith gets a page saying that hypoxic conditions are predicted with 80% certainty in 24 hours • George logs into the CyberCollaboratory, where he joins an ongoing chat with researchers (both local and across the country), who also received the alert, and are looking at the data and model predictions • The researchers agree that the predictions appear to be reasonable given the current conditions • George mobilizes his research team to deploy detailed manual sampling of the affected region the next morning • He uses the CyberCollaboratory to notify students & volunteers from the local region who have indicated an interest in helping with field sampling National Center for Supercomputing Applications

16. Hypoxia Alert Time George tells the CyberCollaboratory that he wants to be notified any time a hypoxia prediction workflow in the Corpus Christi area creates an alert George receives a notification from the CyberCollaboratory of the hypoxia event Jane has a near-real-time workflow running that predicts hypoxia in Corpus Christi CyberIntegratror CyberCollab CyberCollab Produce Event Consume Event Subscribe Event Broker National Center for Supercomputing Applications

17. Hypoxia Alert • When the samplers and crews are mobilized, the data they collect are transmitted back to the data store • Model predictions made by CyberIntegrator meta-workflows are updated automatically • Additional data needs are identified with CyberIntegrator meta-workflows and are transmitted back to the crews through CyberCollaboratory subscriptions • Others monitor visualizations of hypoxia in real time & discuss implications in the CyberCollaboratory • Regulators • Students across the country National Center for Supercomputing Applications

18. D2K workflows C++ code Hypoxia Model Integrator Hypoxia Machine Learning Models Anomaly Detection Replace or Remove Errors Update Boundary Condition Models IM2Learn workflows Visualize Hydrodynamics Water Quality Model Visualize Hypoxia Risk IM2Learn workflows Hydrodynamic Model Fortran numerical models Data Archive Corpus Christi Bay Near-Real-Time Hypoxia Prediction Process (Draft) Sensor net National Center for Supercomputing Applications

19. Hypoxia Alert • Hypoxia researchers across the country monitor the event and compare results with those from previous events at other sites • Jenna at University of Illinois receives an alert and uses the new data to update her nationwide hypoxia data mining analysis. She identifies common characteristics of recent hypoxia events around the nation • She posts her results on the CyberCollaboratory and requests an online peer review • Regulators receive alerts when Jenna’s results receive a positive peer review • They use the study results to consider future regulatory options. National Center for Supercomputing Applications

20. A New Paradigm for Research • Observatories will create a new paradigm for environmental research • Shared infrastructure at large scales • Interdisciplinary teams collaborating remotely to address complex environmental issues • New paradigm will enable improved understanding & management of large-scale natural environmental systems • Cyberinfrastructure can create a nationwide knowledge network for environmental researchers, as well as a link to stakeholders and education National Center for Supercomputing Applications

21. Acknowledgments • Contributors: • NCSA ECID team (Peter Bajcsy, Noshir Contractor, Steve Downey, Joe Futrelle, Hank Green, Rob Kooper, Yong Liu, Luigi Marini, Sean Mason, Jim Myers, Tim Wentling) • Corpus Christi Bay Testbed team (PIs: Jim Bonner, Ben Hodges, David Maidment, Paul Montagna) • Funding sources: • NSF grants BES-0414259, BES-0533513, and SCI-0525308 • Office of Naval Research grant N00014-04-1-0437 National Center for Supercomputing Applications