A Services Oriented Architecture for Water Resources Data

1. A Services Oriented Architecture for Water Resources Data David R. Maidment Center for Research in Water Resources University of Texas at Austin

2. A Services Oriented Architecture for Water Resources Data • CUAHSI and WATERS • WaterML and WaterOneFlow • Hydrologic Information Server • Modeling services

3. A Services Oriented Architecture for Water Resources Data • CUAHSI and WATERS • WaterML and WaterOneFlow • Hydrologic Information Server • Modeling services

4. What is CUAHSI? UCAR • CUAHSI –Consortium of Universities for the Advancement of Hydrologic Science, Inc • Formed in 2001 as a legal entity • Program office in Washington (5 staff) • NSF supports CUAHSI to develop infrastructure and services to advance hydrologic science in US universities Unidata Atmospheric Sciences Earth Sciences Ocean Sciences CUAHSI National Science Foundation Geosciences Directorate HIS

5. CUAHSI Member Institutions 112 US Universities as of September 2007

6. WATERS Network Information System Minnehaha Creek HIS Team WATERS Testbed NSF has funded work at 11 testbed sites, each with its own science agenda. HIS supplies the common information system

7. HIS Team and its Cyberinfrastructure Partners Government: USGS, EPA, NCDC Industry: ESRI, Kisters, Microsoft CUAHSI HIS HIS Team: Texas, SDSC, Utah, Drexel, Duke Domain Sciences: Unidata, NCAR LTER, CZEN GEON Super Computer Centers: NCSA, TACC

8. HIS, WATERS and the CUAHSI Community Government: USGS, EPA, NCDC Industry: ESRI, Kisters, Microsoft CUAHSI HIS WATERS Network Information System HIS Team WATERS Testbed Domain Sciences: Unidata, NCAR LTER, GEON Super computer Centers: NCSA, TACC

9. Government: USGS, EPA, NCDC, USDA Industry: ESRI, Kisters, OpenMI CUAHSI HIS WATERS Network Information System HIS Team WATERS Testbed Domain Sciences: Unidata, NCAR LTER, GEON Super computer Centers: NCSA, TACC International Partners European Commission Water database design and model integration (OpenMI) CSIRO and Bureau of Meteorology Australian Water Resources Information System

10. HIS Goals • Hydrologic Data Access – providing better access to a large volume of high quality hydrologic data across the nation • Support for Observatories – integrating local observations by academic investigators with hydrologic data for a region • Advancement of Hydrologic Science – modeling and analysis of “hydrology in a dynamic earth” • Hydrologic Education – bringing more data into the classroom

11. A Services Oriented Architecture for Water Resources Data • CUAHSI and WATERS • WaterML and WaterOneFlow • Hydrologic Information Server • Modeling services

12. Definition The CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of data sources and functions that are integrated using web services so that they operate as a connected whole.

13. Services Oriented Architecture (from Wikipedia) • Service-oriented Architecture (SOA) is an architectural design pattern that concerns itself with defining loosely-coupled relationships between producers and consumers. • A major focus of Web services is to make functional building blocks accessible over standard Internet protocols that are independent from platforms and programming languages. • The Web Services Description Language (WSDL, pronounced 'wiz-dəl' or spelled out, 'W-S-D-L') is an XML-based language that provides a model for describing Web services. Defined by the World Wide Web Consortium (W3C)

14. Web Pages and Web Services http://www.safl.umn.edu/ Uses Hypertext Markup Language (HTML) Uses WaterML (an eXtended Markup Language for water data)

15. Water Data Water quantity and quality Soil water Rainfall & Snow Modeling Meteorology Remote sensing

16. Water Data Web Sites

17. NWISWeb site output # agency_cd Agency Code # site_no USGS station number # dv_dt date of daily mean streamflow # dv_va daily mean streamflow value, in cubic-feet per-second # dv_cd daily mean streamflow value qualification code # # Sites in this file include: # USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC # agency_cd site_no dv_dt dv_va dv_cd USGS 02087500 2003-09-01 1190 USGS 02087500 2003-09-02 649 USGS 02087500 2003-09-03 525 USGS 02087500 2003-09-04 486 USGS 02087500 2003-09-05 733 USGS 02087500 2003-09-06 585 USGS 02087500 2003-09-07 485 USGS 02087500 2003-09-08 463 USGS 02087500 2003-09-09 673 USGS 02087500 2003-09-10 517 USGS 02087500 2003-09-11 454 Time series of streamflow at a gaging station USGS has committed to supporting CUAHSI’s GetValues function

18. Observation Stations Map for the US Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System USGS National Water Information System NOAA Climate Reference Network

19. Water Quality Measurement Sites in EPA Storet Substantial variation in data availability from states Data from Bora Beran, Drexel University

20. Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

21. Geographic Integration of Storet and TCEQ Data in HIS

22. Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

23. Point Observations Information Model http://www.cuahsi.org/his/webservices.html USGS Data Source GetSites Streamflow gages Network GetSiteInfo Neuse River near Clayton, NC Sites GetVariables GetVariableInfo Discharge, stage (Daily or instantaneous) Variables GetValues Values 206 cfs, 13 August 2006 {Value, Time, Qualifier, Offset} • A data source operates an observation network • A network is a set of observation sites • A site is a point location where one or more variables are measured • A variable is a property describing the flow or quality of water • A value is an observation of a variable at a particular time • A qualifier is a symbol that provides additional information about the value • An offset allows specification of measurements at various depths in water

24. Locations Variable Codes Date Ranges WaterML and WaterOneFlow STORET Data GetSiteInfo GetVariableInfo GetValues Data NAM NWIS WaterML Data WaterOneFlow Web Service Data Repositories Client EXTRACT TRANSFORM LOAD WaterML is an XML language for communicating water data WaterOneFlow is a set of web services based on WaterML

25. Set of query functions Returns data in WaterML WaterOneFlow

26. Data Sources NASA Storet Ameriflux Unidata NCDC Extract NCAR NWIS Transform CUAHSI Web Services Excel Visual Basic ArcGIS C/C++ Load Matlab Fortran Access Java Applications http://www.cuahsi.org/his/ Some operational services

27. GetValues for Dissolved Oxygenat Minnehaha Creek http://his.safl.umn.edu/SAFLMC/cuahsi_1_0.asmx?WSDL

28. A Services Oriented Architecture for Water Resources Data • CUAHSI and WATERS • WaterML and WaterOneFlow • Hydrologic Information Server • Modeling services

29. Supports data discovery, delivery and publication Data discovery – how do I find the data I want? Map interface and observations catalogs Metadata based Search Data delivery – how do I acquire the data I want? Use web services or retrieve from local database Data Publication – how do I publish my observation data? Use Observations Data Model Hydrologic Information Server

30. Water Resource Regions and HUC’s

31. National Hydrography Dataset

32. NHDPlus for Region 17E

33. NHDPlus Reach Catchments ~ 3km2 Average reach length = 2km 2.3 million reaches for continental US About 1000 reach catchments in each 8-digit HUC

34. Slope Elevation Mean annual flow Corresponding velocity Drainage area % of upstream drainage area in different land uses Stream order Reach Attributes

35. Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

36. GetSites GetSiteInfo GetVariables GetVariableInfo GetValues Hydrologic Information Server WaterOneFlow services DASH – data access system for hydrology ArcGISServer Geospatial Data Observations Data Microsoft SQLServer Relational Database

37. Minnehaha Creek Experimental Site http://his.safl.umn.edu/DASH/

38. Dissolved Oxygen (mg/L) Depth (m) Water Temperature (°C) Dissolved Oxygen (% saturation)

39. Syntactic mediation Heterogeneity of format Use WaterML to get data into the same format Semantic mediation Heterogeneity of meaning Each water data source uses its own vocabulary Match these up with a common controlled vocabulary Make standard scientific data queries and have these automatically translated into specific queries on each data source Data Heterogeneity

40. request return return request NAWQA request return return request NAM-12 request return NWIS request return request return return request NARR Objective • Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them What we are doing now ….. Michael Piasecki Drexel University

41. NAWQA NWIS NARR HODM Semantic Mediator What we would like to do ….. GetValues GetValues GetValues GetValues generic request GetValues GetValues Michael Piasecki Drexel University GetValues GetValues

42. Hydroseekhttp://www.hydroseek.org Supports search by location and type of data across multiple observation networks including NWIS and Storet

43. A Services Oriented Architecture for Water Resources Data • CUAHSI and WATERS • WaterML and WaterOneFlow • Hydrologic Information Server • Modeling services

44. Project sponsored by the European Commission to promote integration of water models within the Water Framework Directive • Software standards for model linking • Uses model core as an “engine” • http://www.openMI.org

45. Time, T D Space, L Variables, V OpenMI Conceptual Framework All values are referenced in a what-where-when framework, allowing different data resources or models to communicate data VALUES An application of the data cube to integrate simulation models Jon Goodall, Duke University

46. User interface Typical model architecture Model application Application User interface + engine Engine Simulates a process – flow in a channel Accepts input Provides output Model An engine set up to represent a particular location e.g. a reach of the Thames Write Input data Run Read Engine Write Output data

47. Rainfall Runoff Model River Model Linking modelled quantities

48. User interface User interface Input data Input data Rainfall runoff River GetValues(..) Output data Output data Data transfer at run time

49. Models for the processes Rainfall(database) RR (Sobek-Rainfall -Runoff) River (InfoWorks RS) Sewer (Mouse)

50. 3 Rainfall.GetValues 7 RR.GetValues 6 Sewer.GetValues Data exchange Rainfall(database) 4 RR (Sobek-Rainfall -Runoff) 2RR.GetValues 1 Trigger.GetValues 5 8 River (InfoWorks-RS) call Sewer (Mouse) 9 data