CUAHSI Hydrologic Information System: web services and related technologies

1. CUAHSI Hydrologic Information System:web services and related technologies WaterML & Web Services team: David Valentine Tom Whitenack Tim Whiteaker Matt Rodriguez HIS PIs: David Maidment Ilya Zaslavsky David Tarboton Michael Piasecki Jon Goodall and Rick Hooper (CUAHSI) www.cuahsi.org/his/

2. Hydrologic Information System Service Oriented Architecture Test bed HIS Servers HIS Lite Servers Central HIS servers External data providers Global search (Hydroseek) Deployment to test beds Customizable web interface (DASH) Other popular online clients HTML - XML Desktop clients Data publishing WaterOneFlow Web Services, WaterML WSDL - SOAP Ontology ETL services Controlled vocabularies Metadatacatalogs ArcGIS ODM DataLoader Matlab ODMTools IDL Streaming Data Loading MapWindow Excel Ontology tagging (Hydrotagger) Programming (Fortran, C, VB) Server config tools Modeling (OpenMI) WSDL and ODM registration

3. CUAHSI Member Institutions 112 Universities as of June 2007

4. HIS Goals • Hydrologic Data Access System – better access to a large volume of high quality hydrologic data • Support for Observatories – synthesizing hydrologic data for a region • Advancement of Hydrologic Science – data modeling and advanced analysis • Hydrologic Education – better data in the classroom, basin-focused teaching

5. CUAHSI HIS Partner Institutions Project co-PI in Phase 2 Collaborator in Phase I

6. 10 WATERS Testbed Sites

7. COTS Software HIS Applications Data Windows 2003 Server 4 GB Ram 500 GB Disk Quad Core CPU SQLServer 2005 IIS (Internet Information Server) Visual Studio 2005 NWIS DV ArcGIS Server DASH ODM tools NWIS IID ODDataLoader GIS Data Mxd Services WaterOneFlow Web Services Your ODM ArcGIS 9.2

9. WATERS Network Information System HIS Team WATERS Testbed

10. Government: USGS, EPA, NCDC, USDA Industry: ESRI, Kisters, OpenMI CUAHSI HIS WATERS Network Information System HIS Team WATERS Testbed Domain Sciences: Unidata, NCAR LTER, GEON Super computer Centers: NCSA, TACC International Partners European Commission Water database design and model integration (HarmonIT and OpenMI) CSIRO Land and Water Resources Water Resources Observations Network (WRON)

11. Digital Watershed How can hydrologists integrate observed and modeled data from various sources into a single description of the environment?

12. Water Data Water quantity and quality Soil water Rainfall & Snow Modeling Meteorology Remote sensing

13. Water Data Web Sites

14. NWISWeb site output # agency_cd Agency Code # site_no USGS station number # dv_dt date of daily mean streamflow # dv_va daily mean streamflow value, in cubic-feet per-second # dv_cd daily mean streamflow value qualification code # # Sites in this file include: # USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC # agency_cd site_no dv_dt dv_va dv_cd USGS 02087500 2003-09-01 1190 USGS 02087500 2003-09-02 649 USGS 02087500 2003-09-03 525 USGS 02087500 2003-09-04 486 USGS 02087500 2003-09-05 733 USGS 02087500 2003-09-06 585 USGS 02087500 2003-09-07 485 USGS 02087500 2003-09-08 463 USGS 02087500 2003-09-09 673 USGS 02087500 2003-09-10 517 USGS 02087500 2003-09-11 454 Time series of streamflow at a gaging station USGS has committed to supporting CUAHSI’s GetValues function

15. Observation Stations Map for the US Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System USGS National Water Information System NOAA Climate Reference Network

16. Water Quality Measurement Sites in EPA Storet Substantial variation in data availability from states Data from Bora Beran, Drexel University

17. Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

18. Geographic Integration of Storet and TCEQ Data in HIS

19. Point Observations Information Model USGS Data Source Return network information, and variable information within the network Streamflow gages Network Return site information, including a series catalog of variables measured at a site with their periods of record Neuse River near Clayton, NC Sites ObservationSeries Discharge, stage, start, end (Daily or instantaneous) Return time series of values Values 206 cfs, 13 August 2006 {Value, Time, Qualifier} • A data source operates an observation network • A network is a set of observation sites • A site is a point location where one or more variables are measured • A variable is a property describing the flow or quality of water • An observation series is an array of observations at a given site, for a given variable, with start time and end time • A value is an observation of a variable at a particular time • A qualifier is a symbol that provides additional information about the value

20. Observations Catalog Specifies what variables are measured at each site, over what time interval, and how many observations of each variable are available

21. Data Source and Network Controlled Vocabulary Tables Sites Variables Values Metadata e.g. mg/kg, cfs e.g. depth Streamflow Depth of snow pack Landuse, Vegetation e.g. Non-detect,Estimated, Windspeed, Precipitation A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time Metadata provide information about the context of the observation. Observations Data Model Schema (version 4.0) From Ernest To, David Maidment, CRWR

22. Set of query functions Returns data in WaterML WaterOneFlow NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

23. WaterML key elements Response Types SiteInfo Variables TimeSeries Key Elements site sourceInfo seriesCatalog variable timeSeries values queryInfo GetSiteInfo GetVariableInfo GetValues

24. variablesResponse variables 1 many timeSeriesResponse variable queryInfo timeSeries criteria sourceInfo queryURL variable values Structure of responses sitesResponse queryInfo site criteria siteInfo seriesCatalog 1 queryURL many series variable variableTimeInterval

25. More Information about WaterML…next 20 slides…/we may skip them/ Or check the specification online at http://www.opengeospatial.org/standards/dp

26. Elements Defining Spatial Location SourceInfoType for observation sites for continuous surfaces SiteInfoType DatasetInfoType (other site information) child elements (other dataset information) GeogLocationType GeogLocationType LatLonPointType LatLonPointType LatLonBoxType

27. SiteInfoResponseType • Namespaces • queryInfo • site Network Sites Variables

28. userparameters query URL queryInfo example • Parameters sent to service • URLs called (if external resource)

29. siteInfo • Name • Site Code • Location

30. geoLocation • geogLocation – geographic coordinates • LatLon point • LatLon box • localSiteXY – projected coordinates

31. series • variable – what is measured • valueCount – how many measurements • variableTimeInterval – when is it measured TimePeriodType

32. variable • variableCode – global identifier • variableName • units Sites Variables Values TimePeriodType

33. Compare with… variableTimeInterval • TimePeriodType – date range (including “last n days” • TimeInstantType – single measurement

34. queryInfo name code location site seriesCatalog Series how many variables when SiteInfo response TimePeriodType

35. VariablesResponseType • variable – same as in series element • Code, name, units Sites Variables Values

36. TimeSeriesResponseType • queryInfo • timeSeries • sourceInfo – “where” • variable – “what” • values Sites Variables Values

37. sourceInfo • SiteInfoType • Same as siteInfo element • code, name, location • DataSetInfoType • For data continuous in space • LatLonPointType • LatLonBoxType

38. values • Each time series value recorded in value element • Timestamp, plus metadata for the value, recorded in element’s attributes qualifier ISO Time value

39. value metadata examples • qualifiers • censorCode (lt, gt, nc) • qualityControlLevel (Raw, QC’d, etc.) • methodID • offset • offsetValue • offsetUnitsAbbreviation • offsetDescription • offsetUnitsCode

40. TimeSeries response queryInfo location variable values

41. OGC Harmonization Best Practices WaterML text includes steps for harmonizing with GML/O&M Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope) Align service signatures (getCapabilities) Align terminology with O&M

42. More on OGC Interactions • WaterML is published as OGC Discussion paper • We are working with OGC O&M (Observations & Measurements) authors to reconcile WaterML with OGC specs: • WOML proposal (Water Observations ML) • Think of it as WFS + O&M type service (with GetSiteInfo proxied as OGC’s GetFeatureInfo request) • The plan is to provide WOML-compliant services alongside WaterML 1.1 • There is an international group that Dave Valentine is coordinating, focused on developing a standard water information exchange schema (started at WaterML workshop in Canberra in September’07) • As we are going with OGC through this harmonization, we shall be able to assist WQX with similar mappings (more below on mapping between WaterML and WQX)

43. GetSites GetSiteInfo GetVariables GetVariableInfo GetValues Hydrologic Information Server WaterOneFlow services DASH – data access system for hydrology ArcGISServer Geospatial Data Observations Data & Catalogs Microsoft SQLServer Relational Database

44. 6 5 4 2 3 1 WORKGROUP HIS SERVER ORGANIZATION STEPS FOR REGISTERING OBSERVATION DATA DASH Web Application Web Configuration file Stores information about registered networks MXD Stores information about layers Layer info,symbology, etc. WSDLs, web service URLs Connectionstrings Spatial store WOF services NWIS-IID points NWIS-IID WS USGS SQL Server NWIS-DV points NWIS-DV WS NWIS-IID NCDC ASOS points ASOS WS NWIS-DV STORET points STORET WS ASOS EPA TCEQ points TCEQ WS STORET BearRiver points BearRiver WS TCEQ TCEQ . . . . . . More WS fromODM-WS template More synced layers BearRiver My new points My new WS . . . More databases Background layers(can be in the same or separate spatial store) Geodatabase or collection of shapefilesor both Web services from a common template My new ODM ODMs and catalogs. All instances exposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.) ODMDataLoader

45. New network registration steps Using the ODM DataLoader or other, load your data into a blank ODM instance (this will create all ODM tables that HIS relies on) Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table using the GetSitesTool. Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc. Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content) Restart the HIS service Register and test the new service at the HIS Central: http://water.sdsc.edu/centralhis/ 1 2 3 4 5 6 7

46. Central HIS Data Services Catalog

49. NAWQA NWIS NARR ODM Beyond Syntactic Uniformity: Semantic Mediator What we are doing now ….. GetValues GetValues GetValues GetValues generic request GetValues GetValues Michael Piasecki Drexel University GetValues GetValues

50. Hydroseekhttp://www.hydroseek.net Supports search by location and type of data across multiple observation networks including NWIS, Storet, and university data