1 / 45

Tribal Water Quality Data Management

Tribal Water Quality Data Management. NCAI Exchange Network Tribal Users Meeting April 9, 2008. Overview. Tribal 106 Grants Requirements for Monitoring and Assessment Essential concepts of Water Quality Data Management Understanding WQX Steps to a WQX Submittal Accessing Data

thiery
Download Presentation

Tribal Water Quality Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tribal Water Quality Data Management NCAI Exchange Network Tribal Users Meeting April 9, 2008

  2. Overview • Tribal 106 Grants Requirements for Monitoring and Assessment • Essential concepts of Water Quality Data Management • Understanding WQX • Steps to a WQX Submittal • Accessing Data • Upcoming Developments • How to decide where to begin

  3. Tribal 106 Guidance Requirement • All tribes must submit a TAR (Tribal Assessment Report) to continue eligibility for 106 funds (8-3) • Monitoring Strategy • Water Quality Assessment • STORET-compatible electronic data

  4. 106 Guidance parameters Fundamental Programs: - Dissolved Oxygen (DO) - Temperature - pH - Turbidity Intermediate Programs: - Nitrogen - Phosphorus Mature Programs: - Pathogens - Macroinvertebrates - Physical Habitat indicators

  5. 10 Elements of a Tribal Monitoring Strategy I. Monitoring Strategy II. Monitoring Objectives III. Monitoring Design IV. Core Indicators V. Quality Assurance VI. Data Management VII. Data Analysis/Assessment VIII. Reporting IX. Programmatic Evaluation X. General Support and Infrastructure

  6. Essentials of Data Management • Monitoring data consists of: • Where sample/measurement was collected • When it was collected • What was collected • How it was collected • Why it was collected • Who collected it • It does not take a lot of information to create a good data set.

  7. How does data management fit in? The life of a sample begins here Field Sheets capture data Labs generate data Now what to do with all this data?

  8. First, make sense of what you have Organize data electronically • Use the tools you’re familiar with: • Access • Excel • Other higher-powered database management system (i.e. Oracle) Have a plan for growth Not only will your skill grow, but your data volume will grow. Be prepared for it. Find out what’s already available

  9. Data are valuable, plan for re-use Electronic data are more valuable than data in file cabinets The more data are re-used, the more valuable they become Shared data are of even higher value • Provide for better planning decisions • Incentivize collaborative efforts • Make the most use of the monitoring $$s being invested

  10. How EPA Describes Monitoring Data Three categories of data: • Projects (Why data were collected) - Brief summary of monitoring plan • Monitoring Locations (Where data were collected) - Describe where monitoring takes place • Results (When, How and What were collected) - Measurements of what were monitored

  11. ORGANIZATION ORG Description ORG Electronic Address ORG Phone ORG Address ACTIVITY GROUP Activity Group Name Activity Group Type Activity IDs PROJECT Project Description Project Binary Object MONITORING LOCATION Monitoring Location Identity Monitoring Location Geospatial Monitoring Location Binary Object MONITORING ACTIVITY Activity Description Activity Location Sample Description Sample Prep Subsample Description Activity Binary Object RESULT Result Description Result Binary Object Result Analytical Method Result Lab Info Result Detection Quantitation Level Lab Sample Prep WQX is designed around these concepts

  12. The WQX Schema • The WQX schema is a definition of the structure and format that data must have to be submitted through WQX. • Primary components: Organization, Project, Monitoring Location, Monitoring Activity, Samples, and Results • Data elements currently accommodate physical, chemical, and fish tissue data • Biological monitoring and habitat data flow will be incorporated in the very near future

  13. Submitting data using WQX • Water Quality Exchange (WQX) uses the Exchange Network to flow physical, chemical and fish tissue, and soon biological and habitat data from an tribe’s own database to the existing STORET Warehouse • Tribes that have been awarded Exchange Network Grants will be able to submit data through WQX

  14. How WQX Works Approved XML Submission XML Submission CDX WQX Parse and Load Software Data Context Error Report Schema Validation Receive Error Responses Data Load Confirmation and Summary Data Submitter WQX ODS

  15. WQX in the big picture

  16. Steps to Submitting to WQX • Review the resources that are available • Getting involved with the Exchange Network • Getting a NAAS Account • Getting an Organization ID • Understanding XML (How does my data relate to an XML file?) • Submitting a file • Understanding errors • Understanding the WQX structure • Interacting with WQX

  17. Step 1. Resources Available • The STORET/WQX team has several resources available to help you: • Available at the STORET web site: www.epa.gov/storet • Visit the WQX section: http://www.epa.gov/storet/wqx.html • Diagrams and presentations about how WQX works • Presentations from others on how they’ve implemented WQX (including tribes) • Lists of allowable values for WQX • Available at the Exchange Network Site: http://www.exchangenetwork.net/exchanges/water/wqx.htm • Read the Flow Configuration Document • See example XML Files • Examine the Data Exchange Template • Last but not least, don’t be afraid to ask for help: • Join the STORET list serve: www.epa.gov/storet/listserv.html • Call the STORET hot line: 1-800-424-9067 • end an email to storet@epa.gov

  18. Step 1. Remember this All of this information is available at: www.epa.gov/storet

  19. Step 2. Getting involved in the Exchange Network • Ask yourself the following questions: • Does my organization have a node? • Have we submitted data for another flow? • Did we use a node? Node client? Or CDX Web? • Who is my Node Administrator? Do we have one? • If the answer to these questions is ‘No’ or ‘I don’t know’ visit: www.exchangenetwork.net

  20. Step 3. Getting a NAAS Account • The NAAS or Network Authentication Authorization Service is a user account that you can use to access the Exchange Network. They can be specific for an individual node or for a particular user • NAAS accounts are assigned by Node Administrators. If you don’t have a Node Administrator, contact the CDX help desk at: 888-890-1995 Or epacdx@csc.com

  21. Step 4. Getting an Org ID • An Organization ID (Org ID) is a unique identifier assigned to your organization (i.e. YUROK for the Yurok Tribe). • Org IDs are assigned by EPA upon the request of the user. • You must have an Org ID to be able to submit to WQX • To request an Org ID, send an email to storet@epa.gov • Provide the NAAS accounts that you want to have access to your Organization

  22. Step 5. Understanding XML • XML can be intimidating at first • There are many ways to create XML, some things to do: • The sample files and Data Exchange Template are key to understanding how to create a WQX XML file. • Find a programmer who can spend 20-40 hours on designing an XML file • Have some test data that are representative of the data you’ll be submitting • Test the procedures that the programmer developed (plan for an additional 20-40 hrs testing) • Start small, it takes time to create an XML file, don’t waste development time waiting for a computer to process a file. ***You should only have to do this once, after that, you’re just running the procedure.

  23. Step 6. Submitting a File • Once you have a NAAS account and an Org ID, and have successfully created an XML file, you’re ready to submit. • The original test file should be small, but representative of the data you’ll be submitting • Submit to CDX test first • Once you get a successful submission, keep track of the transaction ID • Send an email to the CDX help desk:…. Requesting access to WQX production. Provide them the transaction ID of the successful submission. • Go ahead an begin submitting to production

  24. Step 7. Understanding Errors • There are two types of errors in WQX: 1. Schema Errors: the submitted file does not conform to the WQX Schema. (Report is available in the file ‘Validation Results’ which is an xml file). 2. Processing Errors: the submitted file does not pass all of the WQX Business Rules (Report is available in the file ‘ProcessingReport.zip’)

  25. Step 8. Understanding the WQX Structure • WQX files should be organized as follows: • Header (with processing instructions) • Organization • Organizational Information • Projects • Monitoring Locations • Activities • Linkage to Projects • Linkage to Monitoring Locations • Results • Activity Groups

  26. Step 9. Interacting with WQX • WQX is designed to function in a Node-to-Node environment (i.e. you manage your data, let your computer and Node keep EPA in sync). • WQX allows Inserts, Updates, and Deletes via a batch process • WQX has Outbound services via the Query and Solicit method data visualization and system synchronization purposes.

  27. Accessing Data The National STORET Data Warehouse serves as the national repository for Modern STORET data as well as WQX data. Users may obtain data from the warehouse through the STORET home page : www.epa.gov/storet Web services providing direct access to warehouse data will also provide enhanced accessibility to the data.

  28. STORET Warehouse Updates • The National STORET Data Warehouse enhancements for improved data accessibility • New Watershed Summary – Available Now • New Web Services – Available Now • Please visit the STORET website for more information : www.epa.gov/storet

  29. Where do Web Services fit? Data in/out services via WQX WH services available via Exch. Net. Data out services through direct access to the WH

  30. All Done with Web Services!!

  31. An example using Monitoring Data

  32. What’s coming next? • WQX 2.0 is near completion • Adds Biological data and Habitat data to the schema • Supports Bio/Habitat metrics and indexes • EPA will continue to support WQX 1.0 for the foreseeable future • XML Generation Tool • Tool that will support organizations that don’t have the ability to create XML • Batch Geo-location tool • Tie Monitoring Locations to the National Hydrography Dataset

  33. Dates to Remember • WQX Version 1.0 data flow in production now • Physical, chemical, microbiological, and fish tissue data only • Add Biological/Habitat data to WQX schema • Bio/Habitat WQX in production (May 08) • Develop Web tool for XML Generation • Tool available for general use (September 08) • Batch Geo-location Tool • Under development available January 2009

  34. What is the XML Generation Tool? • A web-based application that assists a data provider with the creation of a valid submission file for the Water Quality Exchange. • The next generation of SIM and WebSIM, redesigned to target WQX.

  35. Data-sharing for all • This tool is meant to allow those who do not have Exchange Network Nodes the ability to share data over the web • The tool will be easy to use and accept basic input formats (i.e. Excel Spreadsheets). • Data submitted via this tool will be available on the internet within one week of submission.

  36. State/Tribal Database WQX Database How the XML Generation Tool fits in • XML Generation Tool • Import Flat File • Apply Defaults & Translations • Validate & Correct • Export XML • CDX Lite • Submit • Get Status • Download • CDX Web • Online Registration • Log In • WQX System • Load XML File • Update WQX Database • Create Processing Report • State/Tribal System • Retrieve Data from Database • Export XML • State/Tribal Node • Submit • Get Status • Download CDX Node

  37. Demo By the Way, what happened to National WebSIM?

  38. Batch Geo-Locating • New development that EPA is working on • Allow for any new stations submitted via WQX to be automatically georeferenced to the NHD+ • The Georeferenced reach codes would be returned to the data provider, as well an opportunity to QA the georeferenced locations • This will use a concept called ‘Rain Drop’ to ensure that points are tied to the correct NHD reach

  39. If the two points represent Accuracy/Precison issues then note how raindrop results in same indexed location • Note how upper point with straight snap would go to wrong catchment • Note how the path does trace the topo delineation for water

  40. Why this is Important? • Allows for monitoring data to be easily linked to other types of data (i.e. Impaired waters, assessments, permits, etc.) • Allows for upstream/downstream analysis • Greatly expands the usability of monitoring data

  41. Batch Indexing Timeline • This capability should be available January 2009 • Deployed as a service that any data flow could use (not just for WQX) • Also provide some Geospatial QA services (i.e. is the lat/long in the county/HUC/state that the user specified)

  42. Where to Get Started • Figure out what data you have, and what format it is in. • Find out where you stand with the exchange network • Visit: www.epa.gov/storet • Give us a call, we can walk you through any questions.

  43. Web Resources • www.epa.gov/storet • www.exchangenetwork.net/exchanges/water/wqx.htm • www.epa.gov/storet/wqx.html

  44. Question/Comments • Contact: • Dwane Young, EPA (202)566-0616 Young.dwane@epa.gov • Kristen Gunthardt, EPA (202)566-1194 gunthardt.kristen@epa.gov • STORET Help Desk 800-424-9067 storet@epa.gov Join the STORET list serve for more information, including monthly conference calls about STORET and WQX: www.epa.gov/storet/listserv.html

More Related