Integration through Data Management

1. Integration through Data Management The “SEACOOS experience” Accomplishments and Obstacles • SEACOOS approach to data integration • Accomplishments and contributions to IOOS interoperability • Some “lessons learned” • Coordination with nascent Regional Association

2. SEACOOS was initiated in 2002 with ONR funding to develop a coastal ocean information system for FL, GA, SC and NC. SEACOOS Over-Arching Goal: To significantly increase the quantity and quality of environmental information from the coastal ocean of the SE U.S. and facilitate its use in a wide range of societal, scientific, and educational applications.

3. Observational Platforms Contributed by SEACOOS Partners

4. Real-Time Observations: SE Coastal Ocean

5. SEACOOS Modeling Coordinating modelsimulations forthe entire region

6. SEACOOS Information Management Data Management Coordinating Committee (DMCC) Coordination Madilyn Fletcher (USC), Dwayne Porter (USC), P.I. representatives Ed Kearns (U Miami), Mark Luther (USF), Harvey Seim (UNC), Nick Shay (U Miami), Jim Nelson (SkIO) People who get it done Charlton Purvis (USC), Jeremy Cothran (USC), Vembu Subramanian (USF), Jeff Donovan (USF), Sara Haines (UNC), Jesse Cleary (UNC), Liz Williams (U Miami), Tom Cook (U Miami) Web site documentation /portal Chris Calloway (UNC), Claire Eager (UNC)

8. SEACOOS achievements in regional Data Management • “Data commons” -- Protocols for data providers to make their information available. A set of standards for use with a given file format • Methods for data aggregation and display Together – a demonstration of how an RCOOS can function to aggregate and display information

9. Data “commons” • netCDF adopted as the common data language (“SEACOOS CDL”, currently version 2) -- well supported (UCAR, Unidata, Boulder, CO) -- supported under DODS/OPeNDAP -- interfaces to programming and analysis packages (e.g., Perl, Matlab) -- a flexible system for a variety of platforms/sensor types (scalar time series, vector time series, vector profiler time series,scalar gridded maps, vector unstructured maps)

10. Data “commons”(cont.) • “Data Dictionary” -- table that registers known standards with each other -- provides an English language description -- “down-the-road” advantages: options for data providers; flexibility in presentation and interpretation of information(i.e., if format is registered, the providers’ data can be represented)

11. Data Management:the Nuts & Bolts • Data is aggregated and stored • Data is normalized • Data is visualized • Data is disseminated

12. Data Aggregation & Storage • Aggregation format flavors • netCDF (in situ & RS data; model output) • PNG (RS imagery) • Storage • Relational database (in situ, model output, some RS) • Files (RS imagery) • Technology • Perl • PostgreSQL& PostGIS PostgreSQL PostGIS

13. SEACOOS Data Train • A central aggregation site as opposed to distributed network (e.g., LAS) • Link to software community developing GIS-type applications • Powerful visualization tool without limits to numbers of layers

14. Normalization • Reporting time varies • In-situ data • E.g. daily, hourly,half-hourly, every 10 minutes • Remotely-sensed data • E.g. twice daily • Reporting area varies • Remotely-senseddata passes • Round-the-clock updating is resource intensive • Balance the server load

15. Visualization : Example 1 • Production sitehttp://www.seacoos.org

16. Visualization : Example 2 • Development sitehttp://nautilus.baruch.sc.edu/rs • Main engines • PHPPHP-MapScript • Perl • MapServer

17. Visualization Fun : Graphs & Animations • Ad-hoc time-series graphs • Ad-hoc animations

18. Hurricane/Tropical Storm Jeanne 25 - 27 September, 2004 Merged wind observations and shore radar

19. Dissemination • OPeNDAP (DODS) data access • OGC-friendly: WMS, WFS • pick a layer, any layer http://nautilus.baruch.sc.edu/seacoos_misc/show_sea_coos_obs_time_ranges.php

20. Dissemination : example 1 • NC OneMapViewer http://gisdata.usgs.net/website/NC_OneMap/viewer.asp

21. Dissemination : example 2 • IntegratedOceanObservingSystemhurricanedemo http://www.openioos.org

22. Some Lessons :Interoperability Demonstrations • Communication is key : an open source attitude • Mailing lists, bulletin boards, and wiki’s proved invaluable across internal SEACOOS projects and external projects, e.g. Interoperability Demonstrations • Successfully accomplished (capabilities were demonstrated), but had “fire-drill” aspects from perspective of technical personnel • Better coordinated in Summer 2004 than previously, a more mature effort • Better adaptation of OGC WMS and WFS technologies • Additional layers • A better product “under the hood” • However, the push to get out products in the short-term was done at the expense of thoroughness • Process to add new products, identify sources seen as somewhat haphazard • Not the way to build an operational system

23. Other Obstacles / Issues • QA/QC of real-time data and archives • Needs further development; compliant with national standards • Resource availability and allocation • Many personnel have multiple roles • Crucial expertise in a small number of people • Time for documentation • Need for redundancy in comm./processing streams • Data from national providers • not always in a readily accessible form (often requires “screen scrapes”) • Engaging users / getting feedback • “In-reach” within the regional program as well as with external users • Track users and map results

24. Other SEACOOS regional DM activities • Coordination with NDBC, NWS • e.g., data push from COMPS, SABSOON • Stations part of NDBC network • Coordination with developing Regional Association (SECOORA) • Sensor/equipment inventory • Data dictionary (documentation) • Development of metadata tool/metadata system for the RCOOS (“MetaDoor” & documentation, USC) • Coordination with State Agencies

25. Equipment Inventory User Interface ComponentsQuick Search – by variable measured Query Returns: Map Full Search Query Returns: DB records

26. Equipment Inventory Administrative Interface This interface provides database access to equipment managers to add and edit the following information. • Institution: Institution Name, Abbreviation, Observing Group URL, Affiliation • Contacts: Contact Name, Phone Number, Email Address, Contact level • Stations: Institutional Station ID, Station Name, Lat/Long • Equipment: Package Description, Manufacturer, Model, Equipment Type, Vertical Position, Power Requirement, Communication Type, Near Real-time Status, Variables Measured, Change/Calibration History, Active Status, Comments

27. Equipment inventory Next Steps • Data input and involvement from federal data providers for a more accurate picture of the regional observation system (assets, gaps, performance) • Connecting equipment data as metadata to support observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool developed at USC). • Improved data intake capability: automated batch processing for new equipment • Expanded interactive map functionality: Zoom/Pan

28. Metadoor : metadata creation

29. Metadoor : entry context features - Collapsible forms organized into tabbed sections - Fields color coded by requirement need • Help context link for each field • Listbox selections and calendars • - Form presents only subfields needed based on listbox selection

30. Metadoor : planned metadatasupport for • marineXML • sensorML • Platforms and Sensors • Observations and Measurements • OGC Sensor Web • Other metadata harvesters • OGC Catalog services • geodata.gov

31. A friend in the business • Development site: http://nautilus.baruch.sc.edu/rs • Production site: http://www.seacoos.org • Very active listserv dealing with mainly remote-sensing issues: remotesensing@caro-coops.org • mailto: majordomo@caro-coops.org • message text: subscribe remotesensing • Who am I? • Charlton Purvis, University of South Carolina, SEACOOS • cpurvis@sc.edu • Happy to help and share.