1 / 20

Data Access - a common situation

Data Access - a common situation. Joe Fish. Don. Need fish data!. Hmm maybe Don can help…. To: Joe 01/02/09, 3.2, 3, 1 01/02/11, 3.1, 3, 1. *RING RING* Hi Don, I need some upper Waimak fish observations for my model . Any ideas?. Hmm, I’ve got one dataset. I’ll send it through….

evania
Download Presentation

Data Access - a common situation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Access - a common situation Joe Fish Don Need fish data! Hmm maybe Don can help… To: Joe 01/02/09, 3.2, 3, 1 01/02/11, 3.1, 3, 1 *RING RING* Hi Don, I need some upper Waimak fish observations for my model. Any ideas? Hmm, I’ve got one dataset. I’ll send it through… *RING RING* Ok. Got the data. Where is the site located? 10 minutes… 10 minutes… Oh, it’s at laughing jack bridge. Ummm. (papers shuffle) 147.123 -41.588 Coordinates? What reference system?? I think it’s GDA94 Umm, I think observations are done by Fyke net.. Ok. What method is used? Oh…how does this compare to electric fishing? Umm...... Modified From Peter Taylor (CSIRO) DON? *CLICK*

  2. The “Interoperability” Problem “We want all information to be accessible.” What does that mean? • User can discover information (space, time, topic) enabled through well-defined interfaces. • User can understand and assess information (relevance) for example enabled through well-defined vocabularies. • User can access information through well-defined protocols. Notes: Users can be many men and robots! Information can come from many data sources

  3. The “Interoperability Problem” “We want all information to be accessible” • Only information that is wellmanaged is discoverable, understandable and accessible. • Information management starts and information capture / generation (e.g. in the field). • Federated information management and access is enabled through standards.

  4. What do we need to be able to federate biodata?“Jochen’s thoughts”

  5. Towards a New Zealand interoperable biodatainfrastructure- strawperson - Data Management • Consistent Metadata • Consistent Taxa descriptors • Consistent reference information (geospatial, temporal) • Consistent description of data (parameter, method, etc.) Delivery • Agreed open standard web services

  6. Information Management made consistent through a “services stack”- biodata - Standard Web Delivery Services Web Delivery Systems Consistency through Policy, Roles and Tools Metadata Services Taxonomic Information Services Instutiutional Data Archives Observation & Measurement Names Services DS3 ... DS2 (LAKESPI) DS1 (FFDB) Dataset 1 ... Procedures & Methods Information Services

  7. How to manage data at NIWA? (excerpt) • Describe metadata for your dataset according to the NIWA Metadata Management System. • Describe observed variables / parameters / units according to the NIWA Measurement and Observation variables vocabulary. • If you use a standard method (hint: many people / groups are using the method), ensure it is catalogued in the NIWA Procedures and Methods Register. • Ensure the relevant taxa information is catalogued consistent with the NIWA Taxonomic Information Services. WHY? This will ensure the data is archived for the future, and discoverable, accessible and understandable by you your colleagues, and can be delivered through standard web interfaces.

  8. Information Management made consistent through a “services stack”- biodata - Standard Web Delivery Services Web Delivery Web Delivery Web Delivery Web Delivery Web Delivery Web Delivery Consistency through Policy, Roles and Tools Metadata Services Taxonomic Information Services Instutiutional Data Archives Instutiutional Data Archives Instutiutional Data Archives Instutiutional Data Archives Instutiutional Data Archives Instutiutional Data Archives Observation & Measurement Names Services DS3 ... DS3 ... DS3 ... DS3 ... DS3 ... DS3 ... DS2 (LAKESPI) DS2 (LAKESPI) DS2 (LAKESPI) DS2 (LAKESPI) DS2 (LAKESPI) DS2 (LAKESPI) DS1 (FFDB) DS1 (FFDB) DS1 (FFDB) DS1 (FFDB) DS1 (FFDB) DS1 (FFDB) Dataset 1 ... Dataset 1 ... Dataset 1 ... Dataset 1 ... Dataset 1 ... Dataset 1 ... Procedures & Methods Information Services

  9. NIWA EI Delivery Vision in New Zealand: Interoperability through open standards Data User Joe Public Data User NIWA Data User Council Multiple Uses Interoperability App/Portal ALA App/Portal LakeSPI App/Portal NIWA QGIS App/Portal LAWA App/Portal ei.niwa.co.nz Science User (R) Services Stack (Set of Standard Web Services) Data Delivery Standards Data Archive Wellington Council Data Archive Auckland Council Data Archive NIWA (NEMO) Data Archiving Standards QA/Management FFDB QA/Management Excel QA/Management CADDIS Data Collection&QA Standards Observations / Surveys / Data collection(to standards!)

  10. Mission of TFBIS BSS Project Dataset a TFBIS BSSproject user a Dataset b use case a Dataset a user e Dataset c Data Access: Portals, Web services, queries Dataset c ? use case b user b Dataset e Dataset d use case c user d Dataset d user c Dataset e Dataset b Data Sources different formats, different owners, different organisations Data compilation ad-hoc, manual, repeated, costly!! BSS standard formats and vocabularies, standard metadata, standard web service delivery

  11. BSS project - Milestones • Clearly identify usecases (for occupancy data) and barriers and document this. • Define required solutions and specify BSS infrastructure. • Implement infrastructure for a limited set of organisations and data sources. • Role-out across New Zealand, supported through RC, NEMS, MfE, Stats NZ, DOC, Councils, etc.

  12. Data Access - a future situation To: Joe 01/02/09, 3.2, 3, 1 01/02/16, 3.1, 3, 1 Joe Fish To: Joe 01/02/09, 3.2, 3, 1 01/02/15, 3.1, 3, 1 Need fish data! Have fish data! Magic BSS Button To: Joe 01/02/09, 3.2, 3, 1 01/02/14, 3.1, 3, 1 01/02/09, 3.2, 3, 1 01/02/11, 3.1, 3, 1 01/02/14, 3.1, 3, 1 01/03/14, 3.1, 3, 1 01/10/15, 3.1, 3, 1 To: Joe 01/02/09, 3.2, 3, 1 01/02/12, 3.1, 3, 1

  13. Current world of biodata sharing in NZ • Data sitting in institutional or even private archives. • Data (existence) is generally not published. • Data is not discoverable. • Data is not reconcilable. • Lots of effort ($$) in “pulling data together” Data source a

  14. Operations of the biodata services stack Standards BSS Guidelines National tools (e.g. vocabulary services)

  15. A “Common Biodata Infrastructure” CBI Synonyms: “Biodata Services Stack” …

  16. Federated bio observations data systemWhat do we need? Use of (in all ‘linked’ systems): • Consistent geospatial / temporal context • Consistent meta data (ANZLIC, ISO) • Consistent taxa data (NZOR) _____________________________ These are harder… • Consistent environmental feature vocabulary?(e.g. river names, lake names, …) • Consistent (institutional?) methods vocabulary(use (national) standards as reference?) • Consistent observations types vocabulary?(units vocabulary, parameter vocabulary)

  17. A “Common Biodata Infrastructure” CBI Users: DOC, MfE, MPI, Primary Sector, Stats NZ, Regional Councils, CRIs, NGOs Uses: States/Trends (SoE), Tier 1 statistics, Research, ??? Information Feedback (improved models, etc) Uses determine specification Use-Cases / Why do we need it? CBI Can include: Reporting Tools Products / Analytical Tools Standards Data Management Tools Monitoring Tools Data Providers: CRIs, DOC, Regional Councils, Te Papa, NGOs, Citizens

  18. A “Common Biodata Infrastructure” CBI • We are dealing with Sampling data – need vocabularies to ensure consistencySampling data types: • Survey Data (Systematic, well-defined scope)geo-chrono-taxa mapping • Ad-hoc data: Collections, Citizen Science • Vocabularies, e.g. Taxonomies (e.g. NZOR, soil classification) • Includes metadata about entities in vocabularies (e.g. origin / occurrence) • Descriptive Information about entities within a vocabulary • Need to ensure contextual information can be integrated!E.g. climate, land cover etc. Information Types to be managed

  19. Use case: Regional Councils - HBRC Tabulation of ‘connected environmental parameters’derived through interpolation/aggregation (rules required!) x/y, date/time, parameter1, parameter 2, … Data Connection /Amalgamation Tabulations of parameters x/y, date/time, parameter 1 Tabulations of parameters x/y, date/time, parameter 2 Data Delivery Data Archiving Authoritative Data Archive 1 Authoritative Data Archive 2 Standardized datasets Standardized formats / templates Taxonomic vocabulary Data Homogenization Observation and Measurement Nomenclature Metadata Standards Data QA Survey 2 Survey x.. Survey 1 Data Collection

More Related