1 / 42

NMFS Enterprise Data Management

NMFS Enterprise Data Management. DAARWG Meeting. December 8, 2010, 2010 Jim Sargent, NOAA Fisheries Information Architect. The End in Mind: Take Aways. Need to make quality data available to the world It will take a cultural shift - Which is HARD! NMFS EDM: One LO’s approach

lara
Download Presentation

NMFS Enterprise Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NMFS Enterprise Data Management DAARWG Meeting December 8, 2010, 2010 Jim Sargent, NOAA Fisheries Information Architect

  2. The End in Mind: Take Aways • Need to make quality data available to the world • It will take a cultural shift - Which is HARD! • NMFS EDM: One LO’s approach • Understand the comprehensive enterprise-wide process we have been through, and • Possibly leverage the fruits of our labors and learn from challenges • DAARWG recommendations: • Expand scope beyond Access and Archive Requirements to full Data Management Life Cycle • Promote and support EDMC and LO data management efforts • Leverage similarities while respecting diversity of data • Review and support EDMC’s Procedural Directives • Consider: • Adopting a new vision of data management • Doing a comprehensive inventory

  3. Outline • A Case for a New Vision for Data and Data Management • One LO’s approach: NMFS EDM • Recommendations

  4. Deep Water Horizon Data Management Challenges Immediate Response • Need to quickly identify applicable / available data Pre and Post Disaster • Volume and diversity of data • Collected for one-use; potential for multi-use • Infrastructure needed to support multiple access mechanisms • Releasability of data (organizational vs. technical) • Coordination of collection and distribution • Comparison and utilization of modeling outputs and observations • Overall Need for: • Clear and consistent data management policy • Better data documentation (metadata) • An overarching response plan Prepared by Environmental Data Management Committee for NOAA Leadership

  5. It’s the Next Disaster, Do We Even Know What Data We Have?

  6. Flashback “NOAA is like a library without any card catalog • …or even bookshelves” ……..Dr. John A. Knauss, Former NOAA Administrator, 1986

  7. Fast Forward to 2010:We Need to Share Our Data • President's Directive on Open Government • Transparency, participation, and collaboration • Timely publication of quality information • It’s the right thing to do! • Greater than archive and preservation issues • Not all access is done from archives

  8. Everyone is Wresting With This • Other federal agencies • IWGDD, NSF, ICES, ISO • DM maturing as a discipline • DAMA DM Book of Knowledge (DMBOK) • Certified Data Management Professionals • H. R. 5037 April 15, 2010 • Requires Federal agencies to develop public access policies • NMFS Science Center Data Management Reviews

  9. It’s About a Cultural Shift • Need to move from the academic paradigm • Publish or Parish  Share or Parish • Science  Science Information • Science Quality  good enuf • Changing Culture is Hard • Tipping Point, Blink …. Malcolm Gladwell • Switch, Made to Stick … Phil and Dan Heath

  10. A New Vision?? NOAA data assets are recognized and managed as a core agency resource, on par with financial and human resources.

  11. One NOAA Line Office’s Approach…. NMFS Enterprise Data Management (EDM) Program

  12. Bottom up and Top Down Support... Top Down Leadership Science Board • Approvals and support • Policies; • Procedural directives • Guidelines • Best practices NMFS Staff NMFS Staff • Analysis & Research • Recommendations: • Policies • Procedural directives • Guidelines • Best practices • Implementation • Marketing FIMAC Behavior Change Takes Both Bottom Up

  13. Data Stewardship Teams Established • FIMAC Workshop • Teams’ Plans developed • Required Resources Identified • Recommendations presented to and accepted by LC • FIMAC Created Data Inventory Initiated Data Documentation Procedure Directive drafted and vetted Policy drafted and vetted Policy Enacted Jan 15, 2009 IA Position Filled • Teams Created • IMCs Created NMFS Data Catalog established A Brief History of EDM Time FIMC researched NMFS DM and developed EDM Recommendations Research, Analysis, and Recommendations Planning and Development of Teams Implementation 2008 2009 2010

  14. NMFS EDM MISSION: “To effect a cultural change in which all NMFS data are recognized and managed as a core agency resource, on par with financial and human resources.”

  15. NMFS EDM Vision NMFS customers can confidently find, access, and use our data

  16. NMFS EDM Vision NMFS customers can confidently find, access, and use our data • Internal and external constituents

  17. NMFS EDM Vision NMFS customers can confidently find, access, and use our data • Confidence in finding and trust in using our data

  18. NMFS EDM Vision • Using various portals, data.gov, geospatial.gov, etc) • Browse through an ordered hierarchies or taxonomonies • Search using: • Discipline specific key words (controlled vocabularies) • User tags (folksonomies) • Metadata include users’ comments • Minimize the number mouse clicks NMFS customers can confidently find, access, and use our data

  19. NMFS EDM Vision • Through confidentiality and security filters while using standard tools and formats NMFS customers can confidently find, access, and use our data

  20. NMFS EDM Vision • Download selected data with • sufficient documentation, including quality indicators and warnings to effectively and properly use and understand the data NMFS customers can confidently find, access, and use our data

  21. NMFS EDM Vision NMFS customers can confidently find, access, and use our data • All NMFS enterprise data we choose to share

  22. NMFS EDM Vision NMFS customers can confidently find, access, and use our data

  23. NMFS Enterprise Data And Information Management Policy Overview • A high level NMFS Policy Directive implemented by Operational Procedure Directives • Enacted by Eric Schwaab: June 2010 • Directed RAs, SDs, and ODs to send 2-3 people to NMFS Data Stewardship Workshop • General Policy: All data shall: • be visible, accessible and understandable to authorized users; • modeled, named and defined consistently across and within all NMFS programs; • have a standard set of metadata; and • be managed, controlled and shared by data stewards throughout the data management lifecycle. • be publicly available generally within one-year of its collection

  24. Procedural DirectivesPrinciples/Concepts • Developing Procedural Directives an iterative process • Data Documentation Procedural Directive: FMCs shall: • Develop their own plans for documenting and sharing data assets • Inventory and document data assets and tools in NMFs metadata repository, InPort • Measure metadata quality with a rubric • FY11 will be a year of learning and practicing how to document and share our data • Develop, use, and refine metadata standards • Populate InPort with discovery level metadata • Develop practice and refine procedures for sharing

  25. Data Stewardship Teams A Scientist, A Data Librarian A Data Manager

  26. NMFS Data Stewardship Workshop • To develop a shared understanding of data stewardship, empower the data stewardship teams to lead the way for implementing the Data Documentation Procedure Directive, to develop best practice based on the experience during the initial documentation and inventory tasks. • Discuss New Data Management Policies and procedures • Develop best practices • How to document our data • e.g., quality assessment • Spirals in data documentation • Rubrics and metrics • Q&A and Sidebar Communities

  27. Next Steps • Data Doc. PD approved – Dec 2010 • Data Inventory Complete – Dec 2010 • Data Doc. Implementation Plans Due- Feb 2011 • Data Stewardship Workshop – Apr 2011 • First cut Best Practices – Apr 2011 • Data Doc. Implementation Plans Finalized – May 2011 • Address Preservation and Terminology Socialize, Support, Promote, and Market

  28. Critical Success Factors • Management Commitment • Dedicated, Personally Committed Team • FIMAC • Coordination Team • EDM Partners • Steady Hand at the Helm • Executive Sponsorship • Drive Up and Then Drive Down • Harness the mavens • Socialize, Support, Promote, and Market

  29. Challenges • Tipping Point almost reached but not there yet • Budget Shortfall • Perception that field not fully bought into it • Overworked Teams a high risk

  30. DAARWG Recommendations • Expand scope beyond Access and Archive Requirements to full Data Management life Cycle • Support cultural change needed to meet demand for quality data • Promote and Support EDMC and LO DM Efforts • Review and Support Procedural Directives • Leverage similarities while understanding diversity • Establish good data management practices for the whole data management lifecycle • Consider Recommending • A new Vision of Data Management • Conducting a comprehensive inventory with defined metadata

  31. Happy Holidays!

  32. Backup Slides

  33. Data Sharing “Data should be made as widely and freely available as possible while safeguarding the privacy of participants, and protecting confidential and proprietary data” • Data shall be shared in data.gov, as appropriate, as a one-click data asset, a one-click product, or by using a software tool (e.g., FOSS) • Sufficient documentation to understand the data being shared must be published in InPort and referred to or provided with the data • Data Stewards decide what data is appropriate for sharing • Sharing confidential data must conform to Agency policy

  34. Timelines for Sharing Types of Data Does not include provisional, predecisional documents and preliminary analyses leading to final management actions

  35. Principles for Effective Environmental Data Management • Data should be archived and accessible • Adequate resources for end-to-end management • Management activities should involve users • Interagency and international partnerships • Metadata are essential • Expert stewards required for management • Process to decide what data to archive • Archive must support discovery, access, and integration • Effective management requires a formal, ongoing planning process National Research Council Committee on Archiving and Accessing Environmental and Geospatial Data at NOAA, 2007 Prepared by Environmental Data Management Committee for NOAA Leadership

  36. Issues Identification Interestingly, the lowest ranking issue was that NMFS did not have buy-in across FMCs for addressing IM Identifying the Problem The FIMC identified 12 key issues and interviewed their management to determine their priorities • Top priority Issues • Critical gaps • No authoritative data inventory • Insufficient metadata • Data quality and consistency challenges • Ability to integrate data • Administrative systems • Other issues • Data being lost • Data not being archived for perpetuity • Historical data that need rescuing • Communications re: applications and IT return

  37. Issues Identification Interestingly, the lowest ranking issue was that NMFS did not have buy-in across FMCs for addressing IM Identifying the Problem The FIMC identified 12 key issues and interviewed their management to determine their priorities • Top priority Issues • Critical gaps • No authoritative data inventory • Insufficient metadata • Data quality and consistency challenges • Ability to integrate data • Administrative systems • Other issues • Data being lost • Data not being archived for perpetuity • Historical data that need rescuing • Communications re: applications and IT return

  38. NOAA’s Conceptual Framework

  39. DAMA-DMBOK Functional Framework

More Related