EarthCube Workflows Community Group Roadmap Overview

1. http://earthcube.ning.com/group/workflow EarthCubeWorkflows Community GroupRoadmap Overview December 7, 2012 AGU Fall Meeting Funded by NSF with grant EAR-1238216

2. http://earthcube.ning.com/ EarthCube Community Groups Workflows Semantics Data Governance 1,000 participants since Sept 2011

3. Why a Community Group on Workflow • Many capabilities identified by the community in the initial discussions in EarthCube concern workflows: • Workflow Discovery • Workflow Metadata and Provenance • Workflow Execution Management • Data Management within Workflows

4. Steering Committee Geoscientists: • Aaron Braecklel, NCAR • Ibrahim Demir, U. Iowa • Chris Duffy, Penn. State U.   • Gerry Wiener, NCAR Computer scientists: • EwaDeelman, USC/ISI • Yolanda Gil, USC/ISI • Suresh Marru, Indiana U. • Marlon Pierce, Indiana U.

5. http://sites.google.com/site/earthcubeworkflow/http://earthcube.ning.comhttp://sites.google.com/site/earthcubeworkflow/http://earthcube.ning.com Web site, report, EOS letter, etc. Roadmap Workflow primer, glossary Workflow use paradigms Workflow “vignettes” Workflow questionnaire Virtual meetings F2F Workshops

6. Workflow Use Paradigms“Designing a Roadmap for Geoscience Workflows,” Eos Transactions of the American Geophysical Union, 93(24), 2012 • The Center Modeling Paradigm • The Long Tail Paradigm • The High Performance Computing Paradigm • The Whiteboard Paradigm • The Metadata-Rich Paradigm • The Software Marketplace Paradigm Workflow Vignettes • Aviation turbulence forecasting • Wind/Power prediction • Flood maps and forecasting • Water quality • Water metabolism • Mesoscale atmospheric events • Seasonal forecasts of snowmelt • Groundwater management • Landform age calculation • Real-time science mission situational awareness • Field project data services support • Improving crop selection • River modeling at the regional scale

7. Workflows Working Group Workflows Synthesis Center EarthCube Community guidance and priorities Overview Of Proposed Workflows Roadmap needs and assessments Steering Committee Synthesis Center Co-Directors Establish Organization in Months 1-5, with WG Task Forces supported by clusters of Center staff assigned to them, and with 3 concentration thrusts EarthCube Governance EarthCube Working Groups T+6 Phase I 12 months Awareness Thrust Outreach, Identify opportunities, Tech base, Liaisons, Evangelize Communications Task Force Communications Cluster EarthCube Scientists Community Interaction Task Force Community Interaction Cluster Independent Investigators Geoscience Observatories and large projects T+18 Status & Requirements Task Force Status and Requirements Cluster Phase II 18 months Enabling Thrust Learn requirements & practice, Link scientists and technologists EarthCube Workflows Community Group (17 August 2012) EarthCube Infrastructure Development Engagement Cluster Engagement Task Force Geoscience Organizations (ESIP, AGU, etc) T+36 Phase III 24 months Infrastructure Thrust Exemplar prototypes, Impact assessment, Sharing and integration Prototyping Cluster Prototyping Task Force Other Relevant Organizations (OGC, W3C, etc) Assessment Cluster Assessment Task Force

8. Roadmap Outline Roadmap sections request by NSF: • Purpose • Communication • Challenges • Requirements • Status • Solutions • Process • Timeline • Management • Risk EarthCube Workflow Group Steering Committee Communications Task Force Status & Requirements Task Force Engagement Task Force Prototyping Task Force Assessment Task Force Community Interaction Task Force

9. Initial Workhttp://sites.google.com/site/earthcubeworkflow/ Web site, report, EOS letter, etc. Communications Task Force EarthCube WG Steering Committee Status & Requirements Task Force Workflow primer, glossary Workflow use paradigms Engagement Task Force Workflow “vignettes” Prototyping Task Force Workflow questionnaire Assessment Task Force Virtual meetings Community Interaction Task Force F2F Workshops

10. 1. Purpose • The EarthCube Workflows Community Group’s goal is to constitute a broad community within the geosciences that will identify both short-term and long-term challenges for scientific workflows • Aspects of this goal include better education and outreach, better understanding of the different types of workflows, better collaboration between workflow software developers and geoscientists, the identification of gaps, and the identification of grand challenges that workflow technologies do not currently address • Addressing this goal is the central theme of the roadmap. The resulting workflow roadmap will be considered a living document that will be extended and updated as future needs evolve

11. 2. Communication • Lack of awareness by geoscience researchers about workflows • Lack of awareness of workflow researchers about geosciences uses of workflows • Communications TF will help in both directions • So far: web site, workflow paradigms, workflow vignettes, workflow primer, glossary, EOS letter, etc.

12. 3. Challenges • Technical Challenges • Ever increasing complexity of data sets, computational needs • Ever increasing collaborative science processes • A number of various workflow systems available • Broaden Adoption • Limited adoption/lack of understanding of workflow technologies in geosciences • Limited interactions/engagement between geoscientists and workflow system providers • Need to provide guidance to scientists in finding an appropriate workflow system • Reproducibility • Rapidly evolving technologies and infrastructure

13. 4. Requirements • A Status and Requirements Task Forcewill • Gather requirements from users • So far: Workflows Community Questionnaire • A Prototyping Task Forcewill • Select target opportunities and leverage them as reference uses of workflows • Illustrate research challenges, measure baseline • Measure impact of workflows in context

14. 5. Status A Status and Requirements Task Forcewill: • Engage users at the science-focused EarthCube Workshops • Assess the current state-of-the-art in workflow technologies in various domains • Evaluate the workflow technology systems available in the sciences and the commercial sector and assess their uses • Assess the trends and distill major shifts in technologies related to workflows • Disseminate geoscienceworkflow needs and state-of-the-art workflow technologies • So far: Workshops, virtual meetings, Workflows Community Questionnaire

15. 6. Solutions • Issues: • Highly diverse workflow approaches which are dependent on many factors • Requires alignment with goals of geoscience teams • To address issues: Engagement Task Force • Process to identify approaches: • Organization-specific Workflows Maturity Model. • Process to identify technology solution: • Scenario-specific Technology Evaluation Framework. • Supported by "Status and Requirements TF" and "Prototyping TF". • Dissemination of expertise in workflow solution approaches. • So far: collection of “workflow vignettes”

16. 7. Process • Issues: • Partnering with any existing activities, projects, organizations exist • Many existing standards organizations and de facto standards. • Need to partner & integrate with other EarthCube groups. • To address issues: Community Interaction Task Force • Engage existing activities and groups in geosciences, leverage for EarthCube goals • Document existing standards and recommend appropriate standards to EarthCube • Documenting current and potential geoscience workflow use cases • So far: Workshops, invited presentations, initial contact with groups

17. 8. Timeline • Timeline strategy: • Activity prioritization and milestones targeted towards addressing the major issues early on. • Activities categorized into 3 thrust areas: • Awareness Thrust: information gathering • Enabling Thrust: prototyping and dissemination • Infrastructure & Services Thrust: workflow services for the community • Timeline Overview: • Months 1-5: Establishment Phase • Steering Committee and basicl infrastructure: 1 month • Task Forces: established within 3 months • Synthesis Center: Initial operations within 5 months • Months 6-18: The Awareness thrust will be emphasized • Months 18-36: The Enabling thrust will be emphasized • Months 37-60: The Infrastructure and Services thrust will be emphasized

18. 9. Management Communications Task Force WG Steering Committee Workflows Synthesis Center Status & Requirements Task Force Awareness Thrust Engagement Task Force Enabling Thrust Prototyping Task Force Assessment Task Force Infrastructure & Services Thrust Community Interaction Task Force

19. 10. Risk • Not capturing user requirements appropriately • Inadequate communication with user community • Lack of adoption • Choice of inappropriate software methodologies

20. Workflows Working Group Workflows Synthesis Center EarthCube Community guidance and priorities Overview Of Proposed Workflows Roadmap needs and assessments Steering Committee Synthesis Center Co-Directors Establish Organization in Months 1-5, with WG Task Forces supported by clusters of Center staff assigned to them, and with 3 concentration thrusts EarthCube Governance EarthCube Working Groups T+6 Phase I 12 months Awareness Thrust Outreach, Identify opportunities, Tech base, Liaisons, Evangelize Communications Task Force Communications Cluster EarthCube Scientists Community Interaction Task Force Community Interaction Cluster Independent Investigators Geoscience Observatories and large projects T+18 Status & Requirements Task Force Status and Requirements Cluster Phase II 18 months Enabling Thrust Learn requirements & practice, Link scientists and technologists EarthCube Workflows Community Group (17 August 2012) EarthCube Infrastructure Development Engagement Cluster Engagement Task Force Geoscience Organizations (ESIP, AGU, etc) T+36 Phase III 24 months Infrastructure Thrust Exemplar prototypes, Impact assessment, Sharing and integration Prototyping Cluster Prototyping Task Force Other Relevant Organizations (OGC, W3C, etc) Assessment Cluster Assessment Task Force