Trident Scientific Workflow For Neptune

1. TridentScientific Workflow For Neptune Roger S. Barga Architect, Technical Computing Group Microsoft Corporation

2. Trident Project Contributors • Monterrey Bay Aquarium Research Institute • Jim Bellingham • Yanwu Zhang • Mike Goding • University of Washington • Keith Grochow, Dept of Computer Science • Mark Stoermer, Dept. of Oceanography • Donald Averill, Dept. of Oceanography • Microsoft Technical Computing Group • Luciano Digiampietri, intern, UNICAMP Brazil • Nolan Li, intern, Johns Hopkins University • Roger Barga

3. Project Neptune North East Pacific Time-Series Undersea Networked Experiment The world’s first plate-scale undersea observatory

4. Project Neptune: Themes Engineering Challenges • Delivering power and internet into the ocean, uninterrupted operation, device lifetime, data processing and storage • Scientific Research • Plate tectonic processes • Regional ocean/climate dynamics • Gas hydrates, etc

5. Technical Challenges • Disparate, high volume data sets and streams • Several types of CTD devices, ROVs, AUVs • ADCP • ZAP (vertical and horizontal) • 8MPix moveable digital still camera • 3-way hydrophone array • HDTV camera • From raw data to useable data products • Data cleaning, analysis, regridding, interpolation • Support real time, on-demand visualization Scalar Complex Streaming

6. Technical Challenges Support a variety of users interacting with system • Community Easy access to regularly generated data products (model output, images, visualizations, etc.) • Researchers access to quasi-live or historical data through thin client (web browser), ability to both access data and create visualizations on demand, author both data analysis pipelines and visualizations remotely • PIs direct access to their own instrument for live access, add new instruments, introduce new analysis codes and algorithms into the system Never been done before in oceanography • Requirements not easily obtained

7. Principal Goals Of Trident Allow users to • Automate tedious data cleaning and analysis pipelines. • Explore and visualize data, regardless of source. • Compose, run and catalog experiments, save results. • A workflow starter kit, one that will allow users to easily extend Trident functionality. • Learn by exploring and visualizing ocean & model data. By… • Allowing experts to author custom workflow activities, but basic users aren’t forced to see the details. • Allow user access mostly through a web portal, one that is intuitive and requires nominal local resources.

8. A Quick Look At TridentScientific workflow workbench for oceanography Populate Windows WF with custom activities • Introduce gridded data structures; • Define basic operators (data transformations); • Implemented as custom activities; Introduce parameterized activities • Easier for users to design workflows • Tool to convert custom to parameterized activities Invoke and author workflows via web browser Persistent workflows, checkpoints (stop-revise-rerun)

9. Internal Data Structures for Spatiotemporal Data Oceanographic Data • Spatiotemporal information stored in CDF and NetCDF files, in various formats Internal data structures • ISTem3DCollection: A collection of spatiotemporal points. For each point there is an collection of objects to represent measured values; • HyperCube4DOfDoubles: A four dimensional hypercube (grid). For each point there is an array of doubles to represent measured values;

10. Custom Activities In Windows WF

11. Parameterized Activity

12. Converting Custom To Parameterize Activities The conversion is made automatically: Custom Activity Parameterized Activity (web accessible)

13. Converting Custom To Parameterize Activities Select Custom Activities

14. Converting Custom To Parameterize ActivitiesParameterized , and now web accessible

15. This tool is designed to assist users with workflow creation and execution in the described system. It rovides a simple point and click interface and does not require users to write any code or XML. Remote Authoring Via Web Browser

16. demo Trident Scientific Workflow For Neptune Luciano Digiampietri (UNICAMP), Nolan Li (JHU) Interns, Summer 2007 Technical Computing Group at Microsoft

18. To Sum Up Trident • is a very young project (3 months old) • growing by application pull from its contributors • most features are now being designed jointly • Is not quite an alpha release but deployed at both UW and MBARI The screen shots were the baby pictures! There is a lot more to cover and work on…

19. © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Microsoft Research Faculty Summit 2007