Karma Provenance: Why and How? Provenance collection of unmanaged workflows PI: Dr. Beth Plale

1. Karma Provenance: Why and How? Provenance collection of unmanaged workflows PI: Dr. Beth Plale Presenter: Dr. Mehmet Aktas

2. Insert an eye-catching yet meaningful picture here. If possible, show the scale of an instrument. Instant Karma: Applying a Proven Provenance Tool to NASA’s AMSR-E Data Production Stream PI: Michael Goodman, NASA MSFC • Collect and disseminate provenance of AMSR-E (Advanced Microwave Scanning Radiometer – Earth Observing System) standard data products, initially focusing on Sea Ice • Improve the collection, preservation, utility and dissemination of provenance information within the NASA Earth Science community • Customize and integrate Karma, a proven provenance tool into NASA data production • Engage the Sea Ice science team and user community • Adhere to the Open Provenance Model (OPM) Title for the image or graphic • Evaluate current AMSR-E SIPS product generation 06/10 • Extend Karma provenance collection tools for SIPS 09/10 • Enhance Karma Provenance Browser interface 10/10 • Instrument AMSR-E Sea Ice production 12/10 • Evaluate with Sea Ice science team 03/11 • Introduce Provenance Browser to NSIDC DAAC 06/11 • Instrument AMSR-E Sea Ice production in Ops 09/11 • Evaluate with AMSR-E Sea Ice user community 02/12 • Instrument other AMSR-E data streams 02/12 • Apply Karma to Sea Ice data production workflows • Customize Karma’s provenance dissemination user interface • Expand use of Karma to other AMSR-E data production streams Thorsten Markus, NASA GSFC; Beth Plale, Indiana University; Helen Conover, UAHuntsville

3. Objectives • Efficient and lightweight tools that support provenance collection, representation, and use • Our focus is on • Unmanaged workflows • With no assumption of global state • Collection interoperability

4. Karma logical architecture

5. Attributes of Karma (v 3.1.1) • Is modular and programmable • Supports diverse workflow architectures that compose of web services, java classes, message bus listeners • Implements Axis 2 Web Service based Query API • Captures provenance in streaming workflows • No need to know workflow structure in advance • Supports interoperability • Implements Open Provenance Model (OPM) v1.1* to represent provenance graph (access interoperability) • OPM enables provenance information exchange with other OPM-compliant tools * http://eprints.ecs.soton.ac.uk/16148/1/opm-v1.01.pdf

6. Provenance capture example • Script S1 invokes Applic A1, A2, .. Am once • Want to collect provenance about S1, A1, A2, …, Am • Instrument S1 and A1 with sensors, for Applics A2-Am, use Adaptor to parse log files. All provenance creation tools send provenance events to Prov Sys P1

7. Provenance Capture using NetKarma Node 10 Node 1 Node 0 NetKarma Provenance Collection System Partition graph … … Broker BFS driver M M M M M M RM GENI Adaptor R R R R R R PL Collect result from reduce VM Work Pool Work Pool GUSH Log …… VM VM PL PL M: Map R: Reduce Local Disk Local Disk PL: PlanetLab Node VM: Virtual Machine RM: Remote Machine (IN) PL VM RM

8. Twister + BFS Provenance Capture, Retrieval, Visualization Twister Single Run Log-file Twister Multiple Run Log-file Rule file VM VM VM Provenance Adaptor VM Notifications Queued Capture Web Service Message Bus RM RM Notifications Ingested Provenance Repository Karma Service RM RM Provenance Graph Retrieved Retrieve Query Client LM LM: Local Machine VM: Local Virtual Machine RM: Remote Machine (IN) LM VM RM Provenance Visualization Visualize LM

9. Karma Provenance Collection Tool • Open-source software is available at our project Websites: • Karma Project http://pti.iu.edu/d2i/provenance_karma • NetKarma Project http://www.pti.iu.edu/d2i/provenance_netkarma • NASA InstantKarma Project http://www.pti.iu.edu/d2i/provenance_instantkarma • Contact information: • Mehmet Aktas, PhD; maktas@cs.indiana.edu • Beth Plale, PhD; plale@cs.indiana.edu