120 likes | 226 Views
UC DAVIS Department of Computer Science. Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life. The Kepler/pPOD Team Shawn Bowers , Timothy McPhillips, Sean Riddle, Manish Anand, and Bertram Ludäscher
E N D
UCDAVIS Department of Computer Science Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life The Kepler/pPOD Team Shawn Bowers, Timothy McPhillips, Sean Riddle, Manish Anand, and Bertram Ludäscher DAKS Lab, Genome Center, Univ. of California at DavisDept. of Computer Science, Univ. of California at Davis
Background “The AToL initiative (Assembling the Tree of Life) is a large research effort sponsored by the National Science Foundation. Its goal is to reconstruct the evolutionary origins of all living things.” – http://atol.sdsc.edu AToL projects • Investigate relationships among specific groups of organisms • Develop new computational techniques • Expectation that projects will collaborate & share data Technology barriers • Exchanging data between collaborators & other projects • Data “lives” in many different kinds of applications • Similar analyses performed, but ad hoc (manually or scripts) • Provenance of data and results
Project Overview pPOD (processing phylodata) • Develop core database technologies for the AToL community • Data access, data integration, scientific analysis, provenance • Collaboration among Univ. of Pennsylvania, Yale Univ., Univ. of Florida, and UC Davis Kepler/pPOD @ UC Davis • Scientific workflows for phylogenetic data analysis • Workflow execution and data provenance
Existing Applications Tolkin Workflow Automation (Kepler/pPOD) mappings to core model (via Orchestra) TreeBASE Core AToL Data Model • Tools & analyses • Integrate w/ data model • Provenance recording withinand across workflow runs AToL Lab DB Data Integration & Exchange (Orchestra) • Data types for sequences, trees, … • Provenance relationships • Expressive query language (OQL) • Persistence tools • Application schema mappings • Curation (w/ provenance) • Privacy and trust policies • P2P support Basic architecture
Kepler/pPOD workflows Uses • Sequence alignment, tree inference, post-tree analysis, … • Track analyses run and data produced within projects • Use, test, compare different computational techniques Characteristics • Exploratory (design, run, modify, commit, …) • Intertwined with manual steps (e.g., edit alignment) • Many formats, few data types (sequences, trees, matrices, …) • Pipelined (e.g., multiple sets of sequences) Kepler/pPOD Status • “Preview release” of Kepler/pPOD: Kepler + pPOD extensions • workflow design (via Comad) • wrapped apps: Phylip, Clustal, MrBayes, RaXML, tree drawing, … • provenance recording and browsing
GUI components workspace extension access to workflows access to run “traces” new director data types, collections assembly-line processing provenance enabled actor library Cipres web services local applications format conversion Kepler/pPOD workflows
Kepler/pPOD workflows integrated provenance browser data & process dependencies “forward” & “rewind” run multiple views
Proj Seqs Trees Aligns Compute Consensus … … S1 S10 A2 T1 T5 T6 T5 T1 A2 A1 S10 S1 A1 T6 … … … … < < > < > < > > <Seq> <Proj> </Seq> </Proj> <Trees> </Trees> </Aligns> <Aligns> Comad: “Virtual Assembly Lines” • Actors select parts of token stream, forward rest • Special tokens denote collections, metadata, & parameters • Actors insert tokens into and remove tokens from stream • Some advantages of Comad • workflows with loops, branches, composition (subworkflows) • concurrency, pipelining • resilient to change (data nesting, add/remove actors) • simpler workflow designs
“Conventional” All of X and Y stored for A1 A1 X Y A1 “Comad” Store change and explicit dependenciesfor A1 … … … … del(A1) ins(A1) … but (efficiently) representing provenance? • Many approaches require storing all input and output for each actor invocation (transformers) • can lead to significant redundancy in Comad • We use an “XML-diff” approach augmented with data provenance • special provenance tokens … • … insertions, (marked) deletions, invocation dependencies • exploit collections and apply inference rules • only store final result containing input and provenance
Kepler/pPOD Provenance Browser • Reusable “widgets” for viewing different aspects of a trace • Move “forward” and “backward” through execution • Data dependencies, collection structure, actor invocations
Kepler/pPOD Provenance Browser • Collection and invocation view • Incrementally step through execution history • Actor invocation graph shows pipelining, implicit branches
Poster/Demo & Questions … • Please come to our poster/demo :-) • Preview release of Kepler/pPOD available • http://daks.ucdavis.edu/kepler-ppod • Ongoing and future work • Adding more actors for phylogenetic analyses • Extending with “project histories” • Incremental query support • Integrate with AToL Core Data Model