Realizing the Life Science Grid with Taverna

1. Realizing the Life Science Grid with Taverna Tom Oinn, tmo@ebi.ac.uk

2. What’s out there right now?

3. GRID? In general a grid system is, or should be : “A collection of a resources able to act collaboratively in pursuit of an overall objective” A life science grid is therefore : “A collection of resources able to act collaboratively to solve a problem in the life science domain”

4. Life sciences • Massive diversity of • Information classes • Services • Data • Problems • Relatively small data sizes • Relatively small computational load • Challenge is complexity and heterogeneity • Much scientific work is exploratory • Environment must be flexible and easy to reconfigure • Environment must provide facilities for provenance capture

5. Current inventory? • Existing diverse services • Web based, SOAP services, custom protocols such as BioMoby etc. • Existing data resources • Relational, unstructured flat file, XML • May or may not be exposed through some kind of service interface i.e. SRS, BioMart • Existing user communities • Large well funded service and research projects with substantial IT support • Small groups with no IT support, little funding but interesting problems

6. Life scientists • Experts in their domain • Little or no experience with distributed computing • Most bioinformaticians are not computer scientists • Generally not supported by dedicated CS groups • Need to allow these users to make use of their existing expertise but remove concerns such as: • Parallelism • Distributed programming • Fault recovery • Job dispatch and submission • Provenance capture • Logging and auditing

7. Our life science grid : “A collection of existing legacy and novel tools and databases exposed through a variety of technologies able to act collaboratively to solve a problem posed by an ‘IT naïve’ user in the life science domain across the public internet and withlittle or no technological support and as inexpensively as possible.” Users typically have no control over services (provided by 3rd parties) so create a client side integration platform. Should be accessible to an unsupported PhD student with standard networking, a three year old PC and no dedicated IT support.

8. Taverna Workflow Workbench http://taverna.sf.net A ‘super client’ to a variety of disparate services on both intra-net and inter-net

9. For those with wi-fi and little patience... • Project homepage : http://taverna.sf.net • myGrid project page : http://www.mygrid.org.uk • OMII-UK home : http://www.omii.ac.uk • Alberto’s Taverna + EBI mini tutorial : http://www.ebi.ac.uk/Tools/webservices/tutorials/taverna

10. What is Taverna? • Taverna is : • A workflow language based on a dataflow model. • A graphical editing environment for that language. • An invocation system to run instances of that language on data supplied by a user of the system. • When you download it you get all this rolled into a single piece of desktop software • The enactor can be run independently of the GUI

11. Services • Taverna can interoperate the following by default : • SOAP based web services • Biomart data warehouses • Soaplab wrapped command line tools • BioMoby services and object constructors • Inline interpreted scripting (Java based) • Other service classes can be added through an extension point (but you probably don’t need to)

12. Biomart Support

13. Stateful SOAP service support • Add service to services list by pointing Taverna to Web Service Description Language (WSDL) document online • Taverna inspects WSDL, extracts operations • Add operations to workflow, right click to automatically add document builders and splitters for doc/literal style services • Use nested workflow to define polling logic, sub-workflow fails, waits and retries if data is not ready Document builders Service invocation (creates job) Polling loop (check status, fail if not ready) Get results *SOAP is the Simple Object Access Protocol - http://www.w3.org/TR/soap/ & http://www.w3.org/TR/wsdl

14. Soaplab Support Individual tool within category Soaplab server in services list Soaplab services support rich descriptive metadata • Soaplab services are added to the services palette by pointing Taverna at the root of the Soaplab installation. • Individual services within that server are categorized and displayed within categories • Services support polling and provide links to metadata directly within Taverna • http://www.ebi.ac.uk/Tools/webservices/soaplab/guide

15. Biomoby Support • BioMoby provides semantic description of services • Taverna can use this to assist in the service composition at design time • All this provided by the Moby team – Taverna’s extension architecture allows third party developers to contribute in a loosely coupled way

16. Taverna also provides: • Service discovery • Free text search over ‘known’ services. • Semantic search over service repository, relies on manual service annotation and submission of those annotations to the repository. • Provenance tracking • Lineage tracking of result data. • Automatic semantic annotation of data from service annotations. • Possible as the workflow engine creates a ‘managed environment’ with an overview of all data movement. • Result visualization • Common renderers included in base distribution include 3d structure, images, graph rendering • Extensibility • New service classes • New renderer types • New UI elements

17. OMII-UK & myGrid Management

18. Project management • Funded through the Open Middleware Infrastructure Institute (OMII-UK) as part of the myGrid project run by Carole Goble • Four years old, funding secured through 2008 and beyond. • Development team at Manchester & Hinxton, UK • Wide group of ‘friends and allies’ across the world particularly within UK eScience • Implemented in Java, released under LGPL licence.

19. OMII-UK (very quick overview!)

20. Lightweight transient grids

21. Looking forward – transient grids • Science varies widely in scale both in space (CPU cycles required, storage, numbers of services etc) and time (duration of collaborations, stability of VO membership) • Current grid infrastructure is focused on projects with large spatial and temporal scale • Does this existing work map well to scientific problems with different characteristics, especially different temporal characteristics? • What about security…?

22. Collaborative security delegation • A workflow can access multiple resources • These resources can have arbitrary security constraints • It is likely that a given workflow requires more than one principal to be available to complete. • How can we make multiple security agents available to the workflow engine in a principled fashion? • Define the basic unit of a virtual experiment or fast virtual organization to map directly to a peer group within a peer to peer framework • Peer group contains a workflow instance along with any resources required to enact that instance including arbitrarily many security agents, data stores, metadata stores etc. • Services accessed by the workflow may (and usually will) exist outside of the peer group.

23. Peer group example Workflow instance Security Agent (User B) Peer Group (Virtual Experiment) Data Manager External tools, data and services Security Agent (User A)

24. Virtual Experiment Construction • A Virtual Experiment (VE) is created by the construction of a new peer group within the P2P framework • Resources such as workflow engines, data managers and security agents exist as factory services. • Each factory can construct a limited version of itself • Workflow engines with specific workflow definitions loaded • Data managers with specific levels of storage space • Security agents with policies to restrict full use of credentials • These limited proxy objects connect to the peer group • This is a secured operation but as there is no delegation existing security mechanisms are adequate to get this far • Factories may be on the intranet or internet (most likely for workflow services) or on the user’s workstation, PDA or cellphone (for security agents).

25. Collaborative Virtual Experiments • A VE becomes collaborative when more than one user can access the objects within the peer group. • A VE uses collaborative security when more than one user inserts a security agent into the peer group. • Note that the peer group structure also allows multiple views on the same VE as objects can exist in more than one peer group. • For example, you could split the workflow instance into a monitoring and steering component and give some users access to a peer group containing both and others to one containing only the monitoring part. • The peer group has a unique identity which can be used to discover or register it with any registry service available.

26. Get involved!

27. Work with us! • Taverna2 under development, delivery by the end of 2007 • Rewrite of Taverna to support, amongst other things: • Integration with grid technologies through a set of new extensibility points • Transient VO management (short lived virtual organizations, 20 second upwards lifetime!) • More sophisticated computational model • Massive scalability, pipelining of nested token streams, single threaded execution model, transparent reference passing architecture • Monitoring and steering of running processes with arbitrary granularity through an extension point • Implement extensions to interface to your GRID • Get a free and well supported rich client portal for non expert users • Access otherwise out of reach user communities

28. How to contribute • If you have a grid with resources that our community could use • Talk to us, tell us about it • Write a plugin for its resource broker, data system or security model • If you have a scientific community who wants to access such resources • Again, please let us know • We can provide on site training • We are always interested in new application areas for our work • I can be contacted at tmo@ebi.ac.uk, or for more general discussion please join the mailing lists linked from http://taverna.sf.net