Soaplab and the EBI Soaplab Web Services

1. Mahmut Uludag November 2008 Soaplab and the EBI Soaplab Web Services

2. Soaplab2 – more reliable Sesame door to bioinformatics programs What is it? Web Services framework for command line programs/scripts Avoid standard low-level details of Web Services programming Server side ACD files to describe web services Java properties files to configure the server Rare cases: wrappers, adaptors Client side Taverna Soaplab plugin Soaplab Client library & ready to use scripts Programmatic access Use Soaplab Client library, in your Java programs Use any other Web Services client library

3. Soaplab2 – more reliable Sesame door to bioinformatics programs How different than other frameworks? Based on other frameworks (JAX-WS, Axis) Some similarity with Spring Web Services framework Common/generic interface for all programs AnalysisList interface (Soaplab web application level) Query list of available programs and program groups Get short program descriptions Analysis interface (for each program installed) Get detailed program description, input/output specifications Start, terminate jobs Wait started jobs, or start jobs in a blocking way Query job status Retrieve all or partial results Service description using ACD (transformed into XML before used)

4. Soaplab2 – more reliable Sesame door to bioinformatics programs Is it EMBRACE compliant? • Soaplab-1 used RPC-encoded binding style as in many early Web Services • Soaplab-2 is more modular in design • Allows different protocol implementations • Allows even different protocols • SOAP protocol implemented using JAX-WS and Axis-1 • JAX-WS implementation use document/literal wrapped binding style, WS-I compliance tests pass • Axis-1 implementation use RPC-encoded style, supports existing clients • Generic interface • Inputs/outputs are not fully described in WSDL • WSDL accommodates methods to query inputs/outputs

5. Simple example with no program output appl: Sleep [ documentation: "Just sleep for a while" groups: "testing" nonemboss: "Y" executable: "sleep" ] integer: interval [ additional: "Y" parameter: "Y" default: 30 comment: defaults prompt: "Sleeping interval in seconds" ] service is represented by the command-line program sleep optional parameter always puts the value of this parameter on the command line even if the value is the default one Soaplab2 – more reliable Sesame door to bioinformatics programs

6. Helloworld service ACD description appl: HelloWorld [ documentation: "Classic greeting" groups: "Classic, Simple" nonemboss: "Y" executable: "echo" ] string: greeting [ additional: "Y" parameter: "Y" default: "Hello World" comment: "defaults" ] outfile: output [ default: "stdout" ] service is represented by the command-line program echo service output (the greeting) is produced on the standard output of the echo program Soaplab2 – more reliable Sesame door to bioinformatics programs

7. Example server side configuration file base.dir = /apache-tomcat-6/webapps/soaplab2_emboss6 metadata.dir = ${base.dir}/metadata applist = ${metadata.dir}/EBIApplications.xml runtime.dir = /aaa/soaplab-work/soaplab2_emboss6 working.dir = ${runtime.dir}/SANDBOX results.dir = ${runtime.dir}/RESULTS #timeout after one hour jobs.timeout = 3600000 groupA.mylongrunningprogram.jobs.timeout=7500000 jobs.cleaning.interval = 15000 #services.cleaning.interval = 300000 ###classic.helloworld.metadata.file = a/b/c #accept.any.exitcode groupB.mylegacyprogram.accept.any.exitcode Soaplab2 – more reliable Sesame door to bioinformatics programs

8. Example server side configuration file - cont. ### Specifically for those who are using EMBOSS emboss.home = /aaa/bin/EMBOSS-5.0.0/ emboss.data = /aaa/bin/EMBOSS-5.0.0/share/EMBOSS/data/ addtopath.dir = /aaa/bin/clustalw/ addtopath.dir = /aaa/bin/hmmer-2.3.2/binaries env.PATH = /usr/local/bin:/usr/bin:/bin:/aaa/bin/EMBOSS-5.0.0/bin env.LD_LIBRARY_PATH = /sw/arch/lib env.PLPLOT_LIB = /aaa/bin/EMBOSS-5.0.0/share/EMBOSS # applist = ${metadata.dir}/EMBOSSApplications.xml Soaplab2 – more reliable Sesame door to bioinformatics programs

9. Example server side configuration file - cont. # Accessing results by URLs: # -------------------------- results.url = http://localhost:8080/soaplab2/results results.url.target.dir = /path/to/local/directory/visible/to/your/web/server #results.url.ignore Soaplab2 – more reliable Sesame door to bioinformatics programs

10. Example client side configuration file base.dir = /work2/eclipse-workspace/soaplab2 # ------------------------------ # where to find service/services # ------------------------------ host = localhost port = 8080 # ------------------------------ # how to access service/services # ------------------------------ # protocol can be 'jaxws' or 'local' or 'axis1' (so far)‏ protocol = jaxws Soaplab2 – more reliable Sesame door to bioinformatics programs

11. Example client side configuration file - cont. # ------------------------------ # properties for batch testing # ------------------------------ test.data.dir = ${base.dir}/data batch.test.file = ${base.dir}/data/defaultTests.cfg #batch.test.file = ${base.dir}/data/embossTests.cfg #batch.test.file = ${base.dir}/data/embassyTests.cfg #batch.test.file = ${base.dir}/data/ebiServicesTests.cfg #batch.keep.results batch.report.table #batch.max.threads = 25 Soaplab2 – more reliable Sesame door to bioinformatics programs

12. Soaplab2 – more reliable Sesame door to bioinformatics programs New in Soaplab2- for service providers Dependency on an additional CORBA server removed Choice of SOAP protocol implementations JAX-WS, Axis Local protocol Good for testing, also used by Spinet Rich set of ant tasks, and configuration parameters Platform independence More Java (was using Perl launchers before) Third party libraries downloaded using Maven libraries during initial installation

13. Soaplab2 – more reliable Sesame door to bioinformatics programs New in Soaplab2- for clients Reasonable level of backward compatibility RPC/Encoded protocol supported using Axis More robust code More reliable server (cleaning threads for jobs and services) Spinet web interface Run deployed programs through usual HTML forms Taverna plugin version 2 Supports both Soaplab1 and Soaplab2 Still external to Taverna Better documentation

14. Soaplab2 – more reliable Sesame door to bioinformatics programs New in Soaplab2- for developers Easier extensibility with more plugin hook points Class based plugin support (refer them in ACD files) Customized jobs (such as EmbossJob) Input/output data adaptor Customized Parameter class SPI-based hook points (apache.commons.discovery) Persistent storage (such as a relational database) Event listeners and deserializers (from XML to Java objects) Format of service metadata Metadata listing services Enhanced client library Rich and easy to understand API Used in Bioinfo-GRID and Taverna plugins

15. Soaplab2 – more reliable Sesame door to bioinformatics programs Soaplab2 XML (analysis description)‏ XML (analysis description)‏ ACD (analysis description)‏ Taverna XML (analysis description)‏ XML (analysis description)‏ XML (analysis Description)‏ Programmatic access Tomcat (or any servlet engine)‏ Protocol Toolkit (JAX-WS, Axis1, REST, local)‏ Soaplab2 (+ plug-ins)‏ Soaplab2 ready-to-use clients and Java classes for developers acd2xml (convertor)‏ MySQL with results Filesystem with analysis results happy users Command-line driven analysis tools (EMBOSS,…)‏ Remote web resources (web pages)‏ Tools available on computational grids (BioinfoGRID)‏

16. Soaplab2 – more reliable Sesame door to bioinformatics programs Releases and recent work • First Soaplab-2 beta release- October 2007 • First release- February 2008 • Working with most EMBOSS (and EMBASSY) programs • Tested using two test suites derived from EMBOSS QA tests • Jobs timeout (automatic cleaning of hanging jobs) • Around 30 downloads a month • Batch-client • Check server response to concurrent requests • Learn all your services working properly

17. Soaplab2 – more reliable Sesame door to bioinformatics programs Spinet- main page

18. Soaplab2 – more reliable Sesame door to bioinformatics programs Spinet- lets you use Soaplab services

19. Soaplab2 – more reliable Sesame door to bioinformatics programs Data types and typed services • Input/Output through SoaplapMap (which has 4 data types)‏ • String • String array • Byte array (e.g. png graph file)‏ • Byte array array (e.g. list of png files)‏ • Typed services not yet available • Peter Ernst from DKFZ has developed one for Soaplab1, later integrated to Soaplab2 • Example implementation using JAXB and JAX-WS Provider interface • Plans made to integrate with Soaplab2 • Strongly typed web-services • Requires XML input/output support from the underlying programs • EMBOSS, DAS-GFF

20. Soaplab2 – more reliable Sesame door to bioinformatics programs Future work Scalability Generic support for dispatching jobs to queuing systems (LSF dispatcher) Basic queuing support for synchronous concurrent jobs

21. Soaplab2 – more reliable Sesame door to bioinformatics programs EBI Soaplab Web Services • Two public Soaplab-2 web applications • EMBOSS-6.0.1 • EMBOSS-4.1.0 • Includes WS-I compliant, typed services • Includes wrapper Web Services for few EBI Web Services • Limitation on the maximum number of files in the Soaplab run-time directories • All job results removed after one hour • Long running or hanging jobs terminated after one hour • A hierarchical directory structure needed, for example by using a new subdirectory every hour

22. Soaplab2 – more reliable Sesame door to bioinformatics programs Acknowledgements • Martin Senger