CLARIN tools for workflows

1. CLARIN tools for workflows Overview

2. Objective of this document • Determine which are the responsibilities of the different components of CLARIN workflows. • Explain what is expected from the CLARIN Workflow tools. Requirements.

3. Firstly, what is a workflow? • Workflow is a definition of a process in a certain language. It is done in a non-ambiguous way. • A workflow is only the definition of a process. • Since the workflow definition is made in a non-ambiguous way, any software can execute it expecting the same results. It means that workflows can be shared between different applications. • Workflow may be a XML file that a user can write without any other help.

4. Workflow tools • Since a workflow is only a process description, we need 2 other components. • Workflow Engine: Application that process (executes) workflow descriptions. • Workflow Editors: Application for creation of workflow descriptions. • Both applications can be in the same piece of software. Workflow Workbench.

5. Workflow Engines: Expectations • Engines are applications that execute workflows in the way they are described. • Many engines can be developed. Some with GUI for desktop computers some other hosted in servers with schedule features, etc...

6. Workflow Editor: Expectations • For advanced users of CLARIN, a text editor could be enough for a workflow creation. • But CLARIN objective is easiness. In this way to create a workflow in CLARIN must be really easy. • The editors in CLARIN must help as much as possible to the user. This is done, for instance, querying the registry and taking decisions behind the scenes. • Example: User connects two web services using first’s output with second’s input. Metadata stored in the registry specifies that this operation is not possible without a conversion. Also the registry contains info about converters available. The editor must be aware of that and introduce a conversion tool between the web services. This can be done without user’s interaction at all. • SMART editors are possible since the Registry will contain all the required metadata for taking decisions automatically.

7. Example • User wants to connect 2 web services. • Web service X • Web service Y • X’s output needs to be Y’s input. • But X’s output is not in the proper format for being Y’s input. • The workflow editor must be able to find a converters.

8. Example. Finding a converter • The editor needs to do the following tasks: • Get information about web service X. Which format has X’s output? • Get information about web service Y. Which format requires Y’s input? • Find a converter. How can data in format X be transformed in format Y? • All this information must be stored and accessible in the registry

9. Example. Get information of X • It is required to query the registry. Still it is not decided how this query will be but intuitively it should look like: • specObject = GetWebServiceSpecification(X_persistent_id) • This call will return the specification where explains that X’s output format follows the standard A.

10. Example. Get information of Y • Like previously with X, the editor queries the Registry for Y’s specification. • specObject = GetWebServiceSpecification(Y_persistent_id) • This call will return the specification where explains that Y’s input format follows the standard B.

11. Example. Finding a converter • Now the editor needs to find a converter from standard A to standard B. • It will be done using the Registry as well. Also it is not developed yet but it will be something similar to: • ResourceList = GetResourceList(“/standards/converters”) • The editor has downloaded the list of converters (resources under the standards/converters branch). Now inspecting this list of resource descriptions, the editor will find the required converter (if any).

12. Example. Introduce the converter in the workflow • Before the insertion of the converter in the workflow, depending on the user’s profile: • the user will be asked for confirmation or • the user will be noticed of different converters if more than one is available or • simply won’t be notified at all (novice users) • The editor introduces the converter in the workflow between X and Y. • The workflow description will keep this converter information. Engines executing this workflow will need to know about this conversion.

13. Workflow editor: Requirements • Graphical representation of all workflow description elements. (loops, if then clauses, parallelization, exceptions, etc…) • SMART editor. • Different user profiles (Advanced users will want to know more about automatic decisions taken by the workbench while novice users won’t want to decide anything) • Drag & Drop features. • Registry browsing

14. Workflow workbench. • The editor can help even more to the user if it is aware of the workflow execution results. That’s why it is recommended to develop a Workbench (Workflow Editor + Workflow Engine together). • Also for basic users it is easier when everything is centralized in just one application.

15. Workflow Workbench: Requirements • All Editor requirements • Debugging tools: breakpoints, Start-pause execution and inspection of intermediate steps. • Partial workflow execution and reuse of data obtained in previous executions. (To avoid repeating calls to web services while testing) • Output console

16. Very important. SMART applications • Different profiles in CLARIN applications. Don’t bother novice users with advanced questions. • Applications remember similar situations in the past. “You connected two web services similar to this one using converter AtoB. Do you want to do it again?” • Use of community intelligence stored in the registry. “250 users connected these 2 web services in other projects using converter AtoB, do you want to do it as well?”