1 / 45

Triana

Triana. Dr Matthew Shields, School of Computer Science, Cardiff University. A Problem Solving Environment and Grid Workflow Tool. Course Outline. Introduction History and Overview Tools and Toolboxes, Components and Groups How to compose a workflow Triana as a workflow editor.

JasminFlorian
Download Presentation

Triana

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Triana Dr Matthew Shields, School of Computer Science, Cardiff University A Problem Solving Environment and Grid Workflow Tool

  2. Course Outline • Introduction • History and Overview • Tools and Toolboxes, Components and Groups • How to compose a workflow • Triana as a workflow editor

  3. Course Outline cont. • Service-oriented Components • Web services as components • Web service workflow • Peer 2 Peer services as components • Distributed service workflow • Grid-oriented Components • Grid file and job primitives as components • Complex Grid workflow

  4. Install Triana • Unpack the triana.tar.gz to a location of your choice • Set env $TRIANA (Unix) or %TRIANA% (Windows) to the unpacked directory • Run $TRIANA/bin/triana (Unix) or %TRIANA/bin/triana.bat (Windows) • Course website http://www.trianacode.org/coregrid/

  5. About Triana • Distributed Problem Solving Environment • Workflow Based • Multiple Component Domains • Signal Processing (original domain) • Image Processing • Text Processing • Audio • Composite Components

  6. Our Research Motivation • Based on research within the Triana project: • Triana - a vehicle for investigating research into distributed systems • a workflow-based Problem Solving Environment • Goals: • To make the access of distributed services as seamless as possible • To connect heterogeneous Grids • To abstract the core capabilities needed for service-based computing (in P2P, Web services or Grid Computing)

  7. Research Scope • Triana is currently used in many projects • GridOneD - Radio Astronomy • DART - Distributed Audio Retrieval with Triana • WHiP - Workflows Hosted in Portals • Data-Mining Grid • BioDiversity World • See http://www.trianacode.org/collaborations

  8. Components and Definitions • Component is unit of execution • Components are defined in XML files: • Naming information • Input and output ports • Parameter information • Why Components? • To simplify the application design process and to speed up application development

  9. Triana Workflow • Triana is inherently flow based • Data flow - data arriving at component triggers execution • Control flow - control commands trigger execution • Decentralised execution • Data or Control messages sent along communication “pipes” from sender to receiver causes receiver to execute • Synchronous or Asynchronous messaging (Implementation dependant) • Multiple inputs can block or trigger immediately (Component designer defined)

  10. Taskgraph • Internal object based workflow graph representation • Taskgraph - DAG • Tasks • Connections • External XML representation • Simple XML syntax • List of participating Task definitions • Parent/Child connection • Hierarchical (Compound components)

  11. Simple Workflow Example Try It! $TRIANA/bin/triana

  12. Example Continued • Image is converted to Grey scale and then thresholded (set image less than 136 to black and everything higher to white). • difference is then taken between this image and a image where the white lines are thinned out (i.e. detail is taken away). • The resulting image is basically the detail of the white parts which were pruned by the ShrinkWhite unit which gives an outline of an image. • Group the unit and save it as a new compound component

  13. Coalescing Binary Search GEO 600 Coalescing Binary Search Algorithm implemented as a Triana workflow

  14. Grid services Triana, the GAT and the GAP Service Based Computing: Grid Computing: Deployment, discovery and communication with distributed services e.g. P2P and (GSI) Web services Job Submission, File services A Graphical Grid Computing Environment or Portal GAP Interface GAT Interface P2PS JXTA Web Services Condor Unicore GridFTP GRMS WSRF Globus RLS PBS .NET GridLab P2PS Discovery UDDI JXTA Discovery SOAP P2PS Pipes SSH SGE LDR Other.. JXTA Pipes

  15. GAP Interface P2PS JXTA Web Services P2PS Discovery UDDI JXTA Discovery SOAP P2PS Pipes JXTA Pipes GAP Interface • Motivation by GAT (GridLab.org) • A Simple Service based API, for • Service Deployment, • Service Discovery • Pipe Based Communication • Static application interface with multiple middleware bindings • P2PS (name…?) • JXTA • Web services

  16. Service Discovery Dynamic? Decentralized? Communication Message Format SOAP? Transport Protocol TCP? UDP? Service Oriented Comms en_fr hello network bonjour BabelFish GAP babelfish. altavista. com

  17. GAP Interface • Simple Java API • Peer-to-Peer Discovery • Pipe-Based Communication • Multiple Bindings • JXTA • P2PS • Web Services (WSPeer)

  18. P2P GAP Bindings • Two Main Bindings • Jxta • legacy implementation, due to upgrade soon… • P2PS • Simple (lightweight) P2P System • Uses in Triana, NRL (sensor nets) and Simulations (within NS-2) • Useful in highly dynamic environments • Based on some Jxta principles i.e. pipes, rendezvous nodes, dynamic discovery, etc • Capable of building small-world networks (centralised-decentralised)

  19. Discovery Service Broadcast/Locate Adverts + Queries in Discovery Subnet Pipe Service Connect Pipes using Endpoint Resolvers Rendezvous Service Send/Receive Adverts + Queries from known Rendezvous Peers Resolvers Pluggable Network Interfaces For Ns2 simulation environment P2PS Overview P2PS Peer Discovery Service Rendezvous Service Pipe Service UDP Resolver TCP Resolver ??? Resolver UDP TCP ??? Network

  20. Service Creation, Discovery, and invocation • Drag a Audio.Output.Play unit to a new scratchpad • Right click on the Play unit and select create service • Select Local Service [P2PS] and press Create • Leave the default settings and click ok • Use menu File -> New to create a new scratchpad • Select menu Services -> Discover Services • Type Play in the search name panel and click ok • You should see a number of remote Play units • Drag one of the remote Play units to the new scratch pad • Drag a Audio.Input.LoadSound unit and connect to the play • Download a sound file from the tutorial site http://www.trianacode.org/coregrid/ • Double click the LoadSound unit or right click and select properties, choose the sound file, click ok, click the play button.

  21. Service Deployment • Peer-to-peer services • Distributing Audio !!! • Generic example - can try and distribute using other networks • Uses multicast or rendezvous to discover services • Shows the basic interface to distribution • Can publish compound/group componnents • Publish as Web services • Need a running UDDI server for service publication and discovery

  22. WSPeer • High Level Interface to Web Services • Discovery • Invocation • Deployment • Hosting • Abstract from usual Web Service Discovery and Communication Mechanisms (i.e. UDDI and HTTP) • P2PS Web Service Discovery? • Uses Apache AXIS as SOAP Engine • Extends Capabilities of Apache AXIS • Stubless Invocation (including complex types) • Non Standard Transports (i.e. P2PS)

  23. locate publish publish locate deploy deploy invoke invoke launch server UDDI HTTP Server WSPeer Application deploy publish locate invoke WSPeer – P2PS WSPeer – HTTP/UDDI

  24. Web Service Choreography • Web services • Connecting Web services and integrating with other Triana units • Xmethods - http://www.xmethods.com/ • Services -> Import Services • Bible Verses • http://www.stgregorioschurchdc.org/wsdl/Bible.wsdl • Drag the readBible service to a new scratch pad • Connect a Common.String.StringGen and a Common.String.StringViewer component • Double click StringGen for the input window • Psalm 119:105 • Matthew 5:3-12,44-48 • Matthew 5:3-12;John 1:1-5 • Press Play, double click StringViewer to see the output

  25. Multiple Services • Xmethods - http://www.xmethods.com/ • Services -> Import Services • Urdu Translator • http://www.apniurdu.com/SOAP/Urdu2.wsdl • Drag the Translate service to the scratch pad • Drag a Common.Control.Duplicator unit the scratch pad • Connect readBible to Duplicator • connect one output from Duplicator to the StringViewer • Connect the other output to the Translator service • Drag a new StringViewer to the scratch pad and connect the output of the Translator to that • Press play and double click the string viewers • Try different inputs for StringGen

  26. Web Services Resource Framework • WSRF defines conventions for accessing stateful resources (WS-Resources) via Web service interactions • Resources are refered to using EPR in message headers • In Triana we refer to EPR/WS-addresses as a “context” for extendibility, i.e. WS-Security context • Example using Globus 4 CounterService example • Import http://d220.astro.cf.ac.uk:8080/wsrf/services/CounterService?wsdl

  27. The link between context and WSTypeGen is a “trigger” node to create a control dependency • Right click on WSTypeGen1, select Node Editor • Select “In Params” tab, click add, select “Trigger node only” select ok • Double click the WSTypeGen unit, type a number into the “int” field • To set the context on the Add service, double click or right click and select “Set Context” • Press play • Drag another WSTypeGen unit and Add service onto the scratch pad, set another integer and play again

  28. GridLab GAT • Application-level API (Gridlab.org) • Abstract ‘Common’ Capabilities required by Grid Applications • Resource Discovery • Job Submission • File Transfer • Capabilities extracted through a number of Grid Application Scenarios. • Makes life as a Grid programmer easier … • Motivation for the GGF SAGA research group (Simple API for Grid Applications)

  29. Using the GAT GATContext context = new GATContext(); Preferences prefs = new Preferences(); prefs.put("ResourceBroker.adaptor.name", "grms"); URI exe = new URI("file:////bin/hostname"); File outFile = GAT.createFile(context, prefs, new URI("file:///out") SoftwareDescription sd = new SoftwareDescription(); sd.setLocation(exe); sd.setStdout(outFile); ResourceDescription rd = new HardwareResourceDescription(); rd.put("hostname", "bouscat.cs.cf.ac.uk"); JobDescription jd = new JobDescription(sd, rd); ResourceBroker broker = GAT.createResourceBroker(context, prefs); Job job = broker.submitJob(jd);

  30. Visual GAT • GAT Primitives represented as Workflow Objects • Job Tasks • File Tasks • Grid Data Flow implicit in Workflow Connections • File Task -> File Task = Data Transfer • File Task -> Job Task = Pre-Stage File • File Task -> Local Task = File Read • Grid Operations handled by GAT Adaptors • Grid File -> Grid File = GridFTP • HTTP File -> Local Task = HTTP • Grid File -> GRMS Job = GRMS Pre-Stage

  31. Grid Workflow in Triana File Write File (Local/Remote) Pre-Stage Job (Local/Remote) Post-Stage Temporary File File Read Local Java Unit

  32. Visual GAT Alternative • Grid Operations Expressed Explicitly • GridFTP Tasks • File Write Tasks • File Read Tasks • Not Intuitive • Not Future Proof

  33. GridLab GAT File CPI Resource Broker CPI HTTP Adaptor GridFTP Adaptor Data Movement Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor File Input (HTTP) Triana WaveToy.par www. gridlab. org

  34. File Staging (GridFTP) Triana GridLab GAT File CPI Resource Broker CPI HTTP Adaptor GridFTP Adaptor Data Movement Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor WaveToy.par www. gridlab. org Testbed

  35. Job Submission 1 Triana GridLab GAT File CPI Resource Broker CPI HTTP Adaptor GridFTP Adaptor Data Movement Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor GAS GRMS Cactus www. gridlab. org Testbed Data Mvment Testbed WaveToy.par Delphoi Mercury iGrid

  36. Job Submission 2 Triana GridLab GAT File CPI Resource Broker CPI Instant Mess HTTP Adaptor GridFTP Adaptor Data Movement Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor Portal GAS GRMS Cactus www. gridlab. org Testbed Data Mvment Testbed Delphoi Mercury iGrid

  37. Job Submission (GRAM) Triana GridLab GAT File CPI Resource Broker CPI Instant Mess HTTP Adaptor GridFTP Adaptor Data Movement Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor Portal GRAM Cactus www. gridlab. org Testbed Testbed

  38. WSPeer User Interface Web Service GridMonSteer Resource Broker (e.g. GRMS) Grid Side submit run User Side Application (e.g. Triana) GridMonSteer (gms) gms http://www.xyx.org:4802/axis/gms cactus monitor cactus phi_xy.jpg phi_yz.jpg phi_xy.jpg phi_yz.jpg rundir http://www.xyz.org/axis/gms invoke using SOAP

  39. Triana Grid Workflow Summary • GridLab GAT • Application-level API • Abstracts Common Capabilities required by Grid Applications such as Triana • Visual GAT • GAT Primitives represented in Triana as Workflow Objects • Grid Data Flow implicit in Workflow Connections • GridMonSteer • Generic Architecture for Monitoring and Steering Legacy Applications

  40. Future • Java GAT 2.0 (coming soon Vrije Universiteit) • globus-4 web services, a windows shares adaptor (SAMBA/CIFS), an initial set of glite adaptors • Mrs DART - Distributed Audio Retrieval with Triana • Distribute music analysis workflows to P2P network to analyse user digital audio collections and make content based recommendations • Alchemist • Generic version of DART framework for distributed scientific search and analysis • WHiP - Workflows Hosted in Portals • Share workflows in collaborative Web portals for scientific collaboration and knowledge capture

  41. Links • Triana - http://www.trianacode.org/ • WSPeer - http://www.wspeer.org/ • WHiP - http://www.whipplugin.org/ • Mrs DART - http://www.mrsdart.com/ • Java GAT - http://www.cs.vu.nl/ibis/javagat.html

More Related