1 / 19

GRIDCC: Real-time Workflow system

GRIDCC: Real-time Workflow system. A.Stephen McGough, Asif Akram, Li Guo, Marko Krznaric, Luke Dickens David Colling, Janusz Martyniak Roger Powell, Paul Kyberd Constantinos Kotsokalis. What is GRIDCC. Adding real time control of instruments to the Grid What makes this unique:

moke
Download Presentation

GRIDCC: Real-time Workflow system

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GRIDCC: Real-time Workflow system A.Stephen McGough, Asif Akram, Li Guo, Marko Krznaric, Luke Dickens David Colling, Janusz Martyniak Roger Powell, Paul Kyberd Constantinos Kotsokalis WORKS 07, HPDC 07, Monterey Bay California, June 25 2007

  2. What is GRIDCC • Adding real time control of instruments to the Grid • What makes this unique: • Instruments may only beready at specific times • Reservations • Need to ensure other Gridservices are available with instruments • Reservations / SLA’s • Real-time visualisation of live results and steering • Through Virtual Control Room (VCR) • Reservations / SLA’s Grid enabled Remote Instrumentation with Distributed Control and Computation

  3. VCR ProblemSolver GRIDCC – Overview GRID Real Time QoS

  4. Motivation • QoS is important in Workflows • Just running a workflow and hoping is not enough • We need more than WS-Agreement • Though WS-Agreement is very important to us • We want to allow the user to specify their QoS requirements over a workflow • Either as hard constraints • This must finish by time x • As soft constraints • This should finish by time x in y% of cases

  5. Aim of GRIDCC Editor • Portal Based Editor • Drag and Drop • Hiding Complexities • Ease of use • Web Service Registry • QoS Constraint • XML Schema Registry (To be Implemented) • Workflow Monitoring (To be Implemented) • Standard Based • Workflow Engine Independent • Based on Flex 2 for User Interface, Java on Server Side

  6. Adding QoS to a Workflow Partner Link XPath

  7. Must run in less than 10 mins Less than 15 mins (80% confidence) Reserve a resource for this A BPEL workflow <?xml version="1.0" encoding="UTF-8"?> <process name=“…"> <partnerLinks>… </partnerLinks> <variables> <variable element="hdns1:SimpleTriggerID" name="SimpleTriggerID"/>… </variables> <flow> <sequence name="WSIIETestSequence"> <receive createInstance="yes" name="WSIIETestInitiate" operation="IETest" partnerLink="initiate_IETest_PL“…/> <sequence name="WSIIETestInitialiseAndGetStatus"> <invoke inputVariable="getDTInstanceRequestVar" name="WSIIE-getDTInstance" operation="getDTInstance“…> <target linkName="L5"/> <source linkName="L6"/> </invoke> <invoke inputVariable="getDTStatusRequestVar“…>… </invoke> <assign name="GetStatusOutput"> <target linkName="L8"/> <copy> <from part="parameters" query="" variable=""/> <to variable="StatusValue"/> </copy> </assign> </sequence> <switch> <case condition="bpws:getVariableData('StatusValue')!= bpws:getVariableData('StatusValueRunning')"> <sequence name="InitConfigureAndStartIESequence"> <while condition="bpws:getVariableData('StatusValue')!= bpws:getVariableData('StatusValueRunning')" name="UntilStatusIsRunning"> <sequence> … </sequence> </while> </sequence> <reply …/> </sequence> </flow> </process> Using XPath pointers into BPEL document

  8. The whole (Work)flow All 2Ghz Boxes Total Time 100s 100% Reliable resources Simple Example <?xml version="1.0" encoding="UTF-8"?> <QoSRequirements xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="C:\Documents and Settings\asm\Desktop\QoSRequirements.xsd"> <QoSConstraint> <XpathReference>/process/flow</XpathReference> <CPUSpeed>2048000</CPUSpeed> <MaxDurationTime>100</MaxDurationTime> <Reliability>100</Reliability> </QoSConstraint> </QoSRequirements>

  9. Refer to multiple parts of document <?xml version="1.0" encoding="UTF-8"?> <QoSRequirements xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="C:\Documents and Settings\asm\Desktop\QoSRequirements.xsd"> <QoSConstraint> <XpathReference>/process/flow/sequence[1]/sequence[1]/invoke[1]</XpathReference> <XpathReference>/process/flow/sequence[1]/sequence[1]/invoke[2]</XpathReference> <CPUSpeed>2048000</CPUSpeed> <MaxDurationTime>100</MaxDurationTime> <Reliability>100</Reliability> </QoSConstraint> </QoSRequirements>

  10. <?xml version="1.0" encoding="UTF-8"?> <QoSRequirements xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="C:\Documents and Settings\asm\Desktop\QoSRequirements.xsd"> <QoSConstraint> <XpathReference>/process/flow/sequence[1]/sequence[1]/invoke[1]</XpathReference> <ReservationRequired> <StartTime>2006-10-02T23:00:00.0Z</StartTime> <EndTime>2006-10-02T23:59:59.0Z</EndTime> <EndpointName>cpu1.doc.ic.ac.uk</EndpointName> </ReservationRequired> <CPUSpeed>1024000</CPUSpeed> <MaxDurationTime>80</MaxDurationTime> <Reliability>100</Reliability> </QoSConstraint> <QoSConstraint> <XpathReference>/process/flow/sequence[1]/sequence[1]/invoke[2]</XpathReference> <CPUSpeed>2048000</CPUSpeed> <MaxDurationTime>20</MaxDurationTime> <Reliability>50</Reliability> </QoSConstraint> </QoSRequirements> <ReservationMade> <StartTime>2006-10-02T23:00:00.0Z</StartTime> <EndTime>2006-10-02T23:59:59.0Z</EndTime> <EndpointName>cpu3.doc.ic.ac.uk</EndpointName> <Token>dveinvunvuin42235454</Token> </ReservationMade> Make a reservation through Agreement Service (WS-Agreement) Store result here

  11. Workflow Management Service (WfMS) Architecture

  12. Architecture • Planner • Determines if a workflow can be executed • Decides the best resources to use for the workflow • Workflow Engine • Engine to enact the workflow determined within the Planner • ActiveBPEL • Temporary WfWS • Instance of a workflow deployed to the Workflow Engine • Agreement Service • Negotiation service for agreements with the CEs, SEs, IEs, NEs

  13. Architecture • Observer • Monitors the temporary WfWS as it executes checking that it is performing within parameters • Can be seen as an instance of a “Global Problem Solver” • Delegation Service • Provides a mechanism for delegating credentials through the WfMS • Workload Management System • Standard gLite WMS

  14. Planner BPEL & QoS BPEL & QoS Information Service Performance Information Agreement Service What is the Planner? • Makes a “plan” for how to execute a workflow • Can QoS requirements be met • Which resources to use • Whether reservations are required in order to meet QoS • Does not override the right of a user to request reservations • Manipulation of the workflow • The expectation for how the workflow will progress • What will be executed and where

  15. Flow of Documents

  16. Performance Repository WS Interface WfMS Planner Observer BPEL + QoS BPEL Engine Façade F F F Agreement Service SE CE IE GRIDCC WfMS Workflow Optimiser Validator Resource Selector Resource Pruning Reserver SEDA based architecture

  17. Pipeline Stages for WfMS • XML to Object Stream • Convert the XML document into a set of objects • Object Stream to XML • Convert Objects into XML document • Reservation maker • Make reservations through the Agreement Service • Workflow fiddly bits (add in stuff we need) • Automatically deal with adding in security elements to workflow • Insert error checking into workflow, Performance logging of workflows • Workflow Validator • Workflow manipulator • Modify a workflow to improve performance • QoS validate • Validate that the workflow can meet the QoS requirements placed on it • Workflow Resource Selector • Select the resources within the Grid which will allow QoS to be met. • JDL Manipulator - Manipulate JDL documents to use resources better

  18. Questions WORKS 07, HPDC 07, Monterey Bay California, June 25 2007

More Related