Xiao Liu 1 , Jinjun Chen 1 , Zhangjun Wu 2 , Zhiwei Ni 2 , Dong Yuan 1 , Yun Yang 1

1. Handling Recoverable Temporal Violations in Scientific Workflow Systems: A Workflow Rescheduling Based Strategy Xiao Liu1, Jinjun Chen1, Zhangjun Wu2, Zhiwei Ni2, Dong Yuan1, Yun Yang1 1CS3, Swinburne University of Technology Melbourne, Australia 2Institute of Intelligent Management, Hefei University of Technology Hefei, China

2. Outline • Background • Workflow Technology Group • SwinDeW Family • SwinGrid, SwinCloud • Workflow Rescheduling • Workflow Rescheduling • Temporal Violations • A Two Stage Workflow Local Rescheduling Strategy • Evaluation • Summary 2

3. Workflow Technology Group Overview • About Us: WT group is a part of CS3 (The Centre for Complex Software Systems and Services), a Tier-1 university research centreat Swinburne University of Technology. Our group conducts research into workflow technologies for complex software systems and services including peer-to-peer, grid, and cloud computing based e-science, e-business, transactional and inter-organisational workflows. Others: Prof Ryszard Kowalczyk Prof Chengfei Liu Dr Jun Yan (Wollongong) Prof Hai Jin (HUST) Prof Mingshu Li (ISCAS) Prof Qing Wang (ISCAS) Prof Zhiwei Ni (HFUT) Prof Jinpeng Huai (BUAA) Leader: Prof Yun Yang Visitors (7-8/09): Prof Lee Osterweil Prof. Lori Clarke Researchers: Dr Jinjun Chen Nauman Saeed (PhD) Qiang He (PhD) Ke Liu (PhD) Xiao Liu (PhD) Dong Yuan (PhD) Zhangjun Wu (PhD - visitor) 3

4. SwinDeW Family SwinDeW – Swinburne Decentralised Workflow- foundation prototype based on p2p • SwinDeW – past • SwinDeW-A (for Agents) – ARC DP06 • SwinDeW-G (for Grid) – past • SwinDeW-V (for Verification) – current (pending ARC DP) • SwinDeW-E (for eScience) – current (pending ARC DP) • SwinDeW-C (for cloud) – current (ARC LP) • Others: SwinDeW-B / -S / -P / -G – past 4

5. SwinGrid to SwinCloud 5

6. Outline Background Workflow Technology Group SwinDeW Family SwinGrid, SwinCloud Workflow Rescheduling Workflow Rescheduling Temporal Violations A Two Stage Workflow Local Rescheduling Strategy Evaluation Summary 6

7. Workflow Rescheduling • A workflow scheduling is a process that maps the execution of inter-dependent workflow tasks on the distributed resources. It allocates suitable resources to workflow tasks so that the execution can be completed to satisfy objective functions imposed by users. • Workflow rescheduling is a process to regenerate or modify the current scheduling plan, due to • Changes of system environments: resource pool change, resource performance variance, resource break down, etc. • Contract (constraint) violations: task execution failures, temporal violations, cost violations, etc. • Many others: when the current scheduling plan cannot satisfy the pre-defined functional or non-functional objectives

8. Workflow Rescheduling • Workflow rescheduling plays an equally or even more important role as workflow scheduling in highly distributed and dynamic computing environments • As a supplement function to workflow scheduling in the resource management component • As a function in the exception handling component • Example rescheduling strategy: • Rescheduling by Stop and Restart • Rescheduling by Processor Swapping • Simple (little additional programming); high cost for resource reservation

9. Temporal Violations • Most scientific workflows are subjected to • Local temporal constraints (milestones) for workflow segments • Global temporal constraints (deadlines) for workflow instance • Violations of local and global temporal constraints • Detection: Temporal Checkpoint Selection (ICSE08, TOSEM10) • Handling: Time deficits compensation (CCPE07, CCGrid10) • Statistical recoverable temporal violations • Probability based temporal consistency model (BPM08, ICSP09) • Based on “3sigm” rule in statistics • Details omitted, please refer the paper for details

10. Outline Background Workflow Technology Group SwinDeW Family SwinGrid, SwinCloud Workflow Rescheduling Workflow Rescheduling Temporal Violations A Two Stage Workflow Local Rescheduling Strategy Evaluation Summary 10

11. A Two Stage Workflow Local Rescheduling Strategy For handling temporal violations Key objective: reduce or ideally remove the time deficit at the current checkpoint, i.e. to reduce the execution time of the subsequent activities after the checkpoint in the violated workflow segment as much as possible Requirement 1: fighting good balance between time deficit compensation and the completion time of other activities (workflow activities and general tasks, with or without temporal constraints) – from the overall makespan perspective Requirement 2: utilising available resources in the system rather than recruiting additional resources – from the overall cost perspective 11

12. Integrated Task Resource List 12

13. Pseudo-code for An Abstract Strategy 13

14. Evaluation • Two example implementation of our strategy • An ant colony optimisation based strategy • A genetic algorithm based strategy

15. Optimisation on Total Makespan 15

16. Optimisation on Total Cost 16

17. Compensation on Violated Workflow Segment 17

18. Fitness Value 18

19. CPU Time 19

20. Experiment Results on Temporal Violation Rates 20

21. Outline Background Scientific Workflows Workflow Scheduling and Rescheduling Workflow Scheduling Classification Representative Scheduling Algorithms Workflow Rescheduling Classification A Two Stage Workflow Local Rescheduling Strategy Case Study 1:GA Based Rescheduling Case Study 2: ACO Based Rescheduling Comparison Summary 21

22. Summary Workflow Rescheduling – Exception Handling Exception Handling on Scientific Workflow Temporal Violations A Two Stage Workflow Local Rescheduling Strategy ACO, GA Automatic, Cost-Effective Future Work Data movement cost More scheduling algorithms 22

23. The End – Thank You! Any questions or comments? Email: xliu@swin.edu.au Website: http://www.ict.swin.edu.au/personal/xliu/ CS3: http://www.swinburne.edu.au/ict/research/cs3/ 23