Resource Allocation for Distributed Streaming Applications

1. Resource Allocation for Distributed Streaming Applications Qian Zhu and Gagan Agrawal Department of Computer Science and Engineering The Ohio State University ICPP 2008 Conference Sept. 10th, 2008 Portland, Oregon ICPP 2008

2. Data Streaming Applications • Computational Steering • Interactivelycontrol scientific simulations • Computer Vision Based Surveillance • Track people and monitor critical infrastructure • Images captured by multiple cameras • Online Network Intrusion Detection • Analyze connection request logs • Identify unusual patterns 2 ICPP 2008

3. Streaming Applications in Wide Area Environments • Distributed high-volume data sources • Increasing WAN bandwidths • Better than secondary storage bandwidths • Geographically distributed users / consumers of data • Exploit flexibility in resource usage in Grid Environments

4. Our Previous Work • A middleware system GATES • Grid-based AdapTive Executions on Streams • Integration with Grid Standards • Support for self-adaptation • Dynamic allocation and fault-tolerance

5. Resource Allocation in Streaming Grid Applications • Challenges • Pipeline of processing stages • Computation and communication requirements • Long running nature • Dynamic grid resources • Current Approach • Ad Hoc and Heuristics-based • Not considering both bandwidth and computing power ICPP 2008

6. Overview of Our Research • Static Resource Allocation • Subgraph isomorphism based • Handle Network bandwidth and Computing power • Effectiveness value • Goal • To minimize the execution time of the data streaming applications ICPP 2008

7. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 7 ICPP 2008

8. Data Stream Processing Model • Directed Acyclic Graph (DAG) – Gp(Vp, Ep) source Computing Power Requirement Bandwidth Requirement Processing nodes sink ICPP 2008

9. Resource Model • Directed Acyclic Graph (DAG) – GR(VR, ER) Computing power Bandwidth ICPP 2008

10. Problem Description • To Allocate Resources to the Data Stream Application • A mapping from Gp(Vp, Ep) to GR(VR, ER) • Modified Subgraph Isomorphism Based • To choose an isomorphic subgraph of GR • Transporters • Optimal Mapping • Effectiveness value • To minimize the execution time ICPP 2008

11. Example A 1000 B 400 C 200 E 2000 D 100 transporter ICPP 2008

12. Effectiveness Value • Bandwidth only • Including Computing Power Number of transporters A sigmoid function Overhead of adding transporters Computing power match ICPP 2008

13. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 13 ICPP 2008

14. Proposed Algorithm • Background: VF algorithm (L.P.Cordella et al.) • State Space Representation (SSR) • Feasibility rules • Depth-First Search • Pros and Cons • Efficient with small graphs (<200 nodes) • A large number of candidate partial mappings ICPP 2008

15. 1 2 3 4 Proposed Algorithm – Step 1 • Prune Candidate Partial Mappings • Candidate node list • Reduce potential matches • Multiple Partial Mapping set A B C D E F G C D G E F A 1000 3 200 Cand(3)={C,D,G} B 400 C 200 E 2000 D 100 ICPP 2008

16. 1 2 3 4 Proposed Algorithm – Step 2 • Modified Subgraph Isomorphism Mapping • Transporters A B C D E F G C D G E F A 1000 3 200 B 400 C 200 E 2000 D 100 Candidate pair: (3,C) transporter Candidate pair: (3,D) ICPP 2008

17. Handle Computing Power • Computing Node  Network Link • Computing power  Network bandwidth • Effectiveness Value Calculation • Possible Issues: high bandwidth and low computing power • Map one node onto a cluster of network nodes ICPP 2008

18. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 18 ICPP 2008

19. Goals for the Experiments • Demonstrate the Scalability of Our Resource Allocation Algorithm • Demonstrate the High Performance of the Applications ICPP 2008

20. Experiment Setup • Algorithms Compared • Optimal • Streamline • Streaming Applications • Volume Rendering Application • A Synthetic Application ICPP 2008

21. Scalability of the Resource Allocation Algorithm ICPP 2008

22. Application Performance • Volume Rendering 33% 29% 27% Within 4% ICPP 2008

23. Application Performance • A Synthetic Application 40% 36% Within 3% 34% ICPP 2008

24. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 24 ICPP 2008

25. Related Work • Resource Allocation for Stream Processing • Tang et al. (HPCC 06), Ali et al. (PDPTA 02) • Resource Allocation for Grid Computing • Abdu et al. (IPDPS 01), Bhat et al. (Grid 07), Hong et al. (ICPP 03) • Subgraph Isomorphism Algorithms and Applications • Bioinformatics (Online Information 90), VLSI design (ISCAS 95), Mobile robot design (JPR 95) ICPP 2008

26. Conclusion • Modified Subgraph Isomorphism Algorithm for Resource Allocation in Grid Streaming Applications • Handling Network Bandwidth and Computing Power • Comparable Overhead with Streamline • Improved Application Performance ICPP 2008

27. Thank you! ICPP 2008