Performance Prediction Engineering

1. This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation • In Slide Show, click on the right mouse button • Select “Meeting Minder” • Select the “Action Items” tab • Type in action items as they come up • Click OK to dismiss this box This will automatically create an Action Item slide at the end of your presentation with your points entered. Performance Prediction Engineering Francine Berman U. C. San Diego Rich Wolski U. C. San Diego and University of Tennessee

2. The Computational Grid Computer = ensemble of resources “Computer” may consist of • computational sites • dist. databases • remote instruments • visualization • distinct networks

3. Grid Programs Grid programs • may couple distributed and dissimilar resources • may incorporate tasks with different implementations • may adapt to dynamic resource load

4. Performance Models for Grid Programs • Grid applications may couple dissimilar resources • models must accommodate heterogeneity • Grid applications may incorporate tasks with different implementations • model must accommodate multiple task models • Grid applications may adapt to dynamic resource load • models must allow for dynamic parameters

5. Compositional Models • Grid programs can be represented as acompositionof tasks • “Tasks” consist of relevant performance activities • Model parameters may reflect performance variations of grid • may be parameterized by time

6. Using Grid Performance Models • Compositional models particularly useful for grid application scheduling • Application schedulers use performance prediction models to • select resources • estimate potential performance of candidate schedules • compare possible schedules

7. Schedulers and Performance Models • Predictions may be used at different levels of accuracy • predictions can be “engineered” • Knowing something about a prediction can make it more useful • performance range of predictions may provide additional information • meta-information about predictions can improve schedules

8. Performance Prediction Engineering • Performance Prediction Engineering(PPE) System is a methodology for modeling performance in dynamic Grid environments • 3 Components: • Structural performance prediction models • Quantitative meta-information • Dynamic Forecasting

9. Structural Models • Top-level Model = performance equation • describes composition of applicationwithin a specific time frame (performance grammar) • Component models • represent application performance activities (nonterminals) • Model parameters • represent system or application values (terminals)

10. Example: Modeling the Performance of SOR Regular, iterative computation 5 point stencil Divided into a red phase and a black phase 2D grid of data divided into strips Targeted to WS cluster

11. SOR performance equation SOR component models { RComp(p,t), RComm(p,t), BComp(p,t), BComm(p,t)} SOR Structural Model

12. SOR Component Models Dynamic Parameters FracAvailCPU(p,t), BWAvail(x,y,t)

13. Single-User Experiments • Question:How well does the SOR model predict performance in a single-user cluster? • Platform • heterogeneous Sparc cluster • 10 Mbit ethernet connection • quiescent machines and network • Prediction within 3% before memory spill

14. Dedicated Platform Experiments What happens when other users share the system?

15. Non-dedicated SOR Experiments

16. Improving Predictions • Many parameters represent values which vary over time • Range of behavior of time-dependent parameters represented bydistributions • Structural models can be extended to accommodatestochastic parametersand renderstochastic predictions

17. Stochastic Predictions Stochastic predictions capture range of possible behavior

18. stochastic and point-valued parameters component models Structural Model “quality” of performance prediction (lifetime, accuracy, overhead) Stochastic predictions Stochastic Structural Models

19. Stochastic SOR Performance Model • FracAvailCPU, BWAvail given by stochastic parameters • Network Weather Service improved to provide better performance information • First cut: consider stochastic parameters which can adequately be represented by normal distributions • normal distributions make math tractable

20. Experiments with Multi-user Systems • Platform • Sun workstation cluster • 10Mbit ethernet • experiments run in lab environment with additional generated load • Experiments run back-to-back for multiple trials

21. SOR Stochastic Parameters BWAvail FracAvailCPU

22. Data stays within single mode Data changes modes

23. “Single-mode” Experiments • All values captured by stochastic predictions • Maximum absolute error between means and actual values is 10%

24. “Multiple Mode” Experiments • 80% of actual values captured by stochastic prediction • Max discrepancy between stochastic prediction and actual values is 14% • Max absolute error between means and actual values is 39%

25. The Next Step What if performance range of parameters cannot be adequately represented by normal distributions? • Can we identify distributions for model parameters? • Can we combine non-normal distributions efficiently? Is the math tractable? • Can we use empirical data to determine performance ranges if distributions cannot be identified?

26. Using PPE for Application Scheduling Basic Strategy: • Develop structural model for application • Use stochastic parameters to provide information about performance range • Use profiling to determine desired level of accuracy for component models • Use stochastic prediction and meta-information to develop application schedule

27. P2 P1 ExecutionTime P3 Scheduling with Meta-Information • Stochastic predictions provide information about range of behavior • Stochastic predictions and meta-information provide additional information for schedulers

28. Quality of Information • Meta-information = Quality of Information • SOR stochastic predictions provide a measure of accuracy • Other qualitative measures are possible • lifetime • overhead • complexity • Quality of Information attributes can be used to improve scheduling

29. Preliminary Experiments:Application Scheduling with PPE Simple scheduling scenario: • SOR with strip decomposition • Scheduling strategies adjust strip size to minimize execution time • Multi-user cluster • machines connected by 10 Mbit ethernet • available CPU on at least half of the machines is multi-modal with data changing between modes frequently

30. Scheduling Strategies • Mean • data assignments determined using mean (point-valued) application execution estimates • Conservative • data adjusted so that machines with high- variance application execution estimates receive less work • goal is to reduce penalty of being wrong

31. Preliminary Scheduling Results • Conservative scheduling strategy misses big spikes, but is sometimes too conservative.

32. Research Directions • Quality of Information (QoIn) • How can we develop useful mechanisms for obtaining and quantifying performance meta-information? • How do we combine different QoIn measures? • How can QoIn measures enhance scheduling? • Contingency Scheduling • Can we develop schedules which adapt dynamically during execution?

33. More Research Directions • Performance-enhanced Tools • Netsolve enhanced with NWS and AppLeS scheduling methodology • Performance contracts • How should performance information be exchanged and brokered in grid systems? • How can we develop “grid-aware” programs?

34. Project Information • Thanks to Dr. Darema and DARPA for support and very useful feedback. • Performance Prediction Engineering Home Page: http://www-cse.ucsd.edu/groups/hpcl/ apples/PPE/index.html • PPE team: Jennifer Schopf, Neil Spring, Alan Su, Fran Berman, Rich Wolski

35. Up Next: Rich Wolski Dynamic Forecasting for Performance Prediction Engineering with the NetworkWeather Service