1 / 78

The Forgotten Factor: FACTS on Performance Evaluation and its Dependence on Workloads

The Forgotten Factor: FACTS on Performance Evaluation and its Dependence on Workloads. Dror Feitelson Hebrew University. Performance Evaluation. In system design Selection of algorithms Setting parameter values In procurement decisions Value for money Meet usage goals

rafal
Download Presentation

The Forgotten Factor: FACTS on Performance Evaluation and its Dependence on Workloads

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Forgotten Factor: FACTSon Performance Evaluationand its Dependence on Workloads Dror Feitelson Hebrew University

  2. Performance Evaluation • In system design • Selection of algorithms • Setting parameter values • In procurement decisions • Value for money • Meet usage goals • For capacity planing

  3. The Good Old Days… • The skies were blue • The simulation results were conclusive • Our scheme was better than theirs Feitelson & Jette, JSSPP 1997

  4. But in their papers, Their scheme was better than ours!

  5. How could they be so wrong?

  6. Performance evaluation depends on: • The system’s design (What we teach in algorithms and data structures) • Its implementation (What we teach in programming courses) • The workload to which it is subjected • The metric used in the evaluation • Interactions between these factors

  7. Performance evaluation depends on: • The system’s design (What we teach in algorithms and data structures) • Its implementation (What we teach in programming courses) • The workload to which it is subjected • The metric used in the evaluation • Interactions between these factors

  8. Outline for Today • Three examples of how workloads affect performance evaluation • Workload modeling • Research agenda In the context of parallel job scheduling

  9. Example #1 Gang Scheduling and Job Size Distribution

  10. Gang What?!? Time slicing parallel jobs with coordinated context switching Ousterhout matrix Ousterhout, ICDCS 1982

  11. Gang What?!? Time slicing parallel jobs with coordinated context switching Ousterhout matrix Optimization: Alternative scheduling Ousterhout, ICDCS 1982

  12. Packing Jobs Use a buddy system for allocating processors Feitelson & Rudolph, Computer 1990

  13. Packing Jobs Use a buddy system for allocating processors

  14. Packing Jobs Use a buddy system for allocating processors

  15. Packing Jobs Use a buddy system for allocating processors

  16. Packing Jobs Use a buddy system for allocating processors

  17. The Question: • The buddy system leads to internal fragmentation • But it also improves the chances of alternative scheduling, because processors are allocated in predefined groups Which effect dominates the other?

  18. The Answer (part 1): Feitelson & Rudolph, JPDC 1996

  19. The Answer (part 2):

  20. The Answer (part 2):

  21. The Answer (part 2):

  22. The Answer (part 2): • Many small jobs • Many sequential jobs • Many power of two jobs • Practically no jobs use full machine Conclusion: buddy system should work well

  23. Verification Feitelson, JSSPP 1996

  24. Example #2 Parallel Job Scheduling and Job Scaling

  25. Variable Partitioning • Each job gets a dedicated partition for the duration of its execution • Resembles 2D bin packing • Packing large jobs first should lead to better performance • But what about correlation of size and runtime?

  26. “Scan” Algorithm • Keep jobs in separate queues according to size (sizes are powers of 2) • Serve the queues Round Robin, scheduling all jobs from each queue (they pack perfectly) • Assuming constant work model, large jobs only block the machine for a short time Krueger et al., IEEE TPDS 1994

  27. Scaling Models • Constant work • Parallelism for speedup: Amdahl’s Law • Large first  SJF • Constant time • Size and runtime are uncorrelated • Memory bound • Large first  LJF • Full-size jobs lead to blockout Worley, SIAM JSSC 1990

  28. The Data Data: SDSC Paragon, 1995/6

  29. The Data Data: SDSC Paragon, 1995/6

  30. The Data Data: SDSC Paragon, 1995/6

  31. Conclusion • Parallelism used for better results, not for faster results • Constant work model is unrealistic • Memory bound model is reasonable • Scan algorithm will probably not perform well in practice

  32. Example #3 Backfilling and User Runtime Estimation

  33. Backfilling • Variable partitioning can suffer from external fragmentation • Backfilling optimization: move jobs forward to fill in holes in the schedule • Requires knowledge of expected job runtimes

  34. Variants • EASY backfilling Make reservation for first queued job • Conservative backfilling Make reservation for all queued jobs

  35. User Runtime Estimates • Lower estimates improve chance of backfilling and better response time • Too low estimates run the risk of having the job killed • So estimates should be accurate, right?

  36. They Aren’t Mu’alem & Feitelson, IEEE TPDS 2001

  37. Surprising Consequences • Inaccurate estimates actually lead to improved performance • Performance evaluation results may depend on the accuracy of runtime estimates • Example: EASY vs. conservative • Using different workloads • And different metrics

  38. EASY vs. Conservative Using CTC SP2 workload

  39. EASY vs. Conservative Using Jann workload model

  40. EASY vs. Conservative Using Feitelson workload model

  41. Conflicting Results Explained • Jann uses accurate runtime estimates • This leads to a tighter schedule • EASY is not affected too much • Conservative manages less backfilling of long jobs, because respects more reservations

  42. Conservative is bad for the long jobsGood for short ones that are respectedConservativeEASY

  43. Conflicting Results Explained • Response time sensitive to long jobs, which favor EASY • Slowdown sensitive to short jobs, which favor conservative • All this does not happen at CTC, because estimates are so loose that backfill can occur even under conservative

  44. Verification Run CTC workload with accurate estimates

  45. But What About My Model? Simply does not have such small long jobs

  46. Workload Modeling

  47. No Data • Innovative unprecedented systems • Wireless • Hand-held • Use an educated guess • Self similarity • Heavy tails • Zipf distribution

  48. Serendipitous Data • Data may be collected for various reasons • Accounting logs • Audit logs • Debugging logs • Just-so logs • Can lead to wealth of information

  49. NASA Ames iPSC/860 log 42050 jobs from Oct-Dec 1993 userjobnodesruntimedatetime user4 cmd8 32 70 11/10/93 10:13:17 user4 cmd8 32 70 11/10/93 10:19:30 user42 nqs450 32 3300 11/10/93 10:22:07 user41 cmd342 4 54 11/10/93 10:22:37 sysadmin pwd 1 6 11/10/93 10:22:42 user4 cmd8 32 60 11/10/93 10:25:42 sysadmin pwd 1 3 11/10/93 10:30:43 user41 cmd342 4 126 11/10/93 10:31:32 Feitelson & Nitzberg, JSSPP 1995

  50. Distribution of Job Sizes

More Related