1 / 38

THE RESEARCH PROCESS

OPEN VERSUS CLOSED: A CAUTIONARY TALE Bianca Schroeder Adam Wierman Mor Harchol-Balter Computer Science Department Carnegie Mellon University To appear at NSDI 2006. new system has smaller response time!. old. new system. new. This comparison requires testing

amoser
Download Presentation

THE RESEARCH PROCESS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OPEN VERSUS CLOSED:A CAUTIONARY TALEBianca Schroeder Adam Wierman Mor Harchol-BalterComputer Science DepartmentCarnegie Mellon UniversityTo appear at NSDI 2006

  2. new system has smaller response time! old new system new This comparison requires testing the two systems on realistic workloads THE RESEARCH PROCESS standard system

  3. think send receive server MANY WAYS TO GENERATE REALISTIC WORKLOADS User driven Model the behavior of a typical user then fork one process for each user. User requests web page, receives page, reads page, clicks on new link CLOSED SYSTEM MODEL

  4. Trace driven 1:01.12 ip1 GET a.gif HTTP/1.0 1:01.20 ip2 GET b.htm HTTP/1.0 1:01.25 ip1 GET c.jpg HTTP/1.0 1:01.27 ip1 GET d.txt HTTP/1.0 1:01.28 ip3 GET a.htm HTTP/1.0 1:01.35 ip4 GET d.gif HTTP/1.0 1:01.45 ip2 GET e.htm HTTP/1.0 : : arrival times service demands x x x new arrivals server MANY WAYS TO GENERATE REALISTIC WORKLOADS next arrival time from trace file sizes from trace OPEN SYSTEM MODEL

  5. x x x new arrivals server MANY WAYS TO GENERATE REALISTIC WORKLOADS Distribution driven Use distributions of interarrival times and service demands (typically using trace info) interarrival time dist. service demand dist. sample dist. sample dist. OPEN SYSTEM MODEL

  6. OPEN MODEL CLOSED MODEL Arrivals are independent of completions Arrivals are completely dependent on departures There is a fixed population of users, called the Multi-Programming-Level (MPL) There is no max number of simultaneous users

  7. OPEN MODEL WEB WORKLOAD GENERATORS CLOSED MODEL Do you use an open or closed model? Surge • Workload generators for thesame purpose use differentsystem models! • It’s often not clear which model workload generatorsuse! SPECWeb TPC-W Sclient RUBiS WebBench Webjamma

  8. OUR GOAL TODAY What is the impact of the choice of an open or closed model?

  9. OPEN CLOSED HOW DO WE COMPARE OPEN AND CLOSED SYSTEMS? • Fix the service distribution acrossthe systems • Fix the load across the systems adjust load using the arrival rate load depends only on mean arrival rate and mean service demands load depends on MPL, think times, mean of service demands, variability of service demands, scheduling policy, … adjust load using the think time

  10. How do open and closed response times compare? FCFS scheduling open  Poisson arrival process closed  Exponential think times

  11. 1000 100 10 mean response time 0 0.25 0.5 0.75 1 load FCFS scheduling open  Poisson arrival process closed  Exponential think times Open CLOSED <<OPEN Closed (MPL=10)

  12. 1000 100 10 mean response time 0 0.25 0.5 0.75 1 load FCFS scheduling open  Poisson arrival process closed  Exponential think times Open CLOSED OPEN Closed (MPL=1000) Closed (MPL=100) Closed (MPL=10)

  13. OPEN MODEL CLOSED MODEL VS CLOSED  OPEN AS MPL GROWS As MPL grows arrival rate becomes independent of completion rate

  14. 1500 1000 500 mean response time low variability high variability How quickly does Closed  Open? Open Web Workloads Closed (MPL=1000) Closed (MPL=100) Closed (MPL=10)

  15. SUMMARY SO FAR high variability Open >> Closed It depends low variability Open > Closed Open ≈ Closed Small MPL Large MPL

  16. OUR GOAL TODAY What is the impact of the choice of an open or closed model? • What is the impacton the effectivenessof scheduling? • What is the impactin practice? It matters a lot!

  17. Improved design Shortest Remaining Processing Time (SRPT) Standard design Processor Sharing (PS) Compare using a workload generator SCHEDULING IS A KEY COMPONENT OF SYSTEM DESIGN WEB SERVERS Does the effectiveness of scheduling depend on the system model (open vs. closed)?

  18. PLJF FCFS PS SRPT SCHEDULING IN OPEN SYSTEMS OPEN 1000 600 300 0 How do the closed results compare? mean response time 0 .25 .5 .75 1 load

  19. PLJF FCFS PS SRPT PLJF FCFS PS SRPT • Limited impact of variability in closed system • Bounded number of jobs in closed system • Dependencies between completions and arrivalsin closed system reduces burstiness Why? CONTRASTING THE IMPACT OF SCHEDULING OPEN CLOSED 1000 600 300 0 mean response time 0 .25 .5 .75 1 0 .25 .5 .75 1 load load

  20. OUR GOAL TODAY What is the impact of the choice of an open or closed model? It matters a lot! Especially when evaluating scheduling policies What is the impact in practice?

  21. OPEN VS CLOSEDIN PRACTICE 4 CASE STUDIES • Serving static web content • Database backend ofan e-commerce site • Auctioning web site • Supercomputing center testbed implementation trace-based simulation

  22. PS PS SRPT SRPT OPEN VS CLOSEDIN PRACTICE STATIC WEB SERVER OPEN CLOSED 300 200 100 MPL=50 mean response time 0 .25 .5 .75 1 0 .25 .5 .75 1 load load Different models give different conclusion about benefits of SRPT

  23. OPEN CLOSED 10 8 4 0 MPL=50 PS E-COMMERCE SITE PS PESJF PESJF mean response time 20 14 7 0 load load MPL=50 PS AUCTION SITE PS SRPT SRPT 0 .25 .5 .75 1 0 .25 .5 .75 1 load load

  24. How can we identify whether to use an open or closed model? OUR GOAL TODAY What is the impact of the choice of an open or closed model? It matters a lot in practice! Especially when evaluating scheduling policies

  25. CHOOSING A SYSTEM MODEL Web workloads • A site being “Slashdotted” • Online gaming site • Science Institute - USGS • Online dept. store • Financial service provider • Kasparov vs Deep Blue • CMU web server • World cup site Open or Closed?

  26. NEITHER THE OPEN OR CLOSEDMODEL IS COMPLETELY REALISTIC

  27. PARTLY-OPEN MODEL with probability q return to the system think send receive x x x leave system new arrivals server A MORE REALISTIC ALTERNATIVE What parameters affect the load? Does think time affect the load? How do think times affect response times?

  28. PS SRPT THE EFFECT OFTHINK TIME STATIC WEB SERVER 300 200 100 0 mean response time 1 10 100 1000 mean think time

  29. PARTLY-OPEN MODEL with probability q return to the system think send receive q0 q1 x x x ? ? OPEN CLOSED leave system new arrivals server A MORE REALISTIC ALTERNATIVE Workload generators are only Open/Closed! number of requests per visit ↓ number of requests per visit↑

  30. PS open PS SRPT PS closed THE TRANSITION FROM OPEN  CLOSED STATIC WEB SERVER CLOSED 300 200 100 0 OPEN mean response time 0 5 10 15 20 mean number of requests per visit

  31. STATIC WEB E-COMMERCE SITE 200 100 0 9 6 3 0 PS PS SRPT PESJF 15 10 5 0 AUCTIONING PS SRPT 0 5 10 15 20 0 5 10 15 20 0 5 10 15 20 THE PARTLY-OPEN SYSTEM IN PRACTICE mean response time mean number of requests per visit

  32. OPEN Open or Closed? CLOSED CHOOSING A SYSTEM MODEL Web workloads A site being “Slashdotted” (1.2) Financial service provider (1.4) CMU web server (1.8) Kasparov vs Deep Blue (2.4) Science Institute USGS (3.6) Online dept. store (5.4) World cup site (11.6) Online gaming site (12.9) Use partly-open system to decide

  33. What is the impact of the choice of an open or closed model? It matters a lot in practice! Especially when evaluating scheduling policies How can we identify whether to use an open or closed model? OUR GOAL

  34. WRAPUP

  35. PS SRPT high variability It depends Open >> Closed low variability Open ≈ Closed Open > Closed Small MPL Large MPL PS PARTLY-OPEN SRPT PS SRPT OPEN AND CLOSED CAN BEHAVE VERY DIFFERENTLY THESE DIFFERENCES ARE IMPORTANT IN PRACTICE OPEN CLOSED VS

  36. When forced to use either open or closed, the choice can be made using simple heuristics: model the workload using a partly-open system SIMPLE GUIDELINES Understanding the appropriate system model give guidelines about the effectiveness of scheduling IMPACT ON DESIGN A CAUTIONARY TALE Be careful of the underlying model in workload generators: open vs. closed has huge practical impact Web workload generators need to be flexible

  37. OPEN VERSUS CLOSED:A CAUTIONARY TALEBianca Schroeder Adam Wierman Mor Harchol-BalterComputer Science DepartmentCarnegie Mellon UniversityTo appear at NSDI 2006

More Related