System Performance & Scalability

1. System Performance & Scalability i206 Fall 2010 John Chuang

2. John Chuang http://bits.blogs.nytimes.com/2007/11/26/yahoos-cybermonday-meltdown/index.html

3. Computing Trends • Multi-core CPUs • Data centers • Cloud computing • What are the drivers? • scalability, availability, cost-effectiveness John Chuang

4. Lecture Outline • Performance Metrics • Availability • Queuing theory • M/M/1 queue • Scalability • M/M/m queue John Chuang

5. What is Performance? • Users want fast response time and high availability • Managers want happy users, and many of them, while minimizing cost • What are standard measures of system performance? John Chuang

6. Performance Metrics • Response time (seconds) • Throughput (MIPS, Mbps, TPS, ...) • Resource utilization (%) • Availability (%) John Chuang

7. Availability Availability = MTTF / (MTTF + MTTR) • Mean-time-to-failure (MTTF) • Mean-time-to-recover (MTTR) John Chuang

8. Network Client Server Formulate request Message latency Queuing time Processing time Message latency Interpret response Response Time Adapted from: David Messerschmitt John Chuang

9. Queuing Theory 2. Service Time Distribution 6. Service Discipline 1. Arrival Process 4. System Capacity 5. Customer Population 3. Number of Servers John Chuang Source: Raj Jain

10. Kendall’s Notation (1953) 2. Service Time Distribution 6. Service Discipline 1. Arrival Process • A/B/c/k/N/D • A: arrival process • B: service time distribution • c: number of servers • k: system capacity • N: population size • D: service discipline 4. System Capacity 5. Customer Population 3. Number of Servers M: Markov (exponential, memoryless, random, Poisson) D: deterministic E: Erlang H: hyper-exponential G: general FCFS: first come first served FCLS: first come last served RR: round-robin etc. John Chuang

11. Example Systems 8 8 • M/M/1/ / /FCFS (simplified as M/M/1) • Markovian (Poisson, memoryless) arrival • Markovian service time • 1 server • Infinite server capacity • Infinite arrival stream • First-come-first-serve discipline • Other examples: • M/M/1/k (finite capacity) • M/M/m (m servers) • G/D/1 (arbitrary arrival, deterministic service time) John Chuang

12. M/M/1 Queue • Poisson arrival, with average arrival rate of l jobs/sec • Poisson service, with average service rate of m jobs/sec • Single server with infinite queue • System utilization (hopefully < 1): r = l/m • Average number of jobs in system: N =  n·pn = r/(1 - r) • System throughput (if r < 1) : X = l • Average response time (from Little’s Law): R = N/X = 1/(m - l) John Chuang

13. Example: Web Server • Web server receives 40 requests/second • Web server can process 100 requests/second • What is server utilization? • At any given time, how many requests are at server (waiting plus being processed)? • What is the mean total delay at server (waiting plus processing)? • What happens when traffic rate doubles? John Chuang

14. Example: Web Server • l = 40 requests/second • m = 100 requests/second • Utilization = r = l/m = 40/100 = 40% • # of requests = N = r/(1 - r) = 0.67 • Average time spent at server = R = N/X = 0.67/40 = 17ms John Chuang

15. Example: Traffic Doubled • l = 80 requests/second • m = 100 requests/second • Utilization = r = l/m = 80/100 = 80% • # of requests = N = r/(1 - r) = 4 • Average time spent at server = R = N/X = 4/80 = 50ms (more than doubled!) John Chuang

16. Approaching Congestion • l = 99 requests/second • m = 100 requests/second • Utilization = r = l/m = 99/100 = 99% • # of requests = N = r/(1 - r) = 99 • Average time spent at server = R = N/X = 99/99 = 1 second! John Chuang

17. Utilization Affects Performance John Chuang

18. M/M/1/k Queue (Finite Capacity) • r = l/m • N = r/(1-r) – (k+1)rk+1/(1-rk+1) • R = N/X = N/leff • where leff = l(1-Pk) = effective arrival rate • and Pk = rk(1-r)/(1-rk+1) = probability of a full queue • Loss rate = l - leff John Chuang

19. M/M/1/k Response Time John Chuang

20. M/M/1/k Throughput John Chuang

21. Lecture Outline • Performance Metrics • Availability • Queuing theory • M/M/1 queue • Scalability • M/M/m queue John Chuang

22. Scalability • The capability of a system to increase total throughput under an increased load when resources (typically hardware) are added • Cost of additional resource • Performance degradation under increased load John Chuang

23. Scalability Example • Original web server: can process m requests/sec; accepts requests at l/sec • Now request rate increases to 10l/sec and web server is swamped (r = 10l/m)! • Need to add new hardware! John Chuang

24. Which is better? • Option 1: One big web server that can process 10m requests/sec • Option 2: Ten web servers, each can process m requests/sec; each accepts 10% of requests (l/sec per server) • Option 3: Ten web servers, each can process m requests/sec; share single queue (load balancer) that accepts requests at 10l/sec John Chuang

25. l l l l l l l l l l m m m m m m m m m m 10l m m m m m m m m m m Option 1: M/M/1 queue with big server Option 2: (ten M/M/1 queues) 10l 10m Option 3: M/M/10 queue John Chuang

26. M/M/m Queue (m Servers) • r = l/mm • N = mr + rf/(1-r) where and John Chuang

27. Which is Better? m = 10; m = 100; l = 50 Remember: Scalability is not just about performance! John Chuang