Deconstructing SPECweb99

1. Deconstructing SPECweb99 Erich Nahum IBM T.J. Watson Research Center www.research.ibm.com/people/n/nahum nahum@us.ibm.com Erich Nahum

2. Talk Overview • Workload Generators • SPECweb99 • Methodology • Results • Summary and Conclusions Erich Nahum

3. Why Workload Generators? • Allows stress-testing and bug-finding • Gives us some idea of server capacity • Allows us a scientific process to compare approaches • e.g., server models, gigabit adaptors, OS implementations • Assumption is that difference in testbed translates to some difference in real-world • Allows the performance debugging cycle Measure Reproduce Fix and/or improve Find Problem The Performance Debugging Cycle Erich Nahum

4. How does W. Generation Work? • Many clients, one server • match asymmetry of Internet • Server is populated with some kind of synthetic content • Simulated clients produce requests for server • Master process to control clients, aggregate results • Goal is to measure server • not the client or network • Must be robust to conditions • e.g., if server keeps sending 404 not found, will clients notice? Requests Responses Erich Nahum

5. Problems with Workload Generators • Only as good as our understanding of the traffic • Traffic may change over time • generators must too • May not be representative • e.g., are file size distributions from IBM.com similar to mine? • May be ignoring important factors • e.g., browser behavior, WAN conditions, modem connectivity • Still, useful for diagnosing and treating problems Erich Nahum

6. What Server Workload Generators Exist? • Many. In order of publication: • WebStone (SGI) • SPECweb96 (SPEC) • Scalable Client (Rice Univ.) • SURGE (Boston Univ.) • httperf (HP Labs) • SPECweb99 (SPEC) • TPC-W (TPC) • WaspClient (IBM) • WAGON (IBM) • Not to mention those for proxies (e.g. polygraph) • Focus of this talk: SPECweb99 Erich Nahum

7. Why SPECweb99? • Has become the de-facto standard used in Industry: • 141 submissions in 3 years on the SPEC web site • Hardware: Compaq, Dell, Fujitsu, HP, IBM, Sun • OS’es: AIX, HPUX, Linux, Solaris, Windows NT • Servers: Apache, IIS, Netscape, Tux, Zeus • Used within corporations for performance, testing, and marketing • E.g., within IBM, used by AIX, Linux, and 390 groups • Begs the question: how realistic is it? Erich Nahum

8. Server Workload Characterization • Over the years, many observations have been made about Web server behavior: • Request methods • Response codes • Document Popularity • Document Sizes • Transfer Sizes • Protocol use • Inter-arrival times How well does SPECweb99 capture these characteristics? Erich Nahum

9. History: SPECweb96 • SPEC: Systems Performance Evaluation Consortium • Non-profit group with many benchmarks (CPU, FS) • Pay for membership, get source code • First attempt to get somewhat representative • Based on logs from NCSA, HP, Hal Computers • 4 classes of files: • Poisson distribution within each class Erich Nahum

10. SPECweb96 (cont) • Notion of scaling versus load: • number of directories in data set size doubles as expected throughput quadruples (sqrt(throughput/5)*10) • requests spread evenly across all application directories • Process based WG • Clients talk to master via RPC's • Does only GETS, no keep-alive www.spec.org/osg/web96 Erich Nahum

11. Evolution: SPECweb99 • In response to people "gaming" benchmark, now includes rules: • IP maximum segment lifetime (MSL) must be at least 60seconds • Link-layer maximum transmission unit (MTU) must not be larger than 1460bytes (Ethernet frame size) • Dynamic content may not be cached • not clear that this is followed • Servers must log requests. • W3C common log format is sufficient but not mandatory. • Resulting workload must be within 10% of target. • Error rate must be below 1%. • Metric has changed: • now "number of simultaneous conforming connections“:rate of a connection must be greater than 320 Kbps Erich Nahum

12. SPECweb99 (cont) • Directory size has changed: (25 + (400000/122000)* simultaneous conns) / 5.0) • Improved HTTP 1.0/1.1 support: • Keep-alive requests (client closes after N requests) • Cookies • Back-end notion of user demographics • Used for ad rotation • Request includes user_id and last_ad • Request breakdown: • 70.00 % static GET • 12.45 % dynamic GET • 12.60 % dynamic GET with custom ad rotation • 04.80 % dynamic POST • 00.15 % dynamic GET calling CGI code Erich Nahum

13. SPECweb99 (cont) • Other breakdowns: • 30 % HTTP 1.0 with no keep-alive or persistence • 70 % HTTP 1.1 with keep-alive to "model" persistence • still has 4 classes of file size with Poisson distribution • supports Zipf popularity • Client implementation details: • Master-client communication uses sockets • Code includes sample Perl code for CGI • Client configurable to use threads or processes • Much more info on setup, debugging, tuning • All results posted to web page, • including configuration & back end code www.spec.org/osg/web99 Erich Nahum

14. Methodology • Take a log from a large-scale SPECweb99 run • Take a number of available server logs • For each characteristic discussed in the literature: • Show what SPECweb99 does • Compare to results from the literature • Compare to results from a set of sample server logs • Render judgment on how well SPECweb99 does Erich Nahum

15. Sample Logs for Illustration We’ll use statistics generated from these logs as examples. Erich Nahum

16. Talk Overview • Workload Generators • SPECweb99 • Methodology • Results • Summary and Conclusions Erich Nahum

17. Request Methods • AW96, AW00, PQ00, KR01: majority are GETs, few POSTs • SPECweb99: No HEAD request, too many POSTS Erich Nahum

18. Response Codes • AW96, AW00, PQ00, KR01: Most are 200s, next 304’s • SPECweb99 doesn’t capture anything but 200 OK Erich Nahum

19. Resource Popularity • p(r) = C/r^alpha (alpha = 1 true Zipf; others “Zipf-like") • Consistent with CBC95, AW96, CB96, PQ00, KR01 • SPECweb99 does a good job here with alpha = 1 Erich Nahum

20. Resource (File) Sizes • Lognormal body, consistent with results from AW96, CB96, KR01. • SPECweb99 curve is sparse, 4 distinct regions Erich Nahum

21. Tails of the File Size • AW96, CB96: sizes have Pareto tail; Downey01: Sizes are lognormal. • SPECweb99 tail only goes to 900 KB (vs 10 MB for others) Erich Nahum

22. Response (Transfer) Sizes • Lognormal body, consistent with CBC95, AW96, CB96, KR01 • SPECweb99 doesn’t capture zero-byte transfers (304s) Erich Nahum

23. Transfer Sizes w/o 304’s • When 304’s removed, SPECweb99 much closer Erich Nahum

24. Tails of the Transfer Size • SPECweb99 tail is neither lognormal nor pareto • Again, max transfer is only 900 KB Erich Nahum

25. Inter-Arrival Times • Literature gives exponential distr. for session arrivals • KR01: Request inter-arrivals are pareto • Here we look at request inter-arrivals Erich Nahum

26. Tails of Inter-Arrival Times • SPECweb99 has pareto tail • Not all others do, but may be due to truncation • (e.g. log duration of only one day) Erich Nahum

27. HTTP Version • Over time, more and more requests are served using 1.1 • But SPECweb99 is much higher than any other log • Literature doesn’t look at this, so no judgments Erich Nahum

28. Summary and Conclusions • SPECweb99 has a mixed record depending on characteristic: • Methods: OK • Response codes: bad • Document popularity: good • File sizes: OK to bad • Transfer sizes: bad • Inter-arrival times: good • Main problems are: • Needs to capture conditional GETs with IMS for 304’s • Better file size distribution (smoother, larger) Erich Nahum

29. Future Work • Several possibilities for future work: • Compare logs with SURGE • More detail on HTTP 1.1 (requires better workload characterization, e.g. packet traces) • Dynamic content (e.g., TPC-W) (again, requires workload characterization) • Latter 2 will not be easy due to privacy, competitive concerns Erich Nahum

30. Probability • Graph shows 3 distributions with average = 2. • Note average median in some cases ! • Different distributions have different “weight” in tail. Erich Nahum

31. Important Distributions Some Frequently-Seen Distributions: • Normal: • (avg. sigma, variance mu) • Lognormal: • (x >= 0; sigma > 0) • Exponential: • (x >= 0) • Pareto: • (x >= k, shape a, scale k) Erich Nahum

32. Probability Refresher • Lots of variability in workloads • Use probability distributions to express • Want to consider many factors • Some terminology/jargon: • Mean: average of samples • Median : half are bigger, half are smaller • Percentiles: dump samples into N bins (median is 50th percentile number) • Heavy-tailed: • As x->infinity Erich Nahum

33. Session Inter-Arrivals • Inter-arrival time between successive requests • “Think time" • difference between user requests vs. ALL requests • partly depends on definition of boundary • CB96: variability across multiple timescales, "self-similarity", average load very different from peak or heavy load • SCJO01: log-normal, 90% less than 1 minute. • AW96: independent and exponentially distributed • KR01: session arrivals follow poisson distribution, but requests follow pareto with a=1.5 Erich Nahum

34. Protocol Support • IBM.com 2001 logs: • Show roughly 53% of client requests are 1.1 • KA01 study: • 92% of servers claim to support 1.1 (as of Sep 00) • Only 31% actually do; most fail to comply with spec • SCJO01 show: • Avg 6.5 requests per persistent connection • 65% have 2 connections per page, rest more. • 40-50% of objects downloaded by persistent connections Appears that we are in the middle of a slow transition to 1.1 Erich Nahum

35. WebStone • The original workload generator from SGI in 1995 • Process based workload generator, implemented in C • Clients talk to master via sockets • Configurable: # client machines, # client processes, run time • Measured several metrics: avg + max connect time, response time, throughput rate (bits/sec), # pages, # files • 1.0 only does GETS, CGI support added in 2.0 • Static requests, 5 different file sizes: www.mindcraft.com/webstone Erich Nahum

36. SURGE • Scalable URL Reference GEnerator • Barford & Crovella at Boston University CS Dept. • Much more worried about representativeness, captures: • server file size distributions, • request size distribution, • relative file popularity • embedded file references • temporal locality of reference • idle periods ("think times") of users • Process/thread based WG Erich Nahum

37. SURGE (cont) • Notion of “user-equivalent”: • statistical model of a user • active “off” time (between URLS), • inactive “off” time (between pages) • Captures various levels of burstiness • Not validated, shows that load generated is different than SpecWeb96 and has more burstiness in terms of CPU and # active connections www.cs.wisc.edu/~pb Erich Nahum

38. S-Client • Almost all workload generators are closed-loop: • client submits a request, waits for server, maybe thinks for some time, repeat as necessary • Problem with the closed-loop approach: • client can't generate requests faster than the server can respond • limits the generated load to the capacity of the server • in the real world, arrivals don’t depend on server state • i.e., real users have no idea about load on the server when they click on a site, although successive clicks may have this property • in particular, can't overload the server • s-client tries to be open-loop: • by generating connections at a particular rate • independent of server load/capacity Erich Nahum

39. S-Client (cont) • How is s-client open-loop? • connecting asynchronously at a particular rate • using non-blockingconnect() socket call • Connect complete within a particular time? • if yes, continue normally. • if not, socket is closed and new connect initiated. • Other details: • uses single-address space event-driven model like Flash • calls select() on large numbers of file descriptors • can generate large loads • Problems: • client capacity is still limited by active FD's • “arrival” is a TCP connect, not an HTTP request www.cs.rice.edu/CS/Systems/Web-measurement Erich Nahum

40. TPC-W • Transaction Processing Council (TPC-W) • More known for database workloads like TPC-D • Metrics include dollars/transaction (unlike SPEC) • Provides specification, not source • Meant to capture a large e-commerce site • Models online bookstore • web serving, searching, browsing, shopping carts • online transaction processing (OLTP) • decision support (DSS) • secure purchasing (SSL), best sellers, new products • customer registration, administrative updates • Has notion of scaling per user • 5 MB of DB tables per user • 1 KB per shopping item, 25 KB per item in static images Erich Nahum

41. TPC-W (cont) • Remote browser emulator (RBE) • emulates a single user • send HTTP request, parse, wait for thinking, repeat • Metrics: • WIPS: shopping • WIPSb: browsing • WIPSo: ordering • Setups tend to be very large: • multiple image servers, application servers, load balancer • DB back end (typically SMP) • Example: IBM 12-way SMP w/DB2, 9 PCs w/IIS: 1M $ www.tpc.org/tpcw Erich Nahum