Scalable Internet Services

1. Scalable Internet Services Cluster Lessons and Architecture Design for Scalable Services BR01, TACC, Porcupine, SEDA and Capriccio

2. Outline • Overview of cluster services • lessons from giant-scale services (BR01) • SEDA (staged event-driven architecture) • Capriccio ravenben@cs.ucsb.edu

3. Scalable Servers • Clustered services • natural platform for large web services • search engines, DB servers, transactional servers • Key benefit • low cost of computing, COTS vs. SMP • incremental scalability • load balance traffic/requests across servers • Extension from single server model • reliable/fast communication, but partitioned data ravenben@cs.ucsb.edu

4. Goals • Failure transparency • hot-swapping components w/o loss of avail • homogeneous functionality and/or replication • Load balancing • partition data / requests for max service rate • need to colocate requests w/ associated data • Scalability • aggregate performance should scale w/# of servers ravenben@cs.ucsb.edu

5. Two Different Models • Read-mostly data • web servers, DB servers, search engines (query) • replicate across servers + (RR DNS / redirector) client client client client client IP Network (WAN) Round Robin DNS ravenben@cs.ucsb.edu

6. Two Different Models … • Read-write model • mail servers, e-commerce sites, hosted services • small(er) replication factor for stronger consistency client client client client client IP Network (WAN) Load Redirector ravenben@cs.ucsb.edu

7. Key Architecture Challenges • Providing high availability • availability across component failures • Handling flash crowds / peak load • need support for massive concurrency • Other challenges • upgradability: maintaining availability and minimal cost during upgrades in S/W, H/W, functionality • error diagnosis: fast isolation of failures / performance degradation ravenben@cs.ucsb.edu

8. Nuggets • Definition • uptime = (MTBF – MTTR)/MTBF • yield = queries completed / queries offered • harvest = data available / complete data • MTTR • at least as important at MTBF • much easier to tune and quantify • DQ principle • data/query x queries/second  constant • physical bottlenecks limit overall throughput ravenben@cs.ucsb.edu

9. Staged Event-driven Architecture • SEDA (SOSP’05) ravenben@cs.ucsb.edu

10. Break… • Come back in 5 mins • more on threads vs. events… ravenben@cs.ucsb.edu

11. applications application programming interface Dynamic Tap. core router Patchwork distance map network SEDA event-driven framework Java Virtual Machine Tapestry Software Architecture ravenben@cs.ucsb.edu

12. ? A ? B ? C Network ? ? ? ? Impact of Correlated Events + + = • web / application servers • independent requests • maximize individual throughput event handler • correlated requests: A+B+CD • e.g. online continuous queries, sensor aggregation, p2p control layer, streaming data mining ravenben@cs.ucsb.edu

13. Capriccio • User-level light-weight threads (SOSP03) • Argument • threads are the natural programming model • current problems result of implementation • not fundamental flaw • Approach • aim for massive scalability • compiler assistance • linked stacks, block graph scheduling ravenben@cs.ucsb.edu

14. The Price of Concurrency • Why is concurrency hard? • Race conditions • Code complexity • Scalability (no O(n) operations) • Scheduling & resource sensitivity • Inevitable overload • Performance vs. Programmability • No good solution Threads Ideal Ease of Programming Events Threads Performance ravenben@cs.ucsb.edu

15. The Answer: Better Threads • Goals • Simple programming model • Good tools & infrastructure • Languages, compilers, debuggers, etc. • Good performance • Claims • Threads are preferable to events • User-Level threads are key ravenben@cs.ucsb.edu

16. “But Events Are Better!” • Recent arguments for events • Lower runtime overhead • Better live state management • Inexpensive synchronization • More flexible control flow • Better scheduling and locality • All true but… • Lauer & Needham duality argument • Criticisms of specific threads packages • No inherent problem with threads! ravenben@cs.ucsb.edu

17. Criticism: Runtime Overhead • Criticism: Threads don’t perform well for high concurrency • Response • Avoid O(n) operations • Minimize context switch overhead • Simple scalability test • Slightly modified GNU Pth • Thread-per-task vs. single thread • Same performance! ravenben@cs.ucsb.edu

18. Criticism: Synchronization • Criticism: Thread synchronization is heavyweight • Response • Cooperative multitasking works for threads, too! • Also presents same problems • Starvation & fairness • Multiprocessors • Unexpected blocking (page faults, etc.) • Both regimes need help • Compiler / language support for concurrency • Better OS primitives ravenben@cs.ucsb.edu

19. Criticism: Scheduling • Criticism: Thread schedulers are too generic • Can’t use application-specific information • Response • 2D scheduling: task & program location • Threads schedule based on task only • Events schedule by location (e.g. SEDA) • Allows batching • Allows prediction for SRCT • Threads can use 2D, too! • Runtime system tracks current location • Call graph allows prediction Task Program Location Events Threads ravenben@cs.ucsb.edu

20. 900 800 KnotC (Favor Connections) KnotA (Favor Accept) 700 Haboob Mbits / second 600 500 400 300 200 100 0 1 4 16 64 256 1024 4096 16384 Concurrent Clients The Proof’s in the Pudding • User-level threads package • Subset of pthreads • Intercept blocking system calls • No O(n) operations • Support > 100K threads • 5000 lines of C code • Simple web server: Knot • 700 lines of C code • Similar performance • Linear increase, then steady • Drop-off due to poll() overhead ravenben@cs.ucsb.edu

21. Arguments For Threads • More natural programming model • Control flow is more apparent • Exception handling is easier • State management is automatic • Better fit with current tools & hardware • Better existing infrastructure ravenben@cs.ucsb.edu

22. Why Threads: control Flow • Events obscure control flow • For programmers and tools Web Server AcceptConn. ReadRequest PinCache ReadFile WriteResponse Exit ravenben@cs.ucsb.edu

23. Why Threads: control Flow • Events obscure control flow • For programmers and tools Web Server AcceptConn. ReadRequest PinCache ReadFile WriteResponse Exit ravenben@cs.ucsb.edu

24. Why Threads: Exceptions • Exceptions complicate control flow • Harder to understand program flow • Cause bugs in cleanup code Web Server AcceptConn. ReadRequest PinCache ReadFile WriteResponse Exit ravenben@cs.ucsb.edu

25. Why Threads: State Management • Events require manual state management • Hard to know when to free • Use GC or risk bugs Web Server AcceptConn. ReadRequest PinCache ReadFile WriteResponse Exit ravenben@cs.ucsb.edu

26. Why Threads: Existing Infrastructure • Lots of infrastructure for threads • Debuggers • Languages & compilers • Consequences • More amenable to analysis • Less effort to get working systems ravenben@cs.ucsb.edu

27. Building Better Threads • Goals • Simplify the programming model • Thread per concurrent activity • Scalability (100K+ threads) • Support existing APIs and tools • Automate application-specific customization • Mechanisms • User-level threads • Plumbing: avoid O(n) operations • Compile-time analysis • Run-time analysis ravenben@cs.ucsb.edu

28. Case for User-Level Threads • Decouple programming model and OS • Kernel threads • Abstract hardware • Expose device concurrency • User-level threads • Provide clean programming model • Expose logical concurrency • Benefits of user-level threads • Control over concurrency model! • Independent innovation • Enables static analysis • Enables application-specific tuning App User Threads OS ravenben@cs.ucsb.edu

29. Case for User-Level Threads • Decouple programming model and OS • Kernel threads • Abstract hardware • Expose device concurrency • User-level threads • Provide clean programming model • Expose logical concurrency • Benefits of user-level threads • Control over concurrency model! • Independent innovation • Enables static analysis • Enables application-specific tuning App User Threads OS Similar argument to the design of overlay networks ravenben@cs.ucsb.edu

30. Capriccio Internals • Cooperative user-level threads • Fast context switches • Lightweight synchronization • Kernel Mechanisms • Asynchronous I/O (Linux) • Efficiency • Avoid O(n) operations • Fast, flexible scheduling ravenben@cs.ucsb.edu

31. Safety: Linked Stacks • The problem: fixed stacks • Overflow vs. wasted space • LinuxThreads: 2MB/stack • Limits thread numbers • The solution: linked stacks • Allocate space as needed • Compiler analysis • Add runtime checkpoints • Guarantee enough space until next check Fixed Stacks overflow waste Linked Stack ravenben@cs.ucsb.edu

32. Linked Stacks: Algorithm • Parameters • MaxPath • MinChunk • Steps • Break cycles • Trace back • chkpts limit MaxPath length • Special Cases • Function pointers • External calls • Use large stack 3 3 2 5 2 4 3 6 MaxPath = 8 ravenben@cs.ucsb.edu

33. Linked Stacks: Algorithm • Parameters • MaxPath • MinChunk • Steps • Break cycles • Trace back • chkpts limit MaxPath length • Special Cases • Function pointers • External calls • Use large stack 3 3 2 5 2 4 3 6 MaxPath = 8 ravenben@cs.ucsb.edu

34. Linked Stacks: Algorithm • Parameters • MaxPath • MinChunk • Steps • Break cycles • Trace back • chkpts limit MaxPath length • Special Cases • Function pointers • External calls • Use large stack 3 3 2 5 2 4 3 6 MaxPath = 8 ravenben@cs.ucsb.edu

35. Linked Stacks: Algorithm • Parameters • MaxPath • MinChunk • Steps • Break cycles • Trace back • chkpts limit MaxPath length • Special Cases • Function pointers • External calls • Use large stack 3 3 2 5 2 4 3 6 MaxPath = 8 ravenben@cs.ucsb.edu

36. Linked Stacks: Algorithm • Parameters • MaxPath • MinChunk • Steps • Break cycles • Trace back • chkpts limit MaxPath length • Special Cases • Function pointers • External calls • Use large stack 3 3 2 5 2 4 3 6 MaxPath = 8 ravenben@cs.ucsb.edu

37. Linked Stacks: Algorithm • Parameters • MaxPath • MinChunk • Steps • Break cycles • Trace back • chkpts limit MaxPath length • Special Cases • Function pointers • External calls • Use large stack 3 3 2 5 2 4 3 6 MaxPath = 8 ravenben@cs.ucsb.edu

38. Special Cases • Function pointers • categorize f* by # and type of arguments • “guess” which func will/can be called • External functions • users annotate trusted stack bounds on libs • or (re)use a small # of large stack chunks • Result • use/reuse stack chunks much like VM • can efficiently share stack chunks • memory-touch benchmark, factor of 3 reduction in paging cost ravenben@cs.ucsb.edu

39. Scheduling: Blocking Graph • Lessons from event systems • Break app into stages • Schedule based on stage priorities • Allows SRCT scheduling, finding bottlenecks, etc. • Capriccio does this for threads • Deduce stage with stack traces at blocking points • Prioritize based on runtime information Web Server Accept Read Open Read Close Write Close ravenben@cs.ucsb.edu

40. Resource-Aware Scheduling • Track resources used along BG edges • Memory, file descriptors, CPU • Predict future from the past • Algorithm • Increase use when underutilized • Decrease use near saturation • Advantages • Operate near the knee w/o thrashing • Automatic admission control Web Server Accept Read Open Read Close Write Close ravenben@cs.ucsb.edu

41. Pitfalls • What is the max amt of resource? • depends on workload • e.g.: disk thrashing depends on sequential or random seeks • use early signs of thrashing to indicate max capacity • Detecting thrashing • only estimate using “productivity/overhead” • productivity from guessing (threads created, files opened/closed) ravenben@cs.ucsb.edu

42. Thread Performance • Slightly slower thread creation • Faster context switches • Even with stack traces! • Much faster mutexes Time of thread operations (microseconds) ravenben@cs.ucsb.edu

43. Runtime Overhead • Tested Apache 2.0.44 • Stack linking • 78% slowdown for null call • 3-4% overall • Resource statistics • 2% (on all the time) • 0.1% (with sampling) • Stack traces • 8% overhead ravenben@cs.ucsb.edu

44. Microbenchmark: Producer / Consumer ravenben@cs.ucsb.edu

45. Web Server Performance ravenben@cs.ucsb.edu

46. Example of “Great Systems Paper” • observe higher level issue • threads vs. event programming abstraction • use previous work (duality) to identify problem • why are threads not as efficient as events? • good systems design • call graph analysis for linked stacks • resource aware scheduling • good execution • full solid implementation • analysis leading to full understanding of detailed issues • cross-area approach (help from PL research) ravenben@cs.ucsb.edu

47. Acknowledgements • Many slides “borrowed” from the respective talks / papers: • Capriccio (Rob von Behren) • SEDA (Matt Welsh) • Brewer01: “Lessons…” ravenben@cs.ucsb.edu