Adaptive Performance Optimization for Large Scale Web Applications

1. Adaptive Performance Optimization for Large Scale Web Applications • Xiaoliang “David” Wei • Infrastructure, Facebook • www.facebook.com/DavidWei

2. Agenda

3. About Facebook: the challenges

4. Facebook The largest social network

5. Facebook: large scale • A web application with more than 400M users • A large web site • > 9 billion page views per days ( 1 msec / page => 2 human life of time a year ) • most traffiked site • A huge graph: • # of friends for a typical user: 130 • # of edges of the social graph: 130 * 400M ~ 50 Billions • A massive storage: • 25 billions of new contents each month • The large photo site

6. Facebook: viral growth • An open platform • 3rd party developers: > 1 millions • Active applications: > 550 K • Websites with Facebook Connect: > 250 K • Widgets all around the web

7. Facebook: fast evolution • A fast-moving product • Agile development: • One new revision each week • Patches pushed everyday • Urgent fixes 24/7 • Parallel product launches • > 100 experimental launches at the same time

8. Facebook: challenges • A unique large scale system • Large scale • No simple problem • Viral growth • No good assumption on growth and adoption • Fast evolution • Very small time window for product optimization

9. Facebook: under the hood • A relatively small team • ~400 engineers • Ratio: 1 engineer for 1 million users • Millions of lines of PHP codes • 2000 + Javascripts served from Facebook pages • 1000 + style sheets • Tens of thousands of static images (icons and etc)

10. Web application infrastructure

11. Life of a web page request • User clicks on www.facebook.com Web Browser

12. Life of a web page request • User clicks • HTTP request sent to the server • DNS • TCP Handshake • HTTP Request (Path, parameter, cookies and etc) Web Browser NW Delay Servers

13. Life of a web page request • User clicks • HTTP request sent to the server • Web server accept the requests • Validation • Data retrieval • HTML generation • Post processing • Compression • Log Web Browser NW Delay Servers Server Delay

14. Life of a web page request • User clicks • HTTP request sent to the server • Web server accept the response • HTTP response sent to client • Basic HTML • External Images Link • External CSS: style control • External JS: user interaction Web Browser NW Delay Servers Server Delay

15. Life of a web page request • User clicks • HTTP request sent to the server • Web server accept the response • HTTP response sent to client • Browser renders: • Decompression • Analyze HTML, construct DOM • Load images, CSS, JS • Present the contents Web Browser Content Distribution Network (CDN) Client Delay NW Delay Servers Server Delay

16. Life of a web page request • The latency of a web page request: • Client Delay • Network Delay • Server Delay Web Browser Content Distribution Network (CDN) Client Delay NW Delay Servers Server Delay

17. Life of a web page request • The latency of a web page request: • Client Delay: 50% • Network Delay: 25% • Server Delay: 25% Web Browser Content Distribution Network (CDN) Client Delay NW Delay Servers Server Delay

18. Web service: a distributed system • Questions: • Scalability • Consistency • Performance • … Web Browser Content Distribution Network (CDN) Client Delay NW Delay Servers Server Delay

19. Servers: at the beginning • LAMP: • Platform: Linux • Web Server: Apache • Database: MySQL • Business Logic: PHP Web Browser Servers LAMP

20. Servers: more users… Web Server Web Browser MySQL Servers

21. Servers: even more users… Web Server Web Browser Memcache Server MySQL Servers

22. Servers: even more users… Web Server Web Server Web Server Web Browser Memcache Servers Memcache Servers MySQL MySQL MySQL Servers

23. Servers: even more users… Web Server Web Server Web Server Web Browser Memcache Servers Memcache Servers MySQL MySQL MySQL Servers

24. Static Content Distribution Apache+ Cache Apache+ Cache Apache+ Cache Web Browser Content Distribution Network (CDN) Distributed File Systems Servers

25. Static Content Distribution HTTP Server+ Cache HTTP Server+ Cache HTTP Server+ Cache Web Browser Content Distribution Network (CDN) Cache Cache Servers Distributed file System optimized for reads / costs

26. Client side: • 60％～90％ of the end-to-end latency (while traditionally ignored) • Hostile environment: • IE6 / IE7 / IE8 / FF3/ Safari4 / Chrome … • Browser Add-ons • 3rd party scripts Web Browser Content Distribution Network (CDN) Servers

27. Web application performance

28. Performance impacts business • Google: +400ms => -0.6% daily search per user • Bing: +500ms => -1% click, +1.2 sec time to click (2x) • Facebook real time feed stories at summer 2009 (all 50-percentile): • +500ms (~ 25% of full latency) => -3% click • +1 sec ( ~50% of full latency) => -6% click

29. Status of web performance optimization • Decade-old protocols: TCP, HTTP, HTML • constraints & workaround • Fast-changing industry: browser war • Highly heterogeneous environments • Some rules & tools emerging (e.g. Steve Souder’s books, YSlow) • Large scale optimization is difficult

30. Web performance: some best-practice rules • Minimize HTTP requests (TCP initialization overhead) • Gzip resources (Bandwidth optimization) • Preload Components • Reduce DNS Lookup • Split Requests Across Domain (HTTP limits + TCP inefficiency)

31. Web performance: trade-offs • Minimize HTTP requests  Increased unused bytes • Gzip resources  Increased CPU utilization • Preload Components  Increased bandwidth cost • Reduce DNS Lookup  Split Requests Across Domain • Split Requests Across Domain  Reduce DNS Lookup

32. Web performance optimization • Improve speed • Reduce cost (both latency penalty and $ cost) • Operate on the equilibrium • Easy for a single page, hard for large scale systems

33. Web performance: the gap • Large scale • Viral growth • Fast evolution “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Large Scale Operation

34. Web performance: fill the gap • Large scale • Viral growth • Fast evolution “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Interface, Model, Adaptive Infrastructure Large Scale Operation

35. Cycle of web performance R&D @ Facebook

36. Adaptive Performance Optimization

37. Adaptive Performance optimization • Example: “Best Practices for Speeding Up You Web Site” – YSlow Rule 1: Minimize HTTP Request • Combined files: package Javscript and Stylesheets • Sprite Images • …

38. Performance optimization: Case Study • Day 1: Some smart engineers start a project! <Print CSS tag for feature A> <Print CSS tag for feature B> <Print CSS tag for feature C> <print HTML of feature A> <print HTML of feature B> <print HTML of feature C> … “Let’s write a new page with features A, B and C!”

39. Day 2: Some smart engineers read YUI blog and says… <Print CSS tag for feature A> <Print CSS tag for feature B> <Print CSS tag for feature C> <print HTML of feature A> <print HTML of feature B> <print HTML of feature C> … “A & B & C are always used; let’s package them together!”

40. Day 2: Awesome! <Print CSS tag for feature A&B&C in a single package> <print HTML of feature A> <print HTML of feature B> <print HTML of feature C> …

41. Day 3: feature C evolves… <Print CSS tag for feature A&B&C in a single package> <print HTML of feature A> <print HTML of feature B> If (users_signup_for_C()) { <print HTML of feature C>} …

42. Day 3: <Print CSS tag for feature A&B&C> <print HTML of feature A> <print HTML of feature B> If (users_signup_for_C()) { <print HTML of feature C>} … A&B are always used, while C is not. ..

43. Day 4: feature C is deprecated <Print CSS tag for feature A&B&C> <print HTML of feature A> <print HTML of feature B> // no one uses C { <print HTML of feature C>} …

44. Day 4: we start to send unused bits <Print CSS tag for feature A&B&C> <print HTML of feature A> <print HTML of feature B> // no one uses C { <print HTML of feature C>} … It is hard to remember we should remove C here.

45. One months later… <Print CSS tag for feature A&B&C&D&E&F&G…> if (F is used) <print HTML of feature F> <print HTML of feature G> if (F is not used) { <print HTML of feature E>} … Thousands of dead CSS bytes in the package.

46. Performance Optimization: the gap • One months later… <Print CSS tag for feature A&B&C&D&E&F&G…> if (F is used) <print HTML of feature F> <print HTML of feature G> if (F is not used) { <print HTML of feature E>} … “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Large Scale Operation

47. Adaptive Static Resource Optimization Fill the gap: • Build interface to separate requirement declaration and delivery of static resources • Model the trade-off of optimization • Infrastructure to analyze requirement logs and globally optimize the delivery performance “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Interface, Model, and Infrastructure Large Scale Operation

48. Adaptive Packager: Designs

49. Interfaces • Back to Day 1: require_static(A_css); <render HTML of feature A> require_static(B_css); <render HTML of feature B> require_static(C_css);<render HTML of feature C> <deliver all required CSS> <print all rendered HTML> Separate Declaration from actual Delivery Requirement Declaration lives with HTML Global Optimization on Delivery

50. Packager: Global JS/CSS Optimization Offline analysis Online process require_static(A_css); <render HTML of feature A> require_static(B_css); <render HTML of feature B> require_static(C_css); <render HTML of feature C> <deliver all required CSS> <print all rendered HTML> Usage Pattern logs Packaging algorithm “Optimal” packages