Fast Data at Massive Scale

2. Fast Data at Massive Scale Lessons Learned at Facebook Bobby Johnson

3. Me • Director of Engineering • Scaling and Performance • Site Security • Site Reliability • Distributed Systems • Development tools • Customer Service Tools • Took Facebook from 7M users to 120M.

5. Architecture Load Balancer (assigns a web server) Other services Search, Feed, etc (ignore for now) Web Server (PHP assembles data) Memcache (fast) Database (slow, persistent)

6. 1/2 the time is in PHP • 1/4 is in memcache • 1/8 is in database

7. One year ago, almost half the time was memcache

8. Network Incast memcache memcache memcache memcache Switch Many Small Get Requests PHP Client

9. Network Incast memcache memcache memcache memcache Switch Many big data packets PHP Client

10. Clustering memcache 10 objects PHP Client 1 round trip for 10 objects

11. Clustering memcache memcache 5 objects 5 objects PHP Client • 2 round trips total • 1 round trip per server • longest request is 5

12. Clustering memcache memcache memcache 3 objects 4 objects 3 objects PHP Client • 3 round trips total • 1 round trip per server • longest request is 4

13. Clustering • If objects are small, round trips dominate so you want objects clustered • If objects are large, transfer time dominates so you want objects distributed • In a web application you will almost always be dealing with small objects

14. Caching • Basic tools are parallelism and clustering • Clustering is a latency/throughput tradeoff • Application code must be aware • Networking is a burst problem • Dropped packets kill you • TCP quick ack

15. PHP CPU

16. Application Improvements

17. know what your libraries do $results = get_search_results( $needle ); foreach ( $results as $result ) { if ( is_pending_friend( $result[‘id’] ) ) { // we’ll change the links based on this $result[‘pending’] = true; } }

18. know what your libraries do function is_pending_friend( $id ) { // this is short-lived, so don’t cache expensive_db_query( $id …)

19. Databases • Tend to be slower than lighter weight alternatives, so avoid using them • If you do use them partition them right from the start • If a query is _really_ slow, like a few seconds or a few minutes, you probably have a bug where you’re scanning a table • The db should have a command to tell you what index it’s using for a query, and how many rows it’s examining

20. General Lessons • Your best tool is parallelism • Look at your data • Build tools to look at your data • Don’t make assumptions about what components are doing • Algorithmic and system improvements are almost always better than micro-optimization