1 / 20

Adam Cataldo Tuesday, January 24, 2012

Quick Deploy A distributed systems approach to developer productivity. Adam Cataldo Tuesday, January 24, 2012. v. Outline. LinkedIn architecture Quick Deploy Cool technical details. LinkedIn. Largest professional networking site 135M+ members 14 languages

dewey
Download Presentation

Adam Cataldo Tuesday, January 24, 2012

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quick DeployA distributed systems approach to developer productivity Adam Cataldo Tuesday, January 24, 2012 v

  2. Outline • LinkedIn architecture • Quick Deploy • Cool technical details

  3. LinkedIn • Largest professional networking site • 135M+ members • 14 languages • 200+ countries and territories • Estimated 4B+ people searches per year

  4. LinkedIn stack at 20,000 feet • Online system for serving up data to clients • Profile pages, company pages, LinkedIn groups, etc. • Key metric: Latency • Offline system for creating insight from data • People you may know, contact degree of separation, search indexing, etc. • Key metric: Throughput

  5. Offline data processing Voldemort Hadoop Reduce Map Map Reduce Map Reduce Map

  6. Online web application Data Service Client Presentation Business Logic Datastore

  7. Growing Pains for the Online System As time has gone by: • We’ve added new services to support new features • We’ve added new services to make our site faster and more available • Our engineering productivity dropped

  8. Quick Deploy • Developers only locally deploy the services they actually modify • Every service your service depends on is available in some data center • This makes development move much faster! Angelika Clayton

  9. Dev Box Early Integration Datacenter Shared Frontend Browser QD Proxy Local Mid Tier Shared Data Service DB

  10. In Detail EI Frontend EI Data Service Dev-local Mid Tier Proxy Browser QD Cookie QD Cookie QD IC QD IC QD IC QD IC

  11. Proxy Behavior Read identifier from cookie Has QD cookie? Proxy receives request yes no Read identifier from IC Is service local? Route to developer’s local service yes no Route to service in EI

  12. QD Cookie and Invocation Context • A developer creates an identifier key, and registers his local services with the proxy • He identifies himself by adding his key to a cookie from his browser • When a front end gets a request with a QD cookie, it puts the key in the invocation context • Services pass the invocation context with every outbound request

  13. Shared data Huh? Change Alfred’s language to Japanese きんぎょ Oracle MySQL Voldemort Memcached

  14. ActiveMQ Dev Box Early Integration Datacenter Consumer Producer Produces message Consumes message Consumer Queue Never sees message

  15. Timers Dev Box Early Integration Datacenter RPC Target Schedules time-triggered RPC call Service Gets RPC call after timer triggers Service with Timer RPC Target Never gets RPC call

  16. Proxy in detail > 10,000 QPS! Admin Console Python/Flask Web App Proxy Apache Traffic Server Proxy Apache Traffic Server Routing Profiles Apache Zookeeper Proxy Apache Traffic Server

  17. Why Apache Zookeeper • Distributed file system for small files • Basically, a key-value data store, where the keys are file names • Clients can listen for file change and directory change events • Great for storing data • Also, great publish/subscribe system

  18. Dynamic Discovery Dev Box Early Integration Datacenter QD Proxy Browser Shared Frontend Local Mid Tier Shared Data Service DB

  19. Conclusion • The LinkedIn architecture scales well in production • Quick deploy makes the architecture scale for developers • 20% productivity gain across the engineering organization

  20. Thanks Rob Loh JimDumont Swee Lim Steve Ihde

More Related