120 likes | 247 Views
Twitter Copes with Web2.0rhea. The Twits are catching up…. Alex Carian ISM 158 May 6, 2010. From TheRegister.co.uk: “Twitter: It's the end of the sysadmin as we know it”.
E N D
Twitter Copes with Web2.0rhea The Twits are catching up… Alex Carian ISM 158 May 6, 2010
From TheRegister.co.uk:“Twitter: It's the end of the sysadmin as we know it” • “"This is a whole new world… For the longest time, people ran large data systems on a kind of ad hoc basis. We're in a world now where so many people are depending on the real-time web. A system administrator is not just a system administrator anymore.” 1 John Adams, Ops Engineer
The Real-Time Web? “You have to use analytics. You have to grab data. You have to look at where a site is trending and where things are going so you can scale... If you don't start doing this work early, if you don't start collecting this early, you will fail.” - John Adams, Twitter Ops Engineer - Ad hoc solution: designed for a specific problem or task, cannot be adapted to other purposes 3 • Initially, the core of Twitter’s code ran: • Ruby on Rails (application framework) • Apache/Mongrel (web servers)
Twitter’s Aches and Pains - Despite the benefits of the RoR environment, Twitter experiences outages, most notably during Steve Jobs’ Keynote address at Macworld ‘08 3 • A developer, in his spare time, ports a section of the website code into the Scala programming language • Allows for object-oriented and functional programming paradigms • Task that originally took 2.5 hrs now takes… • 20 seconds!
Functional Bits • Twitter maintains Ruby on Rails for building “user facing features” and MySQL for database management1 • Now entrusts back end infrastructure to Unicorn, a RoR application server • Grocery Store model • Still open source • Still based on Mongrel (the former app server)
Solipsism 2.0 • Twitter saw a 1358% increase in traffic in 2009 (comScore) • Utilizes a monitoring system known as ‘Ganglia’ • open source program grown out of UCB and NSF grants • Tracks 15,000 points of site performance, including grid load, cluster load, # of hosts up/down
TWITPOCALYPSE?! • Many suspected Twitter’s service would come unwound after reaching the Unique Tweet ID limit • (2^31 = 4,294,967,295) The “Fail Whale” • Twitter’s ops team mined site data to predict when the limit would be reached • Allowed the team to implement necessary changes to avoid complications • Current number of Tweets posted: ~13,475,000,000 5
Another Word on Sysadmin 2.0… - Hard drive? That’s so 2009…. Or 1959… “"Another discovery that we made when trying to increase the scale of Twitter was that disk is the new tape," he said. "With any sort of social networking operation - juggling followers, sending mail, etc. - disk is extremely slow.” 1 -John Adams • Twitter now uses memcached • Data and objects cached in RAM to reduce the number of times an external data source (such as a database or API) must be read. 4 • Don’t overdo it.
Twitter’s Open Back End • Company uses open sourced technology in most operations • RoR, Unicorn, Ganglia, • Developed several tools at home • Kestrel, message queue server • FlockDB, follower database • Both now released as open source Will Twitter maintain its open source track record?
Sources Cited Metz, Cade. “Twitter: It's the end of the sysadmin as we know it”. The Register, May 4, 2010. <http://www.theregister.co.uk/2010/05/04/twitter_and_the_end_of_the_sysadmin_as_we_know_it/> Metz, Cade. “Twitter jilts Ruby for Scala” The Register, April 1, 2009. http://www.theregister.co.uk/2009/04/01/twitter_on_scala/ Riley, Duncan. “Twitter Fails Macworld Keynote Test” TechCrunch, Jan 2008. <http://techcrunch.com/2008/01/15/twitter-fails-macworld-keynote-test/> “Ad hoc”. Wikipedia “memcached”. Wikipedia “Twitter Twitpocalypse Status”. Accessed May 6, 2010. <http://www.twitpocalypse.com>
Sources Cited Metz, Cade. “Twitter: It's the end of the sysadmin as we know it”. The Register, May 4, 2010. <http://www.theregister.co.uk/2010/05/04/twitter_and_the_end_of_the_sysadmin_as_we_know_it/> Metz, Cade. “Twitter jilts Ruby for Scala” The Register, April 1, 2009. http://www.theregister.co.uk/2009/04/01/twitter_on_scala/ Riley, Duncan. “Twitter Fails Macworld Keynote Test” TechCrunch, Jan 2008. <http://techcrunch.com/2008/01/15/twitter-fails-macworld-keynote-test/> “Ad hoc”. Wikipedia “memcached”. Wikipedia “Twitter Twitpocalypse Status”. Accessed May 6, 2010. <http://www.twitpocalypse.com>