160 likes | 250 Views
Background Processing In Ruby. Khash Sajadi @ khash. Do I need them?. HTTP downloads Image Resizing Search Index Updates Batch Jobs Spam Checks Uploads Provisioning of a Box Sending Emails Throttling Calls Housekeeping Jobs. What’s on the menu?. Delayed Job / Navvy
E N D
Background Processing In Ruby Khash Sajadi @khash
Do I need them? • HTTP downloads • Image Resizing • Search Index Updates • Batch Jobs • Spam Checks • Uploads • Provisioning of a Box • Sending Emails • Throttling Calls • Housekeeping Jobs
What’s on the menu? • Delayed Job / Navvy • Resque / Sidekiq • DIY (EventMachine, CRON, Spawn…) • Hosted Solutions (IronWorker)
Delayed Job / Navvy • ActiveRecord based (Delayed Job) • DB agnostic (Navvy) • Ruby object with Perform method
Delayed Job / Navvy classMyJob < Struct.new(:text, :emails) defperform emails.each { |e| MyMailer.deliver_text_to_email(text, e) } end end Delayed::Job.enqueueMyJob.new( 'loremipsum...', Users.all.collect(&:email) ) rake jobs:work
Resque • Redis backed • Very similar usage to Delayed Job
Resque classArchive @queue = :file_serve defself.perform(repo_id, branch = 'master') repo = Repository.find(repo_id) repo.create_archive(branch) end end Resque.enqueue(Archive, self.id, branch) rake resque:work QUEUE=*
Resque • Resque Scheduler • ResqueWeb
Testing • resque_unit • Delayed Job kill switch
Forking Threads • Resque vs. Sidekiq • MRI < 1.9 and MRI > 1.9 • REE and JRuby
Keep it up • God, Bluepill, upstart • Procfile • Foreman worker: rake rescue:work QUEUE=* scheduler: rake resque:scheduler
Let’s grow • Vertical Scaling (more workers) • Horizontal Scaling (more boxes) • Error / Dead Letter Queues • Auto Scale
Gotcha! • Always use IDs • Beware of serialization • Don’t hog workers • Be stateless / idempotent • Breakup jobs to atomic parts • Beware of concurrency • Logging and exception handling
Beyond Jobs • Use a solid middleware ( RabbitMQ, ActiveMQ, SQS,…) • Transactions • Distributed transactions
Further Reading • Background Jobs in Ruby on Rails http://4loc.wordpress.com/2010/03/10/background-jobs-in-ruby-on-rails/ • Sidekiq vs. Resquehttp://joshrendek.com/2012/11/sidekiq-vs-resque/
Thank you! Khash Sajadi @khash Download the slides at: bit.ly/T9oJNN