1 / 8

An Introduction to Google Percolator

An Introduction to Google Percolator, what is it and what is it used for ? How is it different from Apache Hadoop MapReduce ?

semtechs
Download Presentation

An Introduction to Google Percolator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Google Percolator • What is it ? • What is it used for ? • Percolator Vs MapReduce • Architecture www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  2. Percolator – What is it ? • Incremental updates to Big Data • Developed by Google • Based on Google File System ( GFS ) • Provides transactions and locking • Faster than comparable Map Reduce • Developed by Google due to MapReduce limitations www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  3. Percolator – What is it used for ? • Iterative updates • No need to batch process • Update as data received • Data in multi petabyte range • Strong consistency needed • Improved latency ( 100 x ) • Reduced document age ( 50 % )‏ • Random access to big data repository www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  4. Percolator Vs MapReduce Percolator • Iterative • Transactions • Latency 100 x A • Incremental updates • Code in Java ( mainly )‏ • Google owned • Uses GFS Map Reduce • Batch Processing • No transactions • Latency A • Run time scales with data • Code in C++ • Open source • Uses HDFS www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  5. Percolator – Architecture • Applications are a sequence of observers • An observer is called via a notification • A notification is triggered when table data changes • Application calls TabletServer via RPC • TabletServer calls GFS ChunkServer

  6. Percolator – Architecture • Applications • Series of observers • Observer • Completes task • Updates table • Next Observer called • Via notification • Percolator worker • Scans for changes • Sends notifications

  7. Percolator – Architecture Actual worker diagram including time stamping and locking via Chubby lock server

  8. Contact Us • Feel free to contact us at • www.semtech-solutions.co.nz • info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems

More Related