1 / 8

An Introduction to Apache Flume

An Introduction to Apache Flume, what is it used for and how does it work ? How does it fit into the Hadoop tool set ?

semtechs
Download Presentation

An Introduction to Apache Flume

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Apache Flume • What is it ? • How does it work ? • Architecture • Reliability www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  2. Flume – What is it ? • A data collection service for Hadoop • For distributed systems • Open source • Scaleable • Reliable • Manageable • Fault tolerant www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  3. Flume – How does it work ? • Flumes uses agents which have • A source • Listen for events • Write events to channel • A channel • Queue event data as transactions • A sink • Write event data to target i.e. HDFS • Remove event from queue www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  4. Flume – Architecture • A single agent showing its parts • Generally one agent for a given data type www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  5. Flume – Architecture • Agents can be chained into flows • Avro can be used for data serialization www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  6. Flume – Architecture In complicated flows it may be necessary to think about • Event Data Reliability • Should we have • Complete end to end reliability • Send and forget • Or something in between ? www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  7. Flume – Architecture • Complex flows may have many links www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  8. Contact Us • Feel free to contact us at • www.semtech-solutions.co.nz • info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems

More Related