1 / 9

Apache Kafka Plugin-ORIEN IT

To provide detailed knowledge on Spark And Scala and its prominence in data handling. • To build knowledge on the future trends and career scope by attaining our institutes Spark And Scala Course In Hyderabad. • Scope for employment opportunities with Spark And Scala training. • Sorting out all kinds of doubts regarding Spark And Scala with the experts. webpage:http://www.orienit.com/courses/Spark-and-Scala-Training-in-Hyderabad.html Blogs: http://www.kalyanhadooptraining.com/ http://kalyanbigdatatraining.blogspot.com/

joeconnors
Download Presentation

Apache Kafka Plugin-ORIEN IT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Blogs http://kalyanbigdatatraining.blogspot.com http://www.kalyanhadooptraining.com/ Address: Flat No 204, Annapurna BlockAditya Enclave, AmeerpetHyderabad - 500038Telangana, India. Quick Contact: info@OrienIT.com +91 040 6514 2345 +91 970 320 2345 www.orienit.com

  2. Apache Kafka Plugin The plugin enables us to reliably and efficiently stream large amounts of data/logs onto HBase using the Phoenix API. Apache Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. So, at a high level, producers send messages over the network to the Kafka cluster which in turn serves them up to consumers like this: We are providing PhoenixConsumer to recieves the messages from Kafka Producer. Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  3. Prerequisites: • Phoenix 4.8.0+ • Kafka 0.9.0.0+ • Installation & Setup: • Download and build Phoenix 4.8.0+ • Follow the instructions as specified here to build the project as the Kafka plugin. Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  4. Phoenix Consumer for RegexEventSerializer Example: Create a kafka-consumer-regex.properties file with below properties serializer=regex serializer.rowkeyType=uuid serializer.regex=([^\,]*),([^\,]*),([^\,]*) serializer.columns=c1,c2,c3 jdbcUrl=jdbc:phoenix:localhost table=SAMPLE1 ddl=CREATE TABLE IF NOT EXISTS SAMPLE1 (uid VARCHAR NOT NULL,c1 VARCHAR,c2 VARCHAR,c3 VARCHAR CONSTRAINT pk PRIMARY KEY(uid)) bootstrap.servers=localhost:9092 topics=topic1,topic2 poll.timeout.ms=100 Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  5. Phoenix Consumer for JsonEventSerializer Example: • Create a kafka-consumer-json.properties file with below properties serializer=json serializer.rowkeyType=uuid serializer.columns=c1,c2,c3 jdbcUrl=jdbc:phoenix:localhost table=SAMPLE2 ddl=CREATE TABLE IF NOT EXISTS SAMPLE2(uid VARCHAR NOT NULL,c1 VARCHAR,c2 VARCHAR,c3 VARCHAR CONSTRAINT pk PRIMARY KEY(uid)) bootstrap.servers=localhost:9092 topics=topic1,topic2 poll.timeout.ms=100 Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  6. Phoenix Consumer Execution Procedure: • Start the Kakfa Producer then send some messages • bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic1 • Start the PhoenixConsumer using below command • HADOOP_CLASSPATH=$(hbaseclasspath):/path/to/hbase/conf hadoop jar phoenix-kafka-<version>-minimal.jar org.apache.phoenix.kafka.consumer.PhoenixConsumerTool --file /data/kafka-consumer.properties • The input file must be present on HDFS (not the local filesystem where the command is being run). Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  7. Configuration: Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  8. Configuration: Blog: http://kalyanbigdatatraining.blogspot.com/ www.orienit.com 9703202345

  9. Facebook: https://www.facebook.com/OrienITinstitute/ Twitter: https://twitter.com/Orien_IT Website: www.orienit.com 9703202345

More Related