1 / 7

An Introduction to Apache Avro

An Introduction to Apache Avro, what is it and how is it used ? Plus some code examples to show JSON etc.

semtechs
Download Presentation

An Introduction to Apache Avro

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Apache Avro • What is it ? • Language API's • JSON Example • Schema Evolution • Avro Use Example www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  2. Avro – What is it ? • Data serialization system • Uses JSON based schemas • Uses RPC calls to send data • Schema's sent during data exchange • Integrated with many languages • Fast binary data format or encode with JSON www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  3. Avro – Language API's Avro API's exist for the following languages • Java • C# • C • C++ • Python • Ruby www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  4. Avro – JSON Example This text shows an example JSON schema for a customer record { "namespace": "customer.avro", "type": "record", "name": "customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  5. Avro – Schema Evolution In a fast changing environment it is possible to • Write data to file with a schema • Change the schema • Add extra fields • Delete fields • Rename fields • And still read written file with changed schema www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  6. Avro – Use Example // define schema for customer { "namespace": "customer.avro", "type": "record", "name": "Customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } // compile the schema java -jar /....../avro-tools-1.7.4.jar compile schema customer.avsc . // create a couple of customers Customer cust1 = new Customer("Dave", 33, "12 redmont street", "021234675", "davef@somedom@co.nz" ); Customer cust2 = new Customer("Emily", 27, "13a lake cresent", "022876234", "esuanders@somenz@co.nz" ); // now serialize customers to disk File file = new File("customers.avro"); DatumWriter<Customer> userDatumWriter = new SpecificDatumWriter<Customer>(Customer.class); DataFileWriter<Customer> dataFileWriter = new DataFileWriter<Customer>(customerDatumWriter); dataFileWriter.create(cust1.getSchema(), new File("customers.avro")); dataFileWriter.append(cust1); dataFileWriter.append(cust2); dataFileWriter.close(); www.semtech-solutions.co.nz info@semtech-solutions.co.nz

  7. Contact Us • Feel free to contact us at • www.semtech-solutions.co.nz • info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems

More Related