1 / 17

Best Practices for Real-Time Data

Best Practices for Real-Time Data. Viviane Ribeiro @ viviane_sql viviane@bidobrasil.com. THR2116.

masako
Download Presentation

Best Practices for Real-Time Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Best Practices for Real-Time Data Viviane Ribeiro @viviane_sql viviane@bidobrasil.com THR2116

  2. Real time data analysis brings a lot of innovation and can change the way that companies connect with their customers and partners. It allows us to promote digital transformation inside of business. Transforming Business… Impacting Lives!

  3. Our Big Challenges Ingest, process, and store messages In real time, especially at high volumes Act on the data quickly Generating alerts in real time or presenting the data in a real-time Data Producers and Consumers How these streams of data are being consumed and produced Simplicity, Scalability, Reliability How do we prepare to scale, for data rates and prevent data loss Security Encrypt data, authenticate clients, role based authorizations and ACLS

  4. Real-Time Processing Architecture

  5. Message Ingestion Solutions Azure Event Hubs Kafka Azure IoT Hub Azure Event Hubs is a message queuing solution for ingesting millions of event messages per second. The captured event data can be processed by multiple consumers in parallel. Kafka is an open source message queuing and stream processing application that can scale to handle millions of messages per second from multiple message producers, and route them to multiple consumers. Azure IoT Hub provides bi-directional communication between Internet-connected devices, and a scalable message queue that can handle millions of simultaneously connected devices.

  6. Choosing a Real-Time Message Ingestion

  7. Choosing a Real-Time Message Ingestion Do you need two-way communication between your IoT devices and Azure? If so, choose IoT Hub. Do you need to manage access for individual devices and be able to revoke access to a specific device? If yes, choose IoT Hub. Choosing between Event Hub or Kafka, At the end of the day, you need to ask yourself: what degree of control do I need?

  8. Demo Viviane Ribeiro @Viviane_Sql

  9. Real-Time Processing Architecture

  10. Real-Time Processing Architecture

  11. Streaming Solutions • Azure Stream Analytics • HDInsight with Spark Streaming • Apache Spark in Azure Databricks • HDInsight with Storm • Azure Functions • Azure App Service WebJobs • Programmability • Scala, Python, Java, R • C#, F#, Node.js, PHP • ??? • Pricing model • Streaming Units • Per Cluster hour • Databricks Units • ??? • Integration • Inputs • Sinks • Scalability http://bit.ly/RealtimeArchitectures

  12. Lambda Architecture

  13. Real-Time Analytics with Azure Cosmos DB and Spark

  14. Thank you! Additional Resources  http://bit.ly/RealtimeArchitectures YouTube.com/Vivianesql Linkedin.com/ in/viviane-ribeiro-sql Facebook.com/ VivianeRibeirosql Instagram.com/CientistadeDados Twitter @Viviane_sql Blog: http://vivianeribeiro1.worpress.com Email: viviane@bidobrasil.com

  15. Please evaluate this sessionYour feedback is important to us! Please evaluate this session through MyEvaluations on the mobile appor website. Download the app:https://aka.ms/ignite.mobileApp Go to the website: https://myignite.techcommunity.microsoft.com/evaluations

More Related