Microsoft Azure Event Hub – IOT at Scale

3. Microsoft Azure Event Hub – IOT at Scale Brent Stineman – Technical Evangelist and code monkey brestin@microsoft.com

4. What is the Internet of things?Or Connected Devices if you’re one of the cool kids.

5. Defining the undefinable The Thing a specialized, autonomous piece of hardware capable of performing a simple, focused task but does not have a ‘traditional” user interface. Data Isolated elements sent or received by the thing Connectivity Able to connect to and interact with another thing or device Management Having an identity we can use to manage the thing

6. Canonical Arch for processing events Producers Collection Ingestor(broker) Transformation Long term storage Presentation and action Scalableevent Broker Azure Storage Real-time analytics(Storm/Orleans, etc…) Cloud Gateways(WebAPIs) Web/thickclientdashboards Applications - HD Insight Externaldata sources Your are here Massively Parallel Azure Systems w/ Jason Young Tuesday@10:30am, Aralia Devices Search and query HDInsight and Internet of Things w/ Tony Guidici Tuesday, 2:30pm, Tamarind Field Gateways Batching/storageadapters Data analytics (Excel)

7. Brokered MessagingOne message at a time.

8. Queues Guaranteed FIFO, with push-style API, up to 80GB maximum size, 10,000 queues per namespace, 2,000 messages per second, 256KB per message.

9. Queues – increasing throughput • Express • Decreases latency by storing the messages in memory • Partitions • Managed scale out (at max 16x)

10. Topics A higher level abstraction from queues, enables pub/sub models with similar scale out capabilities and consistent pricing

11. Topics • Filters • Allows subscribers to selectively receive messages • Rules • Aid in filtering and authorization • Volatile Subscribers • Can delete on idle, allowing for transient consumers

12. Introducing Event HubWhen the plumbing gets complicated, you just need to go with the flow

13. What do you need to handle this? An ingestion service that can Support variety (> million concurrent devices) Support velocity (> million events per second) Support volume (> 100s of Tb of event processing) With Buffering to handle variability Durability Low latency Security And be affordable!

14. Event Hub – IOT at Scale Storage & Analytics Custom Code & 3rd Party Services Event Hub Web/Mobile User Interfaces - Hyper Scale - - Fully Managed - - Interoperable - - Secure - - Cost Effective - Integration Services Cloud Services Event Sources

15. Event Hub Publisher concept Every Event Hub has pre-defined number of partitions Every event has a partition key => mapped to one of the partitions Partition 1 Event 1 PartitionKey=A Partition 2 Event 1 PartitionKey=B Partition “n”

16. Getting Started Creating our first hub, sample code

17. Publish Message via .NET/AMQP Critical to scaling

18. Publish Message via HTTP http://aka.ms/thatconfEH

19. Publisher Security <protocol>://<namespace>.servicebus.windows.net/<eventhubname>/publisher/<partitionkey>/messages SharedAccessSignaturesr=https%3a%2f%2fthatconference-ns.servicebus.windows.net%2fthatconference%2fpublishers%2fbstineman%2fmessages&sig=a3MkKnkhMZpTMz%2fnFcsCS%2fcT9rT7Wn%2fZO%2b7eztBRMkk%3d&se=64948904112&skn=Publish

20. Real World Example Publishing Social Media and Vendor Check-ins

21. Direct Receiver • Receiver connects to a partition and pull messages • Limited(5) receivers for each partition Worker “n” Worker 1 Receiver 6 Receiver 1 Receiver 2 Receiver “n” Partition 1 Event 1 Pkey = A Partition 2 Event 2 Pkey = B Partition “n”

22. IEventProcessor– Managed Receivers http://www.nuget.org/packages/Microsoft.Azure.ServiceBus.EventProcessorHost/0.3.0

23. IEventProcessor– Managed Receivers • Register a class that implements an EventProcessor interface • Client SDK will manage balancing receivers across workers Worker 4 Worker “n” Worker 1 Worker 2 Callback for prtn. 1 Callback for prtn. “n” Callback for prtn. 6 Callback for prtn. 1 Callback for prtn. 2 Callback “n” Partition 1 Partition 2 Partition “n” http://www.nuget.org/packages/Microsoft.Azure.ServiceBus.EventProcessorHost/0.3.0

24. Pub-Sub with Event Hub • Consumer groups allow multiple, parallel sets of subscribers to receive the same events Worker “n” Worker “n” Worker “n” Worker 1 Worker 1 Worker 1 Consumer Group A Callback for prtn. 1 Callback for prtn. 1 Callback for prtn. 6 Callback for prtn. 1 Callback for prtn. 6 Callback for prtn. 6 Consumer Group C Consumer Group B Callback for prtn. 2 Callback for prtn. 2 Callback “n” Callback “n” Callback for prtn. 2 Callback “n” Partition 1 Partition 2 Partition “n”

25. IEventProcessor– Partition Management

26. IEventProcessor– Partition Management

27. Real World Example Proximity Sensors

28. Getting StrategicSo what are you going to create?

29. Microsoft Azure Services for IoT Orientation to IoT Organization Characteristics Azure Services Buy Intelligent Systems Service (ISS) Solution • Building software is not a core competency • Focus on time-to-value • Custom protocols and security models • Custom analytics and data processing • Compose higher level services ISS Platform Services Buy Some, Build Some Build • Core competency in building software/services • Focus on flexibility & control, cost minimization • Strategic investment in custom service platform Core/App Platform Services

30. The Value of IoT on Azure • Mix and match services to build your solution • Use components on any platform • Storm on Linux in a VM cluster • HDInsight service on Azure • Web Sites & Power BI

31. Get started Azure Event Hub – IOT at Scale w/ Brent Stineman Monday@1:00pm, Guava Visit azure.microsoft.com Massively Parallel Azure Systems w/ Jason Young Tuesday@10:30am, Aralia HDInsight and Internet of Things w/ Tony Guidici Tuesday, 2:30pm, Tamarind