Using Windows Azure Storage

1. COS304 Using Windows Azure Storage Jai Haridas Senior Development Lead Microsoft Corporation

2. Agenda • Windows Azure Storage • Blobs, Drives, Tables, Queues • Partitioning & Scalability • Q&A

3. What is Windows Azure Storage? • Cloud Storage System • Scalable • Durable • Highly Available • Pay for what you use

4. Abstractions • Blobs– File system in the cloud • Drives–Durable NTFS volumes for Windows Azure applications • Tables– Massively scalable structured storage. • Queues– Reliable storage and delivery of messages

5. Windows Azure Data Storage Concepts Container Blobs Account • Table Entities https://<account>.blob.core.windows.net/<container> Queue Messages https://<account>.table.core.windows.net/<table> https://<account>.queue.core.windows.net/<queue>

6. Windows Azure Blobs • A highly scalable and durable file system in the cloud • An account can create many containers • No limit on number of blobs in a container • Associate metadata with blobs • Share • Share your blobs • Public containers, Shared Access Signature (SAS) • Upload/Download Blobs • Allows range reads • Conditional operations – If-Match, If-Not-Modified-Since etc.

7. How to Upload/Download files? • Goals • It is simple! • Upload your files demo

8. Windows Azure Blobs Types – Block Blobs • Targeted at streaming workloads • Each blob consists of a sequence of blocks • 2 Phase commit: Blocks are uploaded and then separately committed • Efficient continuation and retry • Random range reads possible • Size limit 200GB per blob

9. Block 1 Block 2 Block 3 Block 4 Block 1 Block blobs • File has variable sized blocks • Upload blocks in parallel using PutBlock • Retry failed blocks • Commit the blob using PutBlockList Local file Block 5 Block 2 Block 1 Block 3 Cloud blob

10. Windows Azure Blobs Types – Page Blobs • Targeted at random write workloads • Each blob consists of an array of pages • Size limit 1TB per blob • Page • Each page range write is committed on PUT • Page is 512 bytes in size • Write boundary aligned at multiple of 512 byte • Range reads possible • Pages that do not have data are zeroed out

11. Page blobs • Write 5K bytes – PutPage • Clear 2K bytes starting at offset 1K – ClearPage • Overwrite 2K bytes starting at 2K – PutPage • Truncate blob to 3K - SetMaxBlobSize 5120 5120 5120 0 0 0 0 3072 3072 1024 2048 4096

12. Sharing Your Files • Every blob request must be signed with the account owner’s key • Share your files • The container must be public • Shared Access Signature (SAS) – share pre-authenticated URLs with users • SAS • Use container level access as it allows access to be easily revoked

13. How to Share your files? • Goals • By default – authentication required • Public containers – everyone has read access • SAS – share url with subset without sharing your key! demo

14. What Are Snapshots? • Create a point in time read-only copy of a blob • Every snapshot creates a new read only point in time copy • Restore snapshots using copy blob • Cleanup your snapshots • Charged only for unique blocks or pages i.e. reuse blocks or pages • For reuse, use WritePages or PutBlock& PutBlockList

15. What does unique mean? • Add 2 blocks {1, 2} • Commit the blob • Take a snapshot • Charged for • Base blob’s blocks {1, 2} ID=1 ID=2 ID=1 ID=2 A BB A BB Base blob = alphabets.txt #1 snapshot=2011-04-10T19:26:24.8690267Z Snapshot #1

16. What does unique mean? • Add a new block Id=3 • Commit the base blob • Charged for • Base blob’s blocks {1, 2, 3} • Snapshot to create the second snapshot • Charged for • Base blob’s blocks {1, 2, 3} ID=3 ID=2 ID=2 ID=1 ID=1 ID=2 ID=2 ID=1 ID=1 ID=3 CCC CCC A BB A A A BB BB BB Base blob = alphabets.txt #1 snapshot=2011-04-10T19:26:24.8690267Z Snapshot #2 Base blob = alphabets.txt #2 snapshot=2011-05-10T19:26:24.8690267Z

17. What does unique mean? • Update block id 1 to now contain the same data “A” • Commit the blob • Charged for: • Base blob’s blocks {1, 2, 3} • Snapshot #1’s blocks {1} • Take a snapshot • Charged for: • Base blob’s blocks {1, 2, 3} • Snapshot #1’s blocks {1} ID=2 ID=1 ID=2 ID=3 ID=2 ID=3 ID=1 ID=2 ID=3 ID=1 ID=1 A BB CCC BB A A BB BB CCC A CCC #1 snapshot=2011-04-10T19:26:24.8690267Z Snapshot #3 A #2 snapshot=2011-05-10T19:26:24.8690267Z Base blob = alphabets.txt #3 snapshot=2011-05-10T19:28:24.8690267Z

18. Blobs – Best Practices & Tips • Client Library uses a default of 90s timeout – use size based timeout • Use parallel block upload to reduce latency • Snapshots – For block or page reuse, use PutBlock-PutBlockList/PutPage to update • Shared Access Signature • Use container level policy as it allows revoking permissions • Share SAS URLs using https

19. Blobs – Best Practices & Tips • Create new containers for blobs like log files that have retention period • Delete logs after 1 month - create new containers every month. • Container recreation • Garbage collection can take time until which time container with same name cannot be created. • Use unique names for containers

20. WINDOWS AZURE DRIVE • Provides a durable NTFS volume for Windows Azure applications • Use existing NTFS APIs • Easy migration path to the cloud • Durability and survival of data on application failover or hardware failure • All flushed and un-buffered writes to drive are made durable

21. WINDOWS AZURE DRIVE • A Windows Azure Drive is a Page Blob • Mounts Page Blob as an NTFS drive • Mounted by one VM at a time for read/write • A VM can dynamically mount up to 16 drives • Drives can be up to 1 TB

22. Windows Azure Tables • Massively Scalable and Durable Structured Storage • Provides flexible schema • A storage account can contain many tables • No limit on number of entities (aka rows) in each table • Familiar and Easy to use API • OData Protocol • WCF Data Services - .NET classes and LINQ

23. Simple structured storage… how to build it? • Goals • It is simple! Declare your POCO • Show Add/Update/Delete • Compare various types of queries • How to handle multiple types in a single table? demo

24. Tables – Best Practices & Tips • Use clustered index in queries for performance • Limit large scans • Expect continuation tokens for queries that scan • Split “OR” on keys as individual queries • Point query throws an exception if resource does not exist. • DataServiceContext.IgnoreResourceNotFoundException = true • For “read only” scenarios turn off tracking • DataServiceContext.MergeOption = MergeOption.NoTracking

25. Tables – Best Practices & Tips • Entity Group Transaction • Get transaction semantics • Reduce costs • Do not reuse DataServiceContext across multiple logical operations and discard on failures • AddObject/AttachTo can throw exception if entity is already being tracked

26. Windows Azure Queues • Highly scalable, available and provides reliable message delivery • A storage account can create any number of queues • Access is provided via REST

27. Windows Azure Queue Message • 8K message size limit and default expiry of 7 days • Programming semantics – Ensures that a message can be processed at least once • Get message to make the message invisible • Delete message to remove the message

28. Simple Queue Demo… how to build it? • Goals • Show Put/Get messages • Remove Poison Messages demo

29. Queues – Best Practices & Tips • Make message processing idempotent • Do not rely on order – invisibility time can result in out of order • Messages > 8KB • Use blobs or tables to store and message has a reference • Use dequeue count to detect problems • Visibility expiry time needs to increase • Poison messages

30. Queues – Best Practices & Tips • Use message count to dynamically increase/reduce workers. • Retain one instance that polls once every X time period • One instance polling every second results in 2,678,400 calls per month • Costs around $2.67 • Spawn more instances when you detect backlog

31. Agenda • Windows Azure Storage • Blobs, Drives, Tables and Queues • Partitioning & Scalability • Q&A

32. Know The Scalability Targets • Single Blob Partition • Throughput up to 60 MB/s • Single Queue/Table Partition • Up to 500 transactions (entities or messages) per second • Storage Account • SLA – 99.9% Availability • Capacity – Up to 100 TBs • Transactions – Up to 5000 entities per second • Bandwidth – Up to 3 gigabits per second

33. How To Scale Beyond Known Scalability Targets? • Distribute traffic to multiple partitions • Use multiple storage accounts • When limit is hit, app may see frequent ‘503 server busy’; applications should implement exponential back-off

34. Storage Partitioning – How to Scale? Understanding partitioning is key to understanding scale

35. Automatic Load Balancing - Assignment VIP Legend - Partition FE FE FE - Server Load Master System Master System Master System Reassign Partitions Offload Partitions BE1 BE2 BE3 BE4 Distributed File System • Time between offload to reload is in the order of seconds • Time to decide to load balance is in the order of minutes • Goal is to only reassign a partition only if the system has to

36. Partition Keys In Each Abstraction

37. Partition Keys In Each Abstraction

38. Partition Keys In Each Abstraction

39. Table Inserts: Single Partition (SP) vs. Multiple Partitions (MP)

40. Table Get:Single Partition (SP) vs. Multiple Partitions (MP)

41. Windows Azure Storage Takeaways • Scalable data abstractions to build your applications • Blobs– Files and large objects • Drives– Durable NTFS volume for Windows Azure applications • Tables– Massively scalable structured storage • Queues– Reliable delivery of messages • Easy to use via the Storage Client Library • More info on Windows Azure Storage at: http://blogs.msdn.com/windowsazurestorage/

43. Track Resources Don’t forget to visit the Cloud Power area within the TLC (Blue Section) to see product demos and speak with experts about the Server & Cloud Platform solutions that help drive your business forward. You can also find the latest information about our products at the following links: • Cloud Power - http://www.microsoft.com/cloud/ • Private Cloud - http://www.microsoft.com/privatecloud/ • Windows Server - http://www.microsoft.com/windowsserver/ • Windows Azure - http://www.microsoft.com/windowsazure/ • Microsoft System Center - http://www.microsoft.com/systemcenter/ • Microsoft Forefront - http://www.microsoft.com/forefront/

44. Resources • Connect. Share. Discuss. http://northamerica.msteched.com Learning • Sessions On-Demand & Community • Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers http://microsoft.com/technet http://microsoft.com/msdn

45. Complete an evaluation on CommNet and enter to win!

47. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.