Windows Azure Tables: Programming Cloud Table Storage

1. ES07 Windows Azure Tables: Programming Cloud Table Storage  Niranjan Nilakantan Development Lead Microsoft Corporation  Pablo Castro Software Architect Microsoft Corporation

2. Windows Azure • Windows Azure is the foundation of Microsoft’s Cloud Platform • It is an “Operating System in the Cloud” and provides Essential Services for the cloud • Virtualized Computation • Scalable Storage • Automatic Management • Developer SDK

3. Fundamental Data Abstractions • Blobs – Provide a simple interface for storing named files along with metadata for the file • Tables – Provide structured storage; A Table is a set of entities, which contain a set of properties • Queues – Provide reliable storage and delivery of messages for an application

4. Windows Azure Tables • Provides Structured Storage • Massively Scalable Tables • Billions of entities (rows) and TBs of data • Can use thousands of servers as traffic grows • Highly Available • Can always access your data • Durable • Data is replicated several times • Familiar and Easy to use API • ADO.NET Data Services – .NET 3.5 SP1 • .NET classes and LINQ • REST – with any platform or language

5. Agenda • Data model • Demo – creating tables • Basic CRUD using .NET and REST • Demo – a sample application • Advanced features

6. Data Model • Table • A Storage Account can create many tables • Table name is scoped by Account • Data is stored in Tables • A Table is a set of Entities (rows) • An Entity is a set of Properties (columns) • Entity • Two “key” properties that together are the unique ID of the entity • PartitionKey – enables scalability • RowKey – uniquely identifies the entity within the partition

7. Partitioning • Table Partition – all entities in table with same partition key value • Application controls granularity of partition Partition 1 Partition 2

8. Partitioning Guidelines • Performance • Use a PartitionKey that is common in your queries • Entities with same partition key value are clustered • Scalability • We monitor partition traffic • Automatically load balance partitions • Each partition can potentially be served by a different storage node • Scale to meet the traffic needs of your application • More partitions – makes it easier to balance load

9. Entity Locality • Getting all versions of FAQ is fast • Access single partition • Get all documents before 5/30/2008 takes longer • Have to traverse all partitions Partition 1 Partition 2

10. Entities and Properties • Each Entity can have up to 255 properties • Every Entity has fixed key properties • Partition key • Row key • No fixed schema for rest of properties • 2 entities in the same table can have different properties • Properties stored as <Name, TypedValue> pairs • Each entity has a system maintained version

11. Property Types • Partition key and Row key • String (up to 64KB) • Other properties • String (up to 64KB) • Binary (up to 64KB) • Bool • DateTime • GUID • Int • Int64 • Double

12. Sample Scenario Channels Name – PK Author CreatedDate • Micro-blogging application • Post messages to channels • Subscribe to channels • Retrieve channel messages • Scalability challenges • Large volume of new messages daily • Need to retrieve recent messages quickly • Messages permanently addressable Messages Channel – PK PostedDate – RK Text Rating Users Name – PK Email EmailUpdates Channels

13. demo MicroBlogging Application

14. Table Schema Define the schema as a .NET class [DataServiceKey("PartitionKey", "RowKey")] public class Message { // ChannelName public string PartitionKey { get; set;} // PostedDate public string RowKey { get; set;} // User defined properties public string Text{ get; set; } public intRating{ get; set; } }

15. Insert Entity using .NET Create a new Message and Insert into Table Message message = new Message { PartitionKey = "Channel9", // ChannelName RowKey=DateTime.UtcNow.ToString(), // PostedDate Text = "Hello PDC", Rating = 3 }; serviceUri = new Uri("http://<account>.table.core.windows.net"); varcontext = new DataServiceContext(serviceUri); context.AddObject("Messages", message); DataServiceContextresponse = context.SaveChanges();

16. Insert Entity using REST Create an Atom XML payload and POST it • POST http://<Account>.table.core.windows.net/Messages • ... <!– Atom envelope --> • <m:properties> • <d:PartitionKey>Channel9</d:PartitionKey> <!--ChannelName--> • <d:RowKey>Oct-29</d:RowKey> <!--PostedDate--> • <d:Text>Hello PDC</d:Text> • <d:Rating>3</d:Rating> • </m:properties>

17. Create Table Every Account has a table called “Tables” • To use a table it has to be inserted into “Tables” [DataServiceKey("TableName")] public class TableStorageTable { public string TableName { get; set; } } // Service Uri is “http://<Account>.table.core.windows.net/” DataServiceContextcontext = new DataServiceContext(serviceUri); TableStorageTabletable = new TableStorageTable("Messages"); context.AddObject("Tables", table); DataServiceResponseresponse = context.SaveChanges();

18. Query Entities .NET – LINQ serviceUri = new Uri("http://<account>.table.core.windows.net"); DataServiceContextcontext = new DataServiceContext(serviceUri); varmessages = from message in context.CreateQuery<Message>("Messages") where message.Rating == 3 select message; foreach(Message messagein messages) { } • REST GET http://<serviceUri>/Messages?$filter= Rating eq 3

19. Update a property in an entity Using the .NET API Message message = (from message in context.CreateQuery<Message>("Messages") where message.PartitionKey == "Channel9"   &&message.RowKey == "Oct-29" select message).FirstOrDefault(); message.Text = "Hi there"; context.UpdateObject(message); DataServiceResponse response = context.SaveChanges(); • Using the REST API • PUT http://<serviceUri>/Messages(‘Channel9’, ‘Oct-29’) • <m:properties> • <d:Text>Hi there</d:Text> • <!-- Other properties are the same --> • </m:properties>

20. Delete an Entity Using the .NET API // Get the Message object for ("Channel9", "Oct-29") context.DeleteObject(message); DataServiceResponseresponse = context.SaveChanges(); • Using the REST API • DELETE http://.../Messages(‘Channel9’, ‘Oct-29’)

21. .NET vs. REST ADO.NET Data Services REST Interface Use any HTTP stack Data represented in Atom (XML) Standard HTTP verbs for updates Use URLs to define queries • Client included in .NET Framework 3.5 SP1 • Data represented as .NET objects • DataServiceContext methods for updates • Use LINQ to define queries

22. demo MicroBlogging Application

23. Windows Platform Integration • Reuse .NET skills • Fully compatible with ADO.NET data services • .NET client included in .NET 3.5 SP1 • LINQ support • ASP.NET integration for website authoring • Sample data source control for data binding • ASP.NET dynamic data for instant front-ends • ASP.NET providers for membership, roles, etc.

24. Advanced Topics • Concurrent updates • Pagination • Table consistency

25. Concurrent Updates • Use standard HTTP mechanisms – Etag and If-Match • Get entity – get system maintained version as ETag • Update Entities Locally – change rating • Send Update with version check - IF-Match with Etag • Success if version matches, and update version on Client-A • Precondition failed (412) if version does not match ClientB Client A Version Rating 1: Ch9, Jan-2, 5 2: Ch9, Jan-2, 5 1: Ch9, Jan-2, 4 Error: 412 5 : Ch9, Jan-1, 3 If-Match: 1 Ch9, Jan-2, 4 If-Match: 1 Ch9, Jan-2, 5 1 : Ch9, Jan-2, 2 1 : Ch9, Jan-2, 2 1 : Ch9, Jan-2, 2 2: Ch9, Jan-2, 5 9 : Ch9, Jan-3, 6

26. Getting the Top N entities .NET: LINQ Take(N) function • serviceUri = new Uri("http://<account>.table.core.windows.net"); • DataServiceContextcontext = new DataServiceContext(serviceUri); • varallMessages = context.CreateQuery<Message>("Messages"); • foreach (Message messagein allMessages.Take(100)) • { • Console.WriteLine(message.Name); • } REST: $top=N query string option GET http://<serviceUri>/Messages?$top=100

27. Pagination – Continuation Tokens Send a request • GET http://<serviceUri>/Messages?$filter=...&$top=100 Get continuation token in response header Messages x-ms-continuation-NextPartitionKey: Channel9 x-ms-continuation-NextRowKey: Date101 100 Ch9, Date1, Ch9, Date2, Ch9, … Set HTTP query parameters Ch9,Date100, GET http://<Uri>/Messages?$filter=...&$top=100 Ch9,Date101, • &NextPartitionKey=Channel9 • &NextRowKey=Date101 Ch9, …

28. Single Table Consistency • ACID transactions for single entity CUD • Insert/update/delete • Snapshot isolation for query within a single partition • Consistent view from start time of the query • No dirty (uncommitted) reads • Does not block concurrent updates • No snapshot isolation across partitions • No snapshot isolation across different continuations of a query

29. Cross Table Consistency • Application is responsible for maintaining consistency • Example • When a channel is deleted, delete all the messages for that channel • Failures can occur in the middle • Example - Application fails after deleting some messages • Use Windows Azure Queues to help ensure completion of operation

30. Cross Table Consistency Delete channel Delete messages worker Front end Worker Queue Messages Ch1, Msg1 Del Ch5 Del Ch11 Del Ch1 Del Ch1 2 Ch1, Msg2 Ch1, Msg3 Front End Ch2, Msg1 Ch2, Msg2 Channels Ch1,… Ch3, Msg1 Ch2,… 4. Delete queue entry 3. Delete from Messages 2. Delete Ch1 from Channels 1. Dequeue DelCh1

31. Resuming After Failure Delete channel Delete messages worker Front end Worker 1 Del Ch1h Queue Messages Ch1, Msg1 Del Ch5 Del Ch11 Del Ch1 Del Ch1 2 Ch1, Msg2 Ch1, Msg3 Front End Ch2, Msg1 Worker2 Ch2, Msg2 Channels Ch1,… Ch3, Msg1 Ch2,… 3. DelCh1 is visible again 4. Dequeue DelCh1 again 5. Repeat delete operations 2. Fails after deleting Ch1 and Msg1 1. Dequeue DelCh1 and start delete

32. Future Windows Azure Table Support • Future Support for • Secondary Indexes • Query and retrieve results sorted by other properties • At PDC • Single Index • Query and retrieve results sorted by PartitionKey and RowKey • Entity Groups • Atomic transactions across multiple entities within same partition • Single Entity Transactions • Single Entity Insert, Update, or Delete Atomically

33. Summary • Azure tables are • Massively Scalable • Highly Available • Simple familiar API • Use .NET –- ADO.NET Data Services and LINQ • Or use REST • Leverage your .NET expertise • ASP .NET integration for instant front ends

34. Resources • Links • Windows Azure • http://www.azure.com/windows • ADO.NET Data Services • http://blogs.msdn.com/astoriateam

35. Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

36. Q&A Please use the microphones provided

37. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.