Extending to the blue: Visual Studio 2010 and cloud D evelopment

1. Extending to the blue: Visual Studio 2010 and cloud Development Karl Davies-Barrett Developer and Platform Evangelist Microsoft Malta & Cyprus i-karld@microsoft.com

2. Agenda • The ‘What and Why’ of Windows Azure • Windows Azure Storage • Blobs, Drives, Tables & Queues • Compute • SQL Azure • Project ‘Dallas’

3. The ‘What and Why’ of Windows Azure

4. Platform Continuum On-Premises Servers Hosted Servers Cloud Platform • Bring your own machines, connectivity, software, etc. • Complete control • Complete responsibility • Static capabilities • Upfront capital costs for the infrastructure • Renting machines, connectivity, software • Less control • Fewer responsibilities • Lower capital costs • More flexible • Pay for fixed capacity, even if idle • Shared, multi-tenant infrastructure • Virtualized & dynamic • Scalable & available • Abstracted from the infrastructure • Higher-level services • Pay as you go

5. Workload Patterns Optimal for Cloud “Growing Fast“ “On and Off “ Inactivity Period Compute Compute Average Usage Usage Average Time Time • On and off workloads (e.g. batch job) • Over provisioned capacity is wasted • Time to market can be cumbersome • Successful services needs to grow/scale • Keeping up w/growth is big IT challenge • Complex lead time for deployment “Unpredictable Bursting“ “Predictable Bursting“ Compute Compute Average Usage Average Usage Time Time • Unexpected/unplanned peak in demand • Sudden spike impacts performance • Can’t over provision for extreme cases • Services with micro seasonality trends • Peaks due to periodic increased demand • IT complexity and wasted capacity

6. The Windows Azure Platform • Developer Experience • Use existing skills and tools. ` platform AppFabric Management Relational data Compute Storage Management Connectivity Access control

7. Windows Azure Architecture Details Security and Control Features include storage encryption, access authentication, and over-the-wire encryption using HTTPS. Industry certification is part of the Windows Azure roadmap Fabric Fabric Controller The Fabric Controller automates load balancing and computes resource scaling Compute Storage Geographically distributed, state-of-the-art data centers host your applications and data, internet-accessible from everywhere you choose to allow Computation provides application scalability. Developers can build a combination of web and worker roles. Those roles can be replicated as needed to scale the applications and computational processing power Storage Services allow customers to scale to store large amounts of data – in any format – for any length of time, only paying for what they use or store

8. Windows Azure Platform Benefits Windows Azure SQL Azure Higher Level of Abstraction Hardware Server OS Network Infrastructure Database Server Availability Automated Database Management & Replication Scalability Databases Partitioning Developer Experience Familiar SQL Environment • High Level of Abstraction • Hardware • Server OS • Network Infrastructure • Web Server • Availability • Automated Service Management • Scalability • Instance & Partitions • Developer Experience • Familiar Developer Tools

9. Why Windows Azure? • OS Takes care of your service in the cloud • Deployment • Availability • Patching • Hardware Configuration • You worry about writing the service • Developer Experience!

10. What is Windows Azure? • Developer SDK • Windows Azure SDK • Local compute environment & Mock Storage • Command line tools • Small Managed API • Logging, working storage • Microsoft Visual Studio 2008/10 add-in Compute Storage

11. The Portal and Dev Environment

12. Windows Azure Storage Where’s my Data?

13. Windows Azure Storage • Enables developers, apps, and users to access cloud storage • Scalable, durable, and available • Anywhere at anytime access • Only pay for what the service uses • Easy to use REST and Client Libraries • Blobs, tables, queues • Existing NTFS APIs • Drives

14. Windows Azure: Storage Compute Storage Management

15. Storage Abstractions • Blobs– Provide a simple interface for storing named files along with metadata for the file • Drives– Provides durable NTFS volumes for Windows Azure applications to use • Tables– Provide structured storage. A Table is a set of entities, which contain a set of properties • Queues– Provide reliable storage and delivery of messages for an application

16. Windows Azure Storage Account • User creates a globally unique storage account name • Can choose geo-location to host storage account • US – “North Central” and “South Central” • Europe – “North” and “West” • Asia – “East” and “Southeast” • Can co-locate storage account with compute account • Receive a 512 bit secret key when creating account

17. Account • Blob Container Blob Storage Concepts http://<account>.blob.core.windows.net/<container>/<blobname> • PIC01.JPG • images • VID1.AVI • PIC02.JPG • cohowinery • videos

18. Blob Storage

19. Blob Client Library Example StorageCredentialsAccountAndKeycredentials = newStorageCredentialsAccountAndKey(accountName, key); stringbaseUri = string.Format("http://{0}.blob.core.windows.net", accountName); CloudBlobClientblobClient = newCloudBlobClient(baseUri, credentials); // Create Container CloudBlobContainercloudContainer = blobClient.GetContainerReference(containerName); boolhasCreated = cloudContainer.CreateIfNotExist(); // Access Blob in the Container CloudBlobcloudBlob = cloudContainer.GetBlobReference(blobName); //BlobRequestOptions has retry policy, timeout etc. BlobRequestOptionsoptions = newBlobRequestOptions(); //Upload the local file to Blob service cloudBlob.UploadFile(uploadFileName, options); //Download to local file name cloudBlob.DownloadToFile(downloadFileName, options);

20. Blob Tips • For high throughput clients • Set ServicePointManager::DefaultConnectionLimitto allow parallel connections to cloud service • Default value is 2 • Upload/Download multiple files in parallel • Windows Azure Content Delivery Network (CDN) • Frequently accessed blobs • Blobs accessed from many geographic locations • Cache and serve your Windows Azure Blobs from our network • 18 locations globally (US, Europe, Asia, Australia and South America), and growing

21. Blob Summary • Blob Access Patterns (two types of Blobs) • Block Blob – streaming + commit-based writes • Blob consists of a sequence of blocks • Size limit 200GB per blob • Page Blob – random read/write • Blob consists of an array of pages • Size limit 1TB per blob

22. Storage Abstractions • Blobs– Provide a simple interface for storing named files along with metadata for the file • Drives– Provides durable NTFS volumes for Windows Azure applications to use • Tables– Provide structured storage. A Table is a set of entities, which contain a set of properties • Queues– Provide reliable storage and delivery of messages for an application

23. Windows Azure Drive • Provides a durable NTFS volume for Windows Azure applications to use • Use existing NTFS APIs to access a network attached durable drive • Benefits • Enables existing applications using NTFS to more easily migrate to the cloud • Durability and survival of data on application failover or hardware failure • A Windows Azure Drive is a Page Blob • Mounts Page Blob over the network as an NTFS drive • All flushed and unbuffered writes to drive are made durable to the Page Blob

24. Windows Azure Drive Capabilities • A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD) • Drives can be up to 1TB • A Page Blob can only be mounted by one VM at a time for read/write • A VM can dynamically mount up to 16 drives • Remote Access via Page Blob • Can upload the VHD to a Page Blob using the blob interface, and then mount it as a Drive • Can download the Drive through the Page Blob interface

25. Windows Azure Drive Demo Windows Azure • Use a local VHD • Create/mount a VHD in Win7 via Disk Management (diskmgmt.msc) • Upload a local VHD to Page Blob in Cloud • Have cloud web service to look at the contents of a drive • Show code for • CloudDrive initialization • Mounting a drive • Unmounting a drive

26. Cloud Drive Client Library Sample CloudStorageAccountaccount = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); //Initialize the local cache for drives mounted by this role instance CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB); //Create a cloud drive (PageBlob) CloudDrive drive = account.CreateCloudDrive(pageBlobUri); drive.Create(1000 /* sizeInMB*/); //Mount the network attached drive on the local file system stringpathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None); //Use NTFS APIs to Read/Write files to drive … //Snapshot drive while mounted to create backups UrisnapshotUri = drive.Snapshot(); //Unmountthe drive drive.Unmount();

27. Storage Abstractions • Blobs– Provide a simple interface for storing named files along with metadata for the file • Drives– Provides durable NTFS volumes for Windows Azure applications to use • Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties • Queues– Provide reliable storage and delivery of messages for an application

28. Windows Azure Tables • Provides Massively Scalable Structured Storage • Table can have billions of entities (rows) and TBs of data • Familiar and Easy to use API • WCF (ADO.NET) Data Services • .NET classes and LINQ • REST – with any platform or language

29. Table Storage Concepts Account Table Entity Name =… Email = … customers Name =… Email = … cohowinery Photo ID =… Date =… winephotos Photo ID =… Date =…

30. Entity Properties • Each entity can have up to 255 properties • Mandatory Properties for every entity • PartitionKey & RowKey • Uniquely identifies an entity • Defines the sort order • Timestamp • Optimistic Concurrency • No fixed schema for rest of properties • Each property is stored as a <name, typed value> pair • No schema stored for a table • Properties can be the standard .NET types • String, binary, bool, DateTime, GUID, int, int64, and double

31. PartitionKey and Partitions • Every Table has a PartitionKey • It is the first property (column) of your Table • Used to group entities in the Table into partitions • A Table Partition • All entities in a Table with the same partition key value • RowKey provides uniqueness within a partition • PartitionKey is exposed in the programming model • Allows application to control the granularity of the partitions and scalability of its tables

32. Purpose of the PartitionKey • Entity Locality • Entities in the same partition will be stored together • Efficient querying and cache locality • Entity Group Transactions • Atomically perform multiple Insert/Update/Delete over entities in same partition in a single transaction • Table Scalability • We monitor the usage patterns of partitions • Automatically load balance partitions • Each partition can be served by a different storage node • Scale to meet the traffic needs of your table

33. Partitions and Partition Ranges Server A Table = Movies [MinKey - Comedy) Server A Table = Movies Server B Table = Movies [Comedy - MaxKey)

34. Table Operations • Table • Create, Query, Delete • Entities • Insert • Update • Merge – Partial update • Replace – Update entire entity • Delete • Query • Entity Group Transactions

35. Table Schema Define the schema as a .NET class [DataServiceKey("PartitionKey", "RowKey")] publicclassMovie { /// Movie Category is the partition key publicstringPartitionKey { get; set; } /// Movie Title is the row key publicstringRowKey { get; set; } publicDateTime Timestamp { get; set; } publicintReleaseYear { get; set; } publicdouble Rating { get; set; } publicstring Language { get; set; } publicboolFavorite { get; set; } }

36. Table Client Library Sample stringbaseUri = string.Format("http://{0}.table.core.windows.net", accountName); CloudTableClienttableClient = newCloudTableClient(baseUri, credentials); // Create Movie Table stringtableName = “Movies“; tableClient.CreateTableIfNotExist(tableName); TableServiceContext context = tableClient.GetDataServiceContext(); // Add movie context.AddObject(tableName, newMovie("Action", “White Water Rapids Survival")); context.SaveChangesWithRetries(); // Query movie var q = (from movie in context.CreateQuery<Movie>(tableName) wheremovie.PartitionKey== "Action"&& movie.Rating > 4.0 select movie).AsTableServiceQuery<Movie>(); foreach(MoviemovieToUpdatein q) { movieToUpdate.Favorite = true; context.UpdateObject(movieToUpdate); } context.SaveChangesWithRetries(); SaveChangesOptions.Batch

37. Storage Abstractions • Blobs– Provide a simple interface for storing named files along with metadata for the file • Drives– Provides durable NTFS volumes for Windows Azure applications to use • Tables– Provide structured storage. A Table is a set of entities, which contain a set of properties • Queues – Provide reliable storage and delivery of messages for an application

38. Queue Storage Concepts Account Queue Message customer ID order ID http://… order processing cohowinery customer ID order ID http://…

39. Loosely Coupled Workflow with Queues • Enables workflow between roles • Load work in a queue • Producer can forget about message once it is in queue • Many workers consume the queue Input Queue (Work Items) Azure Queue Worker Role Web Role Worker Role Web Role Worker Role Web Role Worker Role

40. Queue’s Reliable Delivery • Guarantee delivery/processing of messages (two-step consumption) • Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time” • Worker Deletes message when finished processing it If Worker role crashes, message becomes visible for another Worker to process Worker Role Web Role Input Queue (Work Items) Worker Role Azure Queue Web Role Worker Role Web Role Worker Role

41. Queue Client Library Sample stringbaseUri = string.Format("http://{0}.queue.core.windows.net", accountName); CloudQueueClientqueueClient = newCloudQueueClient(baseUri, credentials); //Create Queue CloudQueuequeue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessagemessage = newCloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout //Delete Message queue.DeleteMessage(message);

42. Queue Tips • Messages can be up to 8KB • Use blob to store large messages, and store blob ref in message • A message may be processed more than once • Make message processing idempotent • Work should be repeatable and can be done multiple times • Assume messages put into queue can be processed in any order • For higher throughput • Batch multiple work items into a single message or into a blob • Use multiple queues • Use DequeueCount to remove poison messages • Enforce threshold on message’s dequeue count • Monitor message count to dynamically increase/reduce workers

43. Windows Azure Storage Takeaways • Fundamental data abstractions to build your applications • Blobs – Files and large objects • Drives – NTFS APIs for migrating applications • Tables – Massively scalable structured storage • Queues – Reliable delivery of messages • Easy to use via the Storage Client Library http://www.microsoft.com/windowsazure/

44. Deploying a Cloud Service The recommended workflow for deploying your service is as follows; assuming you use Windows Azure storage services: • Test and debug your service in the development fabric and ensure that your service runs against development storage. (Everything local) • Create a new storage account on Windows Azure developer portal. • Modify your service to run against your storage account in the cloud. (Intermediary step that can help you identify problems) (App on machine, storage in the cloud) • Create a new hosted service in the Windows Azure developer portal. • Deploy your service package to the Windows Azure developer portal and test it in the cloud. (Everything in the cloud)

45. Microsoft Confidential Queues and Compute

46. SQL Azure Your favorite RDMS now available in the cloud!

47. SQL Azure • Today: • SQL Azure Database • Formerly known as SQL Data Services (SDS) • And before that as SQL Server Data Services (SSDS) • In the future: • Reporting • Analysis • Extract/Transform/Load (ETL) services • More

48. Database TDS Database Database SQL Azure DatabaseAn illustration SQL Azure Database Others (Future)

49. Some overall Considerations • Scale and availability are the design points • Storage isn’t a relational database • Stateless • Stateless front ends, store state in storage • Use queues to decouple components • Instrument your application • You are not an Admin on the VM • Mix & match services based on the needs of your application

50. Intro to Microsoft Codename “Dallas”