Cloud Computing Development

1. Cloud Computing Development

2. Cloud Computing Development • Shallow Introduction

3. Introduction

4. What is the cloud computing • Is it computing while in flight? Image Courtesy SevensHeaven.nl

5. What is the cloud computing • Is it computing while in flight? • NO Image Courtesy SevensHeaven.nl

6. What is the cloud computing • What is it about then?

7. What is the cloud computing • What is it about then? • Cloud computing is consumption of computing resources without worrying about specifics.

8. What is the cloud computing • What is it about then? • As well as ability to add or remove resources according to the demand.

9. What is the cloud computing • What is it about then? • Similar to the power grid andtelephone network.

10. What is the cloud computing • What is it about then? • Similar to the power grid andtelephone network.

11. How does it work? • Consumer signs up for the service. (Same as if you get a mobile phone plan) • Consumer uses services according to their needs • Provider sends the bill at the end of the cycle • Consumer pays

12. Provider Models

13. Provider Models

14. Provider Models

15. Providers

16. Providers

17. Providers

18. Provider: Windows Azure • Platform as a service • Windows based • Storage provided through blob storage, drives, SQL Azure • State is stored and propagated with Queues and Tables • Integrated with Visual Studio • Eclipse plug-in for PHP

19. Slide courtesy Vlad Vinogradsky from Microsoft

20. Provider: Google App Engine • Platform as a service • Python or Java based • Storage provided through BigTable • Automatically scales web nodes

21. Provider: Rackspace • Infrastructure as a service • Very Basic just a few Linux or Windows images • Provides storage with CloudFiles • Very Cheap • Open source API • Relatively New

22. Provider: Amazon AWS • Oldest on the market • Many services / Images / Third party providers • Provides computation through EC2 / EMR • Provides state / storage through S3, SQS, RDS, SimpleDB • Multiple APIs

23. Sample Prices

24. Development

25. Practical Considerations • Cloud Development is slightly different from traditional in house model.

26. Practical Considerations • Cloud Development is slightly different from traditional in house model. • Everything is virtualized (most of the time) • Everything is distributed • Per instance reliability is much lower • Overall reliability is much higher

27. Cloud Programming Model

28. Cloud Programming Model • Compute and Interface nodes are not reliable, they can crash and disappear at any time. • Storage and State are reliable and heavily distributed. • At any time we can start more compute or interface nodes and shut them down when demand subsides.

29. Cloud Programming Model on Azure • Compute : Worker Nodes • State: Tables / Queues / SQL • Storage: SQL / Tables / Blobs / Drives • Client Inteface: Web Nodes

30. Cloud Programming Model on AWS • Compute : EC2 Instances • State: S3 / Queues / SimpleDB / RDS • Storage: S3 / SimpleDB / RDS • Client Inteface: S3 / EC2 / CloudFront

31. AWS Details: S3 • S3 = Simple Storage Service • Guaranteed to be reliable • Simple {Key, Value} storage • Keys are stored within buckets • Values could be as large as 5GB • Default Storage Mechanism for AWS

32. AWS Details: Simple DB • Schema less database • Main storage unit is domain ( similar to table ) • Each record can have many attributes, new attributes could be added at any time • Similar to LISP / Scheme attributes • Can query domain for records containing particular attribute • No Joins / Unions with other domains • Eventual Consistency

33. AWS Details: RDS • RDS = Relational Data Storage • MySQL in a cluster mode • Preferred to simply running DB server within instance (ask me why for details)

34. AWS Details: SQS • SQS = Simple Queue System • Massively scalable • Allows to put message in the queue and retrieve later on • Retrieving the message hides it from the other users • When message is processed it is deleted from the queue • If message is not deleted before the timeout it is returned back

35. EC2 = Elastic Compute Cloud • Allows to run arbitrary virtual machinesProvided they are compatible with Amazon’s modified Xen • Kernels and Startup Disks are stored in S3 • Also have large local storage • Machines are not exactly like physical machines • Local storage is not persistentWhen machine is shut down all local data disappears. • Hardware TCP [No packet layer / No Broadcast ] • Can launch many copies of the machine at the same time • Lot’s of preconfigured machines AWS Details: EC2

36. AWS Details: Other Services • EMR = Elastic Map ReduceLet’s run Hadoop jobs on EC2 • CloudFront Content Delivery Network • ELB = Elastic Load Balancer • EBS = Elastic Block StorageS3 backed persistent storage • Public Data Sets - Lots of publicly available data Census ( 1980 , 1990, 2000 ), Wikipedia logs, Freebase dumps, Genetic and Chemistry data

37. Starting Up • Amazon Account • Credentials KeyID : SecretKey • X509 Ceriticate

38. Helpful Tools • S3 Fox - Firefox extension for browsing S3 • Elastic Fox - Firefox extension for operating EC2 • Transmit - Mac utility for S3 ($) • Right Scale - Web based platform for managing everything ( Free / $ )

39. Libraries • Official Amazon Libraries (Java) • Unofficial Libraries - .Net / Ruby / Perl • AWS4C - C/C++/Objective C • Boto - Very popular Python library (official Hadoop/EC2 library)

40. Demo

41. Demo Running Hadoop on EC2

42. Questions ????