Cloud Computing: What a Project Manager Needs to Know

Cloud Computing: What a Project Manager Needs to Know Dr. Patrick D. Allen, PMP Patrick.allen@jhuapl.edu

Purpose • Provide Project Managers with the very basics of the two primary types of Clouds and Cloud Computing, and the questions they should ask when Clouds and their project intersect

Overview • “Computing as a Service” Clouds • Questions PMs should ask • “Data-Focused” Clouds • Relational Databases vs Clouds • Map-Reduce and Accumulo examples • Questions PMs should ask • General Cloud questions PMs should ask • The importance of risk assessments

What’s a Cloud? • Two primary definitions of Clouds presented today: • Compute-power as a Service (Utility Cloud; VMs) • Infrastructure as a Service or • Platforms as a Service or • Software as a Service • A Data-focused Cloud that also runs on VMs • E.g. Hadoop Data File System and data processing • A third emerging type is a “Data Storage” Cloud • PMs need to make sure everyone understands which type is being discussed • If you think you’re discussing a different one, confusion results and expectations will not be met

First Type: Computing as a Service • Instead of using your own computers, you use a Third-Party’s computers at another location (e.g., AWS’s EC2) • Usually all same hardware with a variety of Virtual Machine (VM) configurations to meet customer needs • When hardware dies, it is seamlessly replaced • All hardware and infrastructure and physical security headaches are the responsibility of the Third Party • You’re responsible for secure comms to and from the data stores and the security on the machines you use • You only pay for what you use (memory, computing power or number of virtual machines used) • Great for surge-type activities, such as the census that’s run every ten years, or new venture start-ups • Virtual private clouds are available for better security

First Type: Questions PMs Should Ask –1 • What’s the cost per data stored (Cents per Gigabyte)? • What’s the cost for number of VM’s used? • How secure or private is my data when I store it on a third-party platform? • What security or privacy guarantees are provided? • Will the PII be adequately protected? • Can I test Cloud security before I put real data there? • Would a Cloud be useful for my Continuity of Operations (COOP) plans? • It depends. Do your employees already regularly perform remote operations like teleworking? Do you have a re-routing plan to get them to the Cloud? • Am I starting a new business with limited investment?

First Type: Questions PMs Should Ask – 2 • Can you store classified data on a cloud? • If a properly secured government-accredited private cloud, Maybe • If you are planning to use a Third-Party service, Maybe • As a minimum, use a virtual private cloud (e.g., AWS VPC) • And located entirely in the U.S. (not distributed world wide) • Probably need to limit access to selected personnel at the service provider site (like no foreign access in US Gov Cloud) • US-Gov-only Cloud important for data under export control • Need your security department’s approval, which includes your plan and vetting the provider • Probably need to do penetration testing before use, like “side channel attack” prevention • Not sure if this is yet being used for more than unclassified but sensitive data • For either case, always get a cyber security expert to prepare a risk assessment, and for classified data, a proper accreditation

2nd Type: Data-Focused Cloud–Definitions • Huge Data: Petabytes or larger amounts of data • HDFS is Hadoop Data File System (more on this later) • Relational Database: Think rows and columns, densely populated (like a spreadsheet) • Structured non-relational databases: Cloud-based structured data technologies like Accumulo and HBase running on HDFS • Can be densely or sparsely populated • Tend to use flexible labels of length three to six (more later) • Many different types of data that may have some overlapping elements, but not the same across all types of data • If put into rows and columns it would be a huge table only sparsely populated

Relational Database Example Name Address Age Height John Smith 35 Washington DC 5’10” Jane Doe Baltimore 29 5’8” Fred Flintstone Rockville 55 4’10” Tony D. Tiger Battle Creek 67 6’2” Elmer Fudd DeForest 60 4’6” Peter Parker New York 28 5’5” Gotham Bruce Wayne 36 6’1” Fantasyland Roger Rabbit 41 4’0” Rural Address Peter Rabbit 118 1’1” Wonderland White Rabbit 135 1’11” Find the Namesof those of Age >25 but <60, and > 5’ tall

Sparse Data Example Drivers Licenses Dating Service Facebook Medical Records John Smith John Smith Washington DC Age 35 5’10” Jane Doe Age 29 5’8” Baltimore Peter Parker Peter Parker Age 28 5’5” New York Bruce Wayne Bruce Wayne Gotham 36 6’1” Find the Namesof those of Age >25 but <60, and > 5’ tall from multiple data sets

Accumulo Data Example ID Col. Family Col. Qualifier Time Security Value 31 Apr ‘12 Name Personal 001 PII John Smith Age Personal 31 Apr ‘12 PII 001 35 31 Apr ‘12 Height Personal 001 PII 5’ 10” 31 Apr ‘12 001 City Address PII Wash DC 31 Apr ‘12 Street 001 Address PII K Street 31 Apr ‘12 Number Address PII 001 810 31 Apr ‘12 Name Personal 002 PII Peter Parker Age Personal 31 Apr ‘12 PII 002 28 31 Apr ‘12 Height Personal 002 PII 5’ 5” 31 Apr ‘12 002 City Address PII New York 31 Apr ‘12 Street 002 Address PII 72nd Street 31 Apr ‘12 Number Address PII 002 145 Find the Namesof those of Age >25 but <60, and > 5’ tall

2nd Type: Data-Focused Cloud • Also runs on a VM farm, but uses a “Hadoop” or “Sector” file management system (Hadoop is most widely used) • What does a Hadoop Data File System (HDFS) do for you? • Let’s you store huge amounts of non-relational data • Automatically parallelizes the computations • Automatically sorts results of “map” step • Handles all of the overhead associated with storing, locating and processing your data • Allows for Map-Reduce programs and Direct Access Table-based searches using Hadoopto be run • Can find relationships not easilyvisible in unstructured data and/or large amounts of data

2nd Type: Map-Reduce Program Example • Find the number people per household in census data Key = #, Value = Total Key = HH Size, Value = # Distributed Databases of Household (HH) Census Data Count members of HH Hadoop Auto Sorts Add # HH w/ N members, N = 1 to 25 S 1, 3.5 M HH001, 3 HH001, 3 S HH002, 6 HH004, 3 2, 9.6 M S HH003, 4 3, 6.8 M HH003, 4 S HH004, 3 HH002, 6 4, 5.3 M Reduce Map

2nd Type: Map-Reduce Pros and Cons • Map-Reduce programs are good for: • When you have huge data sets • If your data can't be managed in a relational database • When you are not sure what types of queries you will want to run • If you want to summarize the results of independent processes that can be applied to data in parallel • Map-Reduce programs are not good for: • If you can answer your questions with an existing relational database in a reasonable amount of time, why bother with the overhead of a cloud? • If your data can fit within a relational database, AND • If the queries you plan to run are fairly well-defined THEN • You probably don’t need the overhead of a Cloud

2nd Type: Questions PMs Should Ask – 1 • Do I even need to use a Cloud? • If you have well-structured reasonable amounts of data, stick with a relational database UNLESS you just want the compute power on demand (1st Type of Cloud presented) • If it is required by external authorities (like a customer), yes • Do I have a lot of "surge" events, where you only need to store and process large amounts of data periodically • Then using a cloud makes sense • Do I need to know how to write a Map-Reduce program or an Accumulo Table to use a Cloud? • No, can use pre-defined programs, OR you need someone who knows how write new ones for you • Do I need to know how to design a Map-Reduce program? • No, but it helps so you can ask for realistic output from the Cloud and really leverage the Cloud to solve your data problems

2nd Type: Questions PMs Should Ask – 2 • Do I have access to an existing Cloud I could use? • If it meets your requirements, third-party Clouds work • Make sure of the “fine print” on the guarantees, and whether the recourse of the guarantee is sufficient to match the cost of the failure to guarantee • Have a security expert do a risk assessment before committing • Do I need to build my own instead? • If you have security, privacy or proprietary needs not met by an existing Cloud, might want to build your own • Consider the ongoing maintenance costs (may be primary rationale for moving to a Cloud) • More automation reduces Cloud maintenance costs

General Cloud Questions for PMs • Where is the Cloud located? Can it be restricted to U.S.? • Who gets access to it? • How are the communications to/from the cloud secured? • How does it ingest its data? • How does it store its data? • How do they secure your data at rest? • How does it delete its data? Can you test that it’s gone? • Does it keep your data separate from other people's data? • Do you need/want a virtual private cloud instead? • How often is the hardware upgraded? • How many versions of VMs can you choose from? • Has a security expert performed a risk assessment?

Summary Observations • Cloud computing is here to stay • Many more projects in the future will encounter Clouds in some way that will impact the project • Need to be aware of the strengths and limitations of Clouds and whether they are appropriate for your project • You may not have a choice whether or not to use a Cloud • This briefing listed some of the basic questions you should ask as appropriate to your project • Hopefully some of the mystery (and hype) of the Cloud has been dispelled by this talk • It is useful to be able to design a Map-Reduce program so your expectations of the output are realistic • Always do a cyber risk assessment on a Cloud you plan to use

Contact Info Dr. Patrick D. Allen Johns Hopkins University Applied Physics Lab 11100 Johns Hopkins Road MS 21-N246 Laurel, MD 20723-6099 443-778-9915 v 443-778-3838 f Patrick.allen@jhuapl.edu

Back-up: Terminology Relationship GOOGLE APACHE Structured Data Big Table HDFS Accumulo Map Reduce Environment Hadoop (Map Reduce) Map Reduce Hadoop Data File System (HDFS) File System Google File System (GFS)

Back-up: Sample Map Reduce Program • Map algorithm Map (key: sourceURL, value: text) { for each (targetURL in text) EmitIntermediate (targetURL, sourceURL); } • Reduce Algorithm Reduce (key: targetURL, value: sourceURL) { sourceList[] = null; for each (u in sourceURL) add sourceList[sourceURL]; Emit (targetURL, sourceList[]); }

Back-up: Map Reduce Example 2 For each URL, find all the pages that point to it. targetURL a – URL1 Create list for targetURL a targetURL a – URL1 targetURL a – URL2 targetURL b – URL1 Find targets for source 1 Doc 1 sorted targetURL – sourceURL list targetURL b – URL1 targetURL a – URL2 Create list for targetURL b Find targets for source 2 Doc 2 targetURL b – URL10^9 targetURL c – URL2 targetURL c – URL2 Create list for targetURL c targetURL c – URL10^9 Find targets for source 10^9 targetURL b – URL10^9 Doc 10^9 targetURL c – URL10^9 Create list for targetURL d targetURL d – URL10^9 targetURL d – URL10^9

Cloud Computing: What a Project Manager Needs to Know