1 / 62

Build vs. Buy: Internet Datacenter Operations Research

Build vs. Buy: Internet Datacenter Operations Research. INTAP March 6, 2001 Tokyo. Agenda. My Background and Who is Equinix? – 15 minutes Recent Internet Operations Research – 20 minutes “Build vs.Buy Internet Datacenter” White Paper – 40 minutes

danae
Download Presentation

Build vs. Buy: Internet Datacenter Operations Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Build vs. Buy: Internet DatacenterOperations Research INTAP March 6, 2001 Tokyo

  2. Agenda • My Background and Who is Equinix? – 15 minutes • Recent Internet Operations Research – 20 minutes • “Build vs.Buy Internet Datacenter” White Paper – 40 minutes • Future: Where our Industry is heading – 5 minutes

  3. NSFNET to Post-NSFNET Era Internet Merit OarNet NyserNet NWNet • NSFNET (1987-1994) Regional Tech Meetings • Performance & Operations • Reporting & Coordination • Routing & Network Planning NEARNet BARRNet NSFNET Regional Regional Sesqui SDSC Regional NCAR OarNet Merit NWNet NyserNet ANS Privatization (1994-Now) North American Network Operators Group (NANOG) Meetings UUNet Sprint MCI Regional AT&T Regional Regional Regional MY JOB: 5 Years as Chair of NANOG 12 Years in Internet…No Internet Preso…

  4. Internet Statistics • 87.5% of all Internet statistics are made up. Source: I made it up. Source: I made it up

  5. Equinix, Inc. • Scale – size of Internet Business Exchanges, how many, where they are, future plans • “Virtual Fly Through” Tour of the IBX - 6 minutes

  6. Industry LeadingSecurity Fully RedundantBackup Power Customer FriendlySpaces – Conducting Business & Equipment Preparation • 5 security checkpoints to customer equipment • State of the art video imaging and archiving • On site security personnel 24x365 • Built to 3x power capacity • 3 levels of power backup • Forward looking design for growing industry • Designed from customer experience • Ergonomic work spaces IP/CarrierKnowledgeable Staff • 24x365 availability • Customer trouble-shooting • Smart Hands IBX™ Centers

  7. IBX™ Growth Sites currently open or Under Development

  8. Equinix Internet Business Exchange Model • Infrastructure • Players • Neutrality… Content Providers Neutral Internet Business Exchange Model Internet Service Providers Carriers

  9. Carrier Internet Exchange Model • Infrastructure • Players • Non Carrier-Neutral • No Content Support • Economic Distortions Carrier Operated Internet Exchange Internet Service Providers Carrier Operator

  10. Job Title: Equinix Co-Founder & Chief Technical Liaison • So, my job is to take friends out for lunches and dinners… • Internet Operations Research & • Write White Papers • Summary: What did I learn?

  11. No Internet Operations Documentation To make things more difficult, subtle language differences!

  12. The Internet Operators: All about relationships

  13. My Job: White Papers New Job Title: Equinix Co-Founder & Chief Technical Liaison Document Internet Operations Research • “Interconnection Strategies for ISPs” • “Internet Service Providers and Peering” • “Peering Case Studies” • “Peering Simulation Game” • “Build vs. Buy Internet Datacenter” today

  14. White Paper: Interconnection Strategies for ISPs • Direct circuits replaced with: • OC-12 into Exchange • Fiber cross connects • Cost Drivers: • Big OC-12 Direct Circuit • Cheap Fiber Cross Connects • Aggregation Efficiency over OC-12 (2:1,3:1) • Rack Fees • Exchange Model Cost Formula: ExchangeModelCost=BigDirectCircuit()+(n-1)*x/2+RackFees() OC-12 $23,000/mo XC=$200/mo Rack=$1500/mo So does the Exchange-based model make sense?

  15. Peering Peering Transit $$ WestNet USNet EastNet ISP A Transit ISP Routing Table ISP B Transit $$ White Paper: Internet Service Providers & Peering

  16. White Paper: Case Study: Peering in the Bay Area

  17. White Paper: The Peering Simulation Game

  18. White Paper: Build Vs. Buy Internet DataCenter • Modeling: • When does it make sense to build your own? • To Outsource? • Issues to consider • Economies of Scale? • Multiple Business Models

  19. Service Level Agreements in Principle • Guarantee a higher level quality • Define Measurement Criteria for adhering to higher quality service Level • Meeting Performance Level critical to success of business • Failure to meet Service Levels penalty significantly worse than cost of service • eBay: $64,000/minute of downtime

  20. SLAs in Practice • No SLA Pricing • ISPPriceNO SLA=X • SLA Pricing • ProbabilityOfMissingSLA=p • ISPPriceSLA1=X*(1+p) • Network does not typically change to meet new criteria • CIO Security Audit Story • Equinix Switch SLA: 99.999% Uptime

  21. SLA-Datacenters • State-of-the-art Datacenters • Infrastructure Failure unlikely • SLAs rare, but… • Typically Limited to free month service • (Insufficient to potential loss customer faces) • But what is an Internet Datacenter? • Include hosting? Include Internet Service? Include Caching, Content Distribution? • Models help Define and Frame the Problem…

  22. SLA Datacenter Parameters • If there were Internet Datacenter SLAs • Parameters: • Installation Lead Time • Service Availability • 24/7 Customer Support • Provisioning/Responsiveness • Independent of Transport and Internet Service Issues.

  23. Creating the Build vs. Buy Internet Datacenter Whitepaper • Create a community white paper on issues associated with the building & scaling of Internet Datacenters • Identify the issues that folks should consider when making the build vs. buy decision Today: • Demonstrate the document & models to date and get feedback NOTE WELL: • EARLY STAGE OF BVB WHITE PAPER (v0.4)! • This Model may be of help to you in your SLA Discussions

  24. Toured 32 (now) data centers

  25. Some with very Secure Fiber Access… Source: Technology Center of the Americas Tour

  26. Some with less secure fiber entry… Source: Don’t Ask.

  27. Some Very Secure… http://www.thebunker.net

  28. Some Cat 5 Hurricane proof…

  29. Some not quite cat proof. Loading Dock

  30. Great religious debate over ties Velcro vs. Plastic cable ties vs. String

  31. Time to white paper what I’ve seen & Facilitate BOF Discussion • Datacenter BOF I NANOG Oct ’98 Atlanta • Build vs. Buy White Paper • Internet Datacenter Model • Definition of Internet Datacenter • Focused discussion of Each Component of the Model

  32. Internet Datacenter Model Dependence Relationship Content & Support Internet Service Models Network Transport Business Models Data Center Operations Core Infrastructure Real Estate Increased Value

  33. Customer Business Model Value Content Support Internet Service Model(s) Network Transport Business Model(s) Data Center Operations Core Infrastructure Real Estate Dependence

  34. Internet Datacenter Model Example CP/e*/ASP/Support: BMA IBM Schwab Yahoo! Akamai Site Smith IXP Transit Mkt: C&W Nacamar TeleGlobe AT&T... Transport Mkt: UA WCom Level3 AtlMedia... Data Center Operations Core Infrastructure Real Estate Value Dependence

  35. Size Ultimate Goal Unit cost • Build Financial models that allow an ISP to say: • “Given our size requirements, it makes sense to build our own.” Or • “Given our size requirements, it probably makes sense to outsource this to someone who can realize economies of scale.” + + = Buy Build

  36. Place to put stuff Proximity to Fiber Space Size Utility Power Facilities Yard Flavor of the Real Estate Model Data Center Operations Core Infrastructure Real Estate Scaling of Datacenter Real Estate? Increased Value Relationship

  37. Scaling of the Real Estate Model $/sq. ft. Premium Real Estate Market (Tokyo, NYC,…) Real Estate Market (everywhere else) # Sq. Ft.

  38. Real Estate Size is Important • Choose Too Small: Run out of space, lose economies of scale • Choose Too Big: Cost per Rack very high Vs. • Right of First Refusal (some Internet Datacenters allow this) • Expandability of Space

  39. Commitment to DataCenter • 1 yr contract Vs. • 10 year lease Time to Market Issue – 6 – 9 months to build

  40. Hidden Charges • Meet Me Rooms • Required products: • Must buy port on switch or certain volume of BW • Cross Connect Fees

  41. Size & Finding Real Estate • 100,000 sq ft in downtown DC? • 10,000 sq ft in downtown DC • Vs. • Fiber Miles to Internet Datacenter

  42. Heating Venting Air Conditioning (HVAC) Security Infrastructure Power Infrastructure Fire Suppression Flavor of the Infrastructure Model Data Center Operations Core Infrastructure Real Estate

  43. Heating, Venting, Air Conditioning (HVAC) Infrastructure • Controlled Humidity ~45% • Too much Humidity: System Failure • Too Little Humidity: Static Electricity • Air Temperature:~72 (IBM: 65-85) • Solutions: • DX Systems • Air Cooled • Water-Based • How do these Scale?

  44. HVAC System Cost: Economies of Scale HVAC Cost HVAC Scaling Direct Expansion $1,000/ton Air Cooled $478/ton Chilled Water System $250/ton DX Systems: 25-300 tons of Cooling ~ 10K sq ft or ~150-200 racks # Racks Terramark Outsource of HVAC to FPL…

  45. Crossover point where Chilled Water lowest per-rack solution

  46. Trend? Outsourcing of HVAC FPL HVAC (Miami FL) Water Chiller NOTA: 3 Blocks Away Source: NAP of the Americas Tour

  47. Security Infrastructure • 24x7 Manned • Access: Biometrics, card keys • Video Surveillance Sliding Glass

  48. Power Infrastructure • Sun Netra 1U=90Watts • 40U Rack=3600 Watts! • Power Source • Power Grid (dual feeds) • Co-Generation • UPS/CPS/Generators • Energy Storage • UPS Batteries • Flywheel (no UPS) • Fuel Cells (service) • Monitoring Issues • Redundancy Issues • TBD: Scaling Issues

  49. Fuel Cells for Generators

  50. Fire Suppression Infrastructure • Two Dominant Systems: • VESPA air sampling • Dry pipe, pre action water system vs. FM-200 • Smoke & Heat Required to activate region

More Related