1 / 18

Big Internet Techniques Josh Gagliardi – CTO, Highwinds UCF COT 4210 12 Apr 2012

Big Internet Techniques Josh Gagliardi – CTO, Highwinds UCF COT 4210 12 Apr 2012. Company Profile Overview. Global provider of CDN, IP Software and Content Storage $100M business, highly profitable and financially stable

ulmer
Download Presentation

Big Internet Techniques Josh Gagliardi – CTO, Highwinds UCF COT 4210 12 Apr 2012

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Big Internet Techniques Josh Gagliardi – CTO, Highwinds UCF COT 4210 12 Apr 2012

  2. Company Profile Overview • Global provider of CDN, IP Software and Content Storage • $100M business, highly profitable and financially stable • High-capacity backbone with >3 terabits per second of extensively peered network capacity and PoPs on three continents • Offices: Winter Park, FL; Phoenix, AZ; Amsterdam, Netherlands; Costa Mesa, CA; São Paulo, Brasil • 175 employees

  3. Network Features • >3.5 Tb/s total capacity to the Internet • >650 Gb/s sustained traffic • 56 POPs on 3 continents • 12,000+ BGP peering sessions with end-user networks • ~32 PB of storage across 3,000 servers(<2% NAS/SAN) Code/OS Statistics • ~2.5 million lines = ~1 million production, ~1.5 million test • C++, Java, Perl, PHP, Flex, bash • 85% server, 10% client, 5% glue • 95% non-DB storage

  4. Bandwidth

  5. Mind-Numbing Complexity

  6. Simplified for Customers

  7. Simplified for Customers

  8. Problem Decomposition • Designing • Bootstrapping • Scaling

  9. Design • WHAT IS THE ESSENCE OF THE PROBLEM? • Define State • Ω(n) – proxies needed • count the 1:many mappings • count the IOs

  10. Bootstrap • Other People’s Code • Other People’s Gear • Generic Solutions • DB, SAN, Apache, PHP • Launch early / often

  11. Scale • This is where fortunes are made. • Internet -> quick success, quicker failure • Statistics • Statistical Process Control • Tricks -> { ∑x , ∑x2, N } -> stdev; R; Processing.org • Systems • Hashsplit / Random • Approximate / Fixed-Space Algorithms

  12. US and Europe Results

  13. The Gear Problem

  14. The Lifetime Problem • Rate of Technology Change • Snake Oil ( HA, SSD, cloud computing ) • 1996 – 2012 programmer view • IDE -> EIDE -> ATA -> SATA • SCSI -> iSCSI/FCAL -> SSD • Peeing in the Pool • Microsoft / HTTP / firewalls / the Location header • Criminals / Fraudsters / Script Kiddies

  15. Fixed-Space Algorithms • Router Guys – George Varghese • Big Data – {ORCL++, EMC++} • Algorithmic Techniques • Probabilistic Counting • Bloom Filters • Parallel Distributed Filters • Ratkiewicz Trending • Perspective Shift • Long Tail / The “Other” bucket

  16. The “Other” Bucket

  17. Josh Gagliardi CTO, Highwinds +1.407.215.2432 josh.gagliardi@highwinds.com About Highwinds: www.highwinds.com Programming Internships Available

More Related