Terabit Backbones A Reality Check

1. Terabit BackbonesA Reality Check Vijay Gill <vgill@mfnx.net>

2. Agenda • Current State of the Internet • Side detour through • VPNs • DiffServ/QoS/CoS • The Converged Core (hype machine that goes to 11)

3. State Of the Internet Address Reality Based Internet Economics • Amount of state at any level should be constrained and must not exceed Moore’s Law for economically viable solutions. • Ideally – growth of state should trail Moore’s Law We’re in trouble “If you’re not scared, you don’t understand” – Mike O’Dell

4. Growth of State • Recent trends show high growth in Internet state (routes, prefixes etc.) • Isolate this growth as a predictor of future growth • Compare growth to Moore’s law

5. Source: Tony Li (Procket Networks)

6. Source: Tony Li (Procket Networks)

7. The Very Bad News • Growth rate is Increasing • Hyper-exponential growth • Will eventually outgrow Moore’s law • Moore’s law may fail

8. Source: Tony Li (Procket Networks)

9. The Real Problems • If we don’t fix these, the other problems won’t matter • Hyper-exponential growth will exceed Moore’s law • Safety margins are at risk • We need concerted effort on a new routing architecture • Multi-homing must not require global prefixes • Example: IPv6 plus EIDs

10. BGP Advertisement Span • Nov 1999 - 16,000 individual addresses • Dec 2000 - 12,100 individual addresses • Increase in the average prefix length from /18.03 to /18.44. • Dense peering (Rise of Exchange Points) and Multi-homing Source: Geoff Houston

11. State Now • # of Paths vs. # of Prefixes • Large amounts of peering • CPU to crunch RIB to populate FIB • More state requires more CPU time • Leads to Delayed Convergence • BGP – TCP rate limited, just adding pure CPU isn’t the entire answer • Issue is with propagating state around

12. Convergence Times

13. Problem With State • Issues with interactions of increased state, CPU, and message processing • Run to completion processing <-> missed hellos • IGP meltdowns • Time diameter exceeds hold down • Pegged CPU on primary causes slave to initiate takeover • Decoupled Hello processing from Routing Process

14. VoIP? What VoIP? • IGPs Converge on average converge an order of magnitude faster than BGP • Leads to temporary black holing • Router reboots (like that ever happens) • IGP converges away, BGP teardown • Router comes back up • IGP converges and places router in forwarding path • BGP is still converging • Packets check in, but don’t check out

15. VPNs - Operational Reality Check • Vendors can barely keep one routing table straight • Customer Enragement Feature, IBGP withdraw bugs • Into this mess, we’re going to throw in another couple of hundred routing tables like some VPN proposals? • Potential for several thousand internal customer prefixes inside our Edge routers • Revenge of RIP • Provider Provisioned VPNs – Just Say No.

16. What Is Going to Work • Some people will optimize for high touch edges – Provider provisioned VPNs etc. • But if they are talking with the rest of the world, welcome to the new reality – It sucks. • For the Rest….

17. “Already, data dominates voice traffic on our networks” -Fred Douglis, ATT Labs These exhibits were originally published in Peter Ewens, Simon Landless, and Stagg Newman, "Showing some backbone," The McKinsey Quarterly, 2001 Number 1, and can be found on the publication's Web site, www.mckinseyquarterly.com. Used by permission.

18. What to optimize for • Optimize for IP • Parallel backbones • Some ISPs already have to do this based on volume of traffic for IP alone • Do not cross the streams • Voice traffic has well known properties • Utilize them • Optical network – Utilize DWDM and OXCs to virtualize the fiber

19. Solution Internet (IP) Internet (IP) VPN VPN Voice/Video Voice/Video CES CES Multi Service Optical Transport Fabric

20. NEWS FLASH • Simple & Stupid Trumps Complex & Smart Every Time • Networks Powered by PowerPoint ™ • Stuff looks good on slides, then we try and hire people to implement and operate it • Operational Reality Beats PowerPoint every time

21. The Converged Core ™ • For the fortunate few • Utilize OXCs + DWDM to impose arbitrary topologies onto fiber • For the rest trying to run IP over Voice… • Nice knowing you…. • Voice - Use SONET as normal, it’s not growing very fast, so don’t mess with it • WCOM, T

22. Network Design Principle • The main problem is SCALING • Everything else is a secondary • If we can scale, we’re doing something right • State Mitigation • Partition State • What you don’t know, can’t hurt you

23. Common Backbone • Application Unaware • Rapid innovation • Clean separation between transport, service, and application • Allows new applications to be constructed without modification to the transport fabric. • Less Complex (overall)

24. Why A Common Backbone? • Spend once, use many • Easier capacity planning and implementation • Elastic Demand • Increase of N on edge necessitates 3-4 N core growth • Flexibility in upgrading bandwidth allows you to drop pricing faster than rivals

25. By carrying more traffic, a carrier can lower costs by up to 64% These exhibits were originally published in Peter Ewens, Simon Landless, and Stagg Newman, "Showing some backbone," The McKinsey Quarterly, 2001 Number 1, and can be found on the publication's Web site, www.mckinseyquarterly.com. Used by permission.

26. Bandwidth Blender - Set on Frappe 18,000 16,000 14,000 PRICE 12,000 Price per STM-1 ($m) 10,000 8,000 6,000 4,000 COST 2,000 2005 1996 1997 1998 1999 2000 2001 2002 2003 2004 Historical and forecast market price and unit cost of Transatlantic STM-1 circuit (on 25 year IRU lease) Source: KPCB

27. Problem • We keep hearing the phrase ‘bandwidth glut’ • So are we experiencing a glut or not? • No matter how much terabits of core bandwidth gets turned up…. • Capacity Constraints are at the edges • Go drop 2-4 racks in colocation facilities • Q in QoS stands for Quantity, not Quality • We don’t need to boil the oceans, all we want is a poached fish

28. How To Build A Stupid Backbone • Optical backbones cannot scale at the STS-1 level • High speed backbone reduces complexity and increases manageability…. • Impose a Hierarchy • Optical Backbone provides High-speed provisioning/management: OC-192/48 • Sub-rate clouds multiplex lower speed traffic onto core lightpaths

29. Client equipment Core OXC Multi-Service Platform Regional-Core Network Infrastructure Core network Metro SubNetwork Metro SubNetwork Metro SubNetwork

30. Requirements • Support multiple services • Voice, VPN, Internet, Private Line • Improving service availability with stable approaches where possible • Use MPLS if your SONET ring builds are taking too long (anyone still building SONET rings for data?) • If you have to use MPLS….

31. Stabilize The Edge • LSPs re-instantiated as p2p links in IGP • e.g. ATL to DEN LSP looks like p2p link with metric XYZ • Helps obviate BGP blackholing issues

32. Stabilize The Core • Global instability propagated via BGP • Fate sharing with the global Internet • All decisions are made at the edge where the traffic comes in • Rethink functionality of BGP in the core

33. LSP Distribution • LDP alongside RSVP • Routers on edge of RSVP domain do fan-out • Multiple Levels of Label Stacking • Backup LSPs • Primary and Backup in RSVP Core • Speed convergence • Removes hold down issues (signaling too fast in a bouncing network) • Protect path should be separate from working • There are other ways, including RSVP E2E

34. Implementation • IP + Optical • Virtual Fiber • Mesh Protection • Overlay • We already know where the big traffic will be • NFL Cities, London, Paris, Frankfurt, Amsterdam • DWDM + Routers

35. IP + Optical IP / Routers • Virtual Fiber • Embed Arbitrary fiber topology onto physical fiber. • Mesh restoration. • Private Line • Increased Velocity of service provisioning • Higher cost, added complexity Optical Switching Fiber DWDM / 3R

36. Backbone Fiber Metro Collectors DWDM Terminal Optical Switch Core Edge

37. IP + Optical Network Big Flow Big Flow Out of port capacity, switching speeds on routers? Bypass intermediate hops

38. Dual Network Layers • Optical Core (DWDM Fronted by OXC) • Fast Lightpath provisioning • Remember - Routers are very expensive OEO devices • Metro/Sub-rate Collectors • Multiservice Platforms, Edge Optical Switches • Groom into lightpaths or dense fiber. • Demux in the PoP (light or fiber) • Eat Own Dog Food • Utilize customer private line provisioning internally to run IP network.

39. Questions Vijay “Route around the congestion, we must” Gill Many thanks to Tellium (Bala Rajagopalan and Krishna Bala) for providing icons at short notice! Nota Bene – This is not a statement of direction for MFN!