MAX experience with capacity planning

1. MAX experience with capacity planning Dan Magorian, MAX Director of Engineering and Operations magorian@maxgigapop.net Quilt Peering Workshop, St Louis MO Oct 13, 2006

2. Planning?? You expect us to be able to plan in this environment?!! Quilt Peering Workshop, St Louis MO Oct 13, 2006

3. No, seriously… • Actually, I wanted to capture a conversation I had with Tim Lance (Nysernet) yesterday, following his presentation on construction of their 32 AoA NYC pop, and following Thurs’ business model panel. • Most RONs and XPs are run as businesses. However much we’re part of higher ed, however much of a mission we have to facilitate research, in the long term our growth and sustainability come down to making good business decisions. • We have income from our participants, and expenditures building and running our networks. Those better at least balance or you won’t be around long, unless you’re a funded research project. • Whether you’re trying to decide whether to build out a new pop, or set service pricing, are both fundamentally business decisions. Quilt Peering Workshop, St Louis MO Oct 13, 2006

4. So what is a business decision? • A fancy term for an educated gamble. • You try to figure what you need to do, then you go for it (or not) and make it happen. At the end of the day, you see whether you were right or wrong. During the day, there’re sometimes options to change or bail, and sometimes not. But there’s always exposure. • For example, 3 years ago we pulled the trigger on a new suite at Level3 in Mclean. We were in a dilemma because only a tiny suite was left on the first floor, and the second floor didn’t have dc power. We decided to go for it, which in 20/20 hindsight was the wrong decision, because we rapidly outgrew it and now are facing a complex migration to an expansion space. Also, we decided to place a T640 in Mclean, which some people opposed because where NewNet would be was unknown at the time, but which turned out to have been a right guess. Lose some, win some. Quilt Peering Workshop, St Louis MO Oct 13, 2006

5. The point about planning is • It’s the process of educating yourself about the business decisions (gambles) you’re going to take, so that you win more than you lose • Whether it’s $500k for a router or $12M for a new pop, you have financial exposure upfront before you know whether you guessed right or wrong. This is the part most of us don’t like very much. While people in higher ed aren’t usually fired for white elephants, in companies execs making wrong $1B merger decisions are. • So whatever the amount, you sometimes find yourself betting the farm. Gambling a lot of your organization’s cash on a business decision that looks good to the best of your planning ability, but which may not turn out to be right, often for reasons not under your control. Build it, and hope that you guessed right that they’ll come. Quilt Peering Workshop, St Louis MO Oct 13, 2006

6. So what does this philosophy about business decisions have to do with optical capacity planning? • A lot, actually. • MAX runs a regional optical network in the Washington/ Baltimore/Northern VA area with 7 pop locations and 400 fiber miles in 4 production and 5 research overlapping rings, lit by 4 different dwdm systems. We’ve done this since 2000.  • We cover a small footprint, but we're very complex. • R&E nets are not a traditional layer 1 capacity planning environment, where transport engineers provision based on requirements thrown over the wall by data engineers whose network traffic doubles every 18 months.  • These are the “revenue-based networking” folks who come to NANOG (revenue-based and cost-based are Dave Farmer’s terms). Quilt Peering Workshop, St Louis MO Oct 13, 2006

7. So is R & E “cost-based networking” or “politics-based networking”? • Our R&E environment often needs massive overprovisioning to accommodate big flows with good performance. • That needs to be accomplished despite understaffing and often chronic underfunding. We’re not telcos trying to gouge customers for every Mb, with staffs of thousands to push paperwork. • So often capacity planning is more about horsetrading fiber and other resources to get to where you’re guessing you need to be next year, as much for politics as for actual capacity growth.  • This leads to far more interesting heterogeneity and complexity than would be the case for a revenue-based network able to spend $500M to build out a shiny new homogeneous optical network, who knows what their customers need and builds based on traffic. Quilt Peering Workshop, St Louis MO Oct 13, 2006

8. “Enough philosophy, how about some methodology?” • Max sells ip, mpls, and lambda transport services over 4 production dwdm rings. Since we have rings, we try to provision lambdas with optical protection where possible. Fiber cuts happen. • Recently we’ve had high growth in 10G lambda demand. Due to dropping 10G xponder costs, we now only provision 10G lambdas, and will serve gige lambdas via muxponders (8 giges to a 10G). This works because people rarely want OC48 2.5Gs, usually giges • Due to Movaz’ low 10G chassis density (2 10Gs with protect per 4U RayExpress), we are using add-in dwdm from Ekinops, a French startup. These are muxed into the Movaz as protected external waves (aliens). Still in test because their software is early, and unclear still whether space savings trades against complexity. Quilt Peering Workshop, St Louis MO Oct 13, 2006

9. Dwdm system cost tradeoffs • Interesting comparison of "system" vs "component" dwdm costs. A pair of 10G dwdm xponders is currently $6k-10k/pr bought as components (or as parts of unknown-brand dwdm systems ) • But we're currently paying $24k/pr for system-integrated xponders from Movaz and Zhone/Luxn. This says that there's a "systems integration tax" of 250-400% that you pay for stuff like Movaz developing all their software inc GMPLS, which is paid for by their hardware costs. • Nothing new there; Cisco, Juniper etc all do that, it's how the industry works. Quilt Peering Workshop, St Louis MO Oct 13, 2006

10. More dwdm system cost tradeoffs • But there are also "bare bones" dwdm systems that have very little integration or software, eg MRV and JDSU. These are basically all sfps and xfps, that have a lower markup over the component costs, and offer fewer bells and whistles like variable attenuation etc. • More data would be interesting to characterize vendors on this scale from components on one end to "economy" like JDSU & MRV to "midrange" like Movaz to "Cadillacs" like Cisco 454s and Ciena Corestreams. • Personally I'm a fan of DIY optical systems, but on production nets it's hard to get away with anything ad hoc. Your price of failure is so high that it dwarfs any difference in xponder costs. In the R & E world, people are more concerned with the diff between what the optical systems guys are charging, which are considered cheap, vs the router guys who are still trying to get $75k for one 10G int. Quilt Peering Workshop, St Louis MO Oct 13, 2006

11. So where are we going?  • With emerging next-gen ROADMs signalling to tunable lasers for capacity on demand, the research picture for dynamic resources looks good. Heard talks about DRAGON, GLIF, and others. • Getting this technology migrated from DRAGON and HOPI test networks out into the real world of production where things have to work well, is our next major challenge. • Make no mistake: transitioning from the current world of statically provisioned net resources to dynamic is a major sea change. • But the world is changing, and we have to position ourselves in that emerging world. All while betting the farm on business decisions that will hopefully work out right and create growth. Quilt Peering Workshop, St Louis MO Oct 13, 2006