An Approach to IP Network Traffic Engineering

1. An Approach to IP Network Traffic Engineering NANOG Miami, FL Chris Liljenstolpe Cable & Wireless chris@cw.net

2. Scope and Purpose • Describes C&W’s Traffic Engineering methodology as well as some of the reasoning behind it. • Not “The One True Way,” but a method that works for us.

3. Scale of the Previous Design • Originally a flat network - one layer of routers interconnected over a complete PVC mesh. • A Network “event” in 1998 on AS3561 “educated” the engineering staff on IGP scaling issues. • This “event” lead to a week of network instability as it was re-engineered. • At one time there were 380+ routers in the direct mesh, accounting for 30k PVCs in the network & 760+ direct IGP associations per router.

4. Hierarchy • At one time there were 380+ routers in the direct mesh, accounting for 30k PVCs & IGP associations. • Currently there are no more than 80 routers in any one mesh due to the addition of hierarchy. • Due to the shrinking mesh sizes, and code optimization efforts, calculation times have dropped from 4 hours to 20 minutes.

5. Online vs. Offline • We like to always know where our traffic is and where it is routed. • Calculating optimal routing takes time on dedicated compute platforms…

6. Layer 2 vs. Layer 3 • Utilizing IGP metrics to adjust traffic flows on an IP network leads to network-wide (and sometimes/usually, unplanned) effects in a large network, due to flooding. • This can lead to the network equivalent of the midway game “Hit the groundhog”

7. IGP Use • The IGP (in our case 2 level IS-IS) is only used for link state signaling in normal and most failure mode conditions. • In the worst case dual failure mode condition, the IGP does provide real next-hop calculations.

8. IGP Metrics • Because of the direct router-router adjacencies provided by the underlying network, a large set of IGP metrics are not needed. • The set in use is small, and only used to select primary vs. secondary path, and discourage “expensive” link utilization in a multi-point failure that leads to multi-hop routing.

9. ATM to MPLS for TE • ATM w/ PVC’s worked quite nicely • Except for ATM overhead • And lack of high-speed router interfaces • For our traffic engineering network, we are treating MPLS as an IP friendly ATM (actually more like Frame Relay, but never mind)

10. Will ’s Replace MPLS? • Only when the bandwidth required for any router-router pair approaches the bandwidth available from a single  on the DWDM plant AND the cost of a port on an OXC is significantly cheaper than an equivalent bandwidth port on an MPLS switch. • When that occurs, the ’s will be provisioned just as the MPLS LSP’s are – statically with resilience. • GMPLS may be the technology used to signal the path over the OXC, just as MPLS is used for the LSP’s today.

11. Tools • Currently the tools that compute the paths, and configure the layer 2 and layer 3 equipment with those paths are all developed and maintained in-house. • Some have been in continual development and “tweak” mode for 6 years.

12. Futures • Most link failures will be detected and handled at the layer 2 traffic engineering layer, instead of at layer 3. • Path redundancy will grow from 2 to 4 paths per router-router pair. • Developments optimization mathematics originally researched for circuit path layout and analog circuit design will be utilized in the path layout tools. • Networks other than the IP backbone will utilize the traffic engineering core.