MPLS-TE Doesn’t Scale Adrian Farrel Old Dog Consulting adrian@olddog.co.uk

1. MPLS-TE Doesn’t ScaleAdrian FarrelOld Dog Consultingadrian@olddog.co.uk www.mpls2007.com

2. Is the Sky Falling? • The only way to get your attention is to be alarmist • MPLS-TE is perfectly functional in today’s networks • But: • MPLS-TE will not scale indefinitely • The problem is the well-known “full mesh” or “n-squared” problem • The number of LSPs scales as the square of the number of PEs

3. What Do We Want to Achieve? • MPLS-TE is an important feature for many SPs • Allow traffic to be groomed • Optimize use of network resources • Provide quality of service guaranties • Carriers look to provide edge-to-edge tunnels across their core networks • Differentiated Services • VPNs • VLANS and pseudowires • Multimedia content distribution • Normal IP traffic

4. What is the Scope of the Problem? • Consider a service provider network with 1000 PEs • This is not outrageously large • Such a network may be broken into areas or ASes • Consider a full mesh of PE-PE TE-LSPs • Consider parallel tunnels for different services, QoS levels, and for protection • May give rise to multiples of 999,000 LSPs in the core • What is the capacity of a core LSR? • What is the capacity of a management system?

5. What Are the Scaling Limits? • Management • NMS • How many LSPs can the NMS process • Management protocols • Reporting on large numbers of LSPs may overload the management network • LSR issues • Memory capacity • Per LSP data requirements • CPU capacity – largely an RSVP-TE protocol issue • Degradation of LSP setup times • Soft state addressed by Refresh Reduction • MPLS forwarding plane • Number of labels (Only 1048559 per interface)

6. The Snowflake Topology • Example network for analysis • Meshed core of P nodes • Called P1 nodes • Each Pi+1 node connected to just one Pi node • PE nodes connected to just one Pn node • Well-defined connectivity and symmetry allows many important metrics to be computed • Number of levels & number of nodes per level may be varied • We can vary the number of P1 nodes • We can vary the ratio of Pi+1 to Pi • We can vary the value n • We can vary the number of PE nodes per Pn node PE P2 P1

7. Analysing the Snowflake Topology • Define • Pn a node at the nth level (level 1 is core) • Sn the number of nodes at the nth level • Mn the multiplier at the nth level (how many Pn+1 nodes are connected to a Pn node) • Ln number of LSPs seen by a Pn node • Discover • LPE = 2*(SPE - 1) • L2 = M2*(2*SPE - M2 - 1) • L1 = M1*M2*(2*SPE – M2*(M1 + 1)) • Practical numbers • S1 = 10, M1 = 10, and M2 = 20 • SPE = 2000 • LPE = 3998 • L2 = 79580 • L1 = 756000

8. The Ladder Topology • Example network for analysis • Core of P1 nodes looks like a ladder • Similar to many nationalnetworks • Symmetrical trees subtendedto core • Each Pi+1 node connected to just one Pi node • Each PE node connected to just one P node • Again: • Well-defined connectivity and symmetry allows many important metrics to be computed • Number of levels & number of nodes per level may be varied

9. Analysing the Ladder Topology • Same definitions as for snowflake network • E the number of subtended edge nodes (PEs) to each spar-node (E = M1*M2) • Discover • LPE = 2*(SPE - 1) • L2 = 2*M2*(SPE - 1) - M2*(M2 - 1) • L1 ≈ E*E*S1*S1/2 + E*E*S1 + 3*E*E - E*M2 • Practical numbers • S 1 = 10, M1 = 10, and M2 = 20 • E = 200 • SPE = 2000 • LPE = 3998 • L2 = 79580 • L1 = 2516000

10. Option 1 – Solve a Different Problem! • If a full mesh of PE-PE LSPs is too big, don’t build it! • This is the bottom line if we don’t fix the problem • The suggestion is to build a full mesh of Pn-to-Pn LSPs, and perform routing or routing-based MPLS between Pn and PE • Scaling improves from O(10002)to O(1002) • But we lose functionality • Why did we want a PE-PE mesh? • How do we handle private address spaces? • What if the traffic is not routable? • This may simply not be good enough to provide the function

11. Option 2 – LSP Hierarchies • Well-known, core MPLS function • Label stacks • Forwarding Adjacencies (RFC 4206) • Configured or automatic grooming • Possible to build a full or partialmesh of hierarchical tunnels • For example connect all P2 nodes • Each P2 node must encapsulate each PE-PE LSP in the correct tunnel • Each P1 node only sees the P2-P2 tunnels

12. Scaling Properties of Hierarchies - Snowflake • Note that PE-PE tunnels don’t help • P1-P1 tunnels are also no benefit (core is fully meshed) • P2 nodes see all PE-PE LSPs and new tunnels • L2 = M2*(2*SPE - M2 - 1) + 2*(S2 - 1) • Situation at P1 nodes is much better • L1 = M1*(2*S2 - M1 - 1) • Numbers (S1 = 10, M1 = 10, and M2 = 20) Flat 2-Level Hierarchy SPE 2000 2000 LPE 3998 3998 L27958079778 L1 756000 1890 • Maybe insert another layer (P3 ) to increase the scaling? • L3 remains high

13. Scaling Properties of Hierarchies - Ladder • Note that PE-PE tunnels don’t help • But P1-P1 tunnels are good because core is not fully-meshed • L1≈ S1*S1/2 + 2*S1 + 2*E*E*(S1 - 1) - E*M2 - 2 • Another level of hierarchy is also possible • Add a mesh of P2-P2 tunnels • L1 = S1*S1/2 + 2*S1 + 2*M1*M1*S1 - M1(M1 + 1) – 2 • L2 = 2*M2*(S(PE) - 1) - M2*(M2 - 1) + 2*(S(1)*M(1) - 1) • Numbers (S 1 = 10, M1 = 10, and M2 = 20) Flat 2-Level 3-Level Hierarchy Hierarchy SPE 2000 2000 2000 LPE 3998 3998 3998 L2 79580 79580 79778 L1 2516000716060 1958

14. Issues and Drawbacks for Hierarchies • Scaling is not good enough! • Impact on layer adjacent to PEs is negligible • Actually impact is slightly negative • Management burden • Plan and operate a secondary mesh • Effectively the same burden as managing PEs or a layered network • Possible to consider auto-mesh techniques • Fast Reroute protection is a problem • FRR struggles to protect tunnel end-points • Not obvious how to arrange the hierarchy when the network is not symmetrical • E.g., some PEs closer to the core

15. Option 3 – Multipoint-to-Point LSPs • LSPs merge automatically as they converge on the destination • Reduces the number of LSPs toward the egress • Other LSP properties (e.g.,bandwidth) must be cumulative • TE is still possible, butde-merge is not considered • Should count “LSP state” not number of LSPs • New definition • Xn the amount of LSP state held at each Pn node • For flat and hierarchical networks: • Each LSP adds one state at ingress or egress • Each LSP adds two states at each transit node

16. Scaling Properties of MP2P LSPs - Snowflake XPE = 2*(SPE - 1) X2 = SPE*(M2 + 1) X1=M1*M2*(S1 - 2) + SPE*(M1 + 1) • Numbers (S1 = 10, M1 = 10, and M2 = 20) Flat 2-Level Hierarchy P2MP SPE 2000 2000 2000 XPE 3998 3998 3998 X215916015935842000 X1 1512000 3780 23600

17. Scaling Properties of MP2P LSPs - Ladder XPE = 2*(SPE - 1) X2 = (M2 + 1)*S1*E X1≤ (4 + M1)*S1*E - M1*E • Numbers (S1 = 10, M1 = 10, and M2 = 20) Flat 2-Level 3-Level P2MP Hierarchy Hierarchy SPE 2000 2000 2000 2000 XPE 3998 3998 3998 3998 X2159160 159160 159358 42000 X1 5032000 1433998 3898 26000

18. Issues and Drawbacks for MP2P LSPs • Clear scaling benefits • Better than flat networks • Only thing that improves the situation adjacent to PEs • But… • Data plane support • This will only ever be a packet/frame/cell technology • Control plane support • RSVP does have MP2P support • RSVP-TE features not yet specified or implemented • De-aggregation and disambiguation • May be necessary to use label stack so that egress can detect sender of data • OAM may be more complex and require source labels • New management applications needed • FRR still to be designed

19. Other Topics for Investigation • Cost-effectiveness of the network • Revenue only generated by PEs • K = S(PE)/(S(1)+S(2) + ... + S(n)) • Many ways to improve scaling reduce cost-effectiveness • Fast Reroute • What are the implications of FRR to scaling? • Can scaling contributions be designed that can be protected by FRR? • Point-to-multipoint • What are the scaling properties of P2MP MPLS-TE? • Domain boundaries (in particular AS boundaries) • Boundaries such as at area and AS borders cause constrictions • How can we reduce the number of LSPs seen by ABRs and ASBRs?

20. Conclusions, Next Steps, and References • MPLS-TE is not a scaling issue today • But it won’t scale arbitrarily • We need to plan now for tomorrow’s scalability • Hierarchical LSPs are not as good as expected • MP2P LSPs may offer a better solution • More research and implementation is needed • draft-ietf-mpls-te-scaling-analysis-01.txt • Seisho Yaukawa (NTT) • Adrian Farrel (Old Dog Consulting) • Olufemi Komolafe (Cisco Systems)

21. Questions? adrian@olddog.co.uk