How to Secure Your Job: Implement MPLS VPNs

1. How to Secure Your Job: Implement MPLS VPNs roger@gottsponer.ch NANOG24 / Miami, Florida February 10-12, 2002

2. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

3. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

4. The Idea behind this Talk • This talk should show some lessons we learned whenimplementing and running a real life MPLS VPN network • Nextra Switzerland is running a national MPLS VPNnetwork since 2.5 years (Q3 1999) • about 30 PoPs, Cisco based • The Nextra Group is running a pan-European MPLS VPNnetwork since about one year (Q1 2001) • present in most European countries • provides seamless VPN services between autonomous Nextra companies • I will not discuss whether MPLS is a good thing or not :-)

5. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

6. Lessons Learned A reload a day keeps the TAC away just kidding…

7. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

8. Service Architecture • Is entirely based on MPLS VPN ‘Routing Realms’ • Each Added Value Service has its own Routing Realm/VPN • Each Customer has several Routing Realms/VPNs • Customers are given access to Services by interconnecting ‘Customer’ VPNs and ‘Service’ VPNs • Limited Set of Building Blocks • Full Mesh Routing Realm • Hub/Spoke Realm • (Point-Point Realm) • Inter-Realm ‘Conduits’ • Inter-Realm ‘Adapters’

9. Lego-Brick #1: Full-Mesh VPN

10. Lego-Brick #2: Hub & Spoke VPN Hub Spoke Spoke Spoke

11. Resource Lego-Brick #3: Conduits X Mbps

12. Resource Lego-Brick #4: Adapters

13. The Four Network Building Blocks

14. Multiple VPNs per Customer

15. Internet Managed Firewall IAS for PN Nextra Managed Firewall

16. Internet Standard VPN IAS Customer Managed Firewall

17. Redundant Load Sharing PN HSRP RedundantAccess POP

18. Disaster Recovery VPN Primary Computing Center (IP: 10.5.5.0/24) Backup Computing Center (IP: 10.5.5.0/24) Fast (?) Re-routing POP A POP B

19. ISDN PN Dial Backup VPDN Home Gateway

20. PartnerNet PartnerNet CustomerA CustomerB

21. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

22. The Use of Route-Reflectors • You do not want to maintain a full VPNv4-iBGP mesh • Use at least two VPNv4-RRs for redundancy reasons • Do not use the IPv4-RRs as VPNv4 Route-Reflectors • IPv4 Routing instabilities could affect VPN routing • If PE reloads it will get all 100’000 IPv4 routes first. So it will take several minutes until the VPN routes are restored • If RRs are separated, both feeds are done simultaneously • So you need at least four RRs --> what about core routers? • Use global-local-massive-extreme unique RDs • A different RD per VRF, not per VPN • otherwise RR will break iBGP load-sharing • Check next-hops of BGP routes you get from the RR :-)

23. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

24. Baby Jumbo Frames • MPLS labels are 4 Bytes long • Available MTU on a link gets decreased by x*4 Bytes • Due to Path-MTU-Discovery, the sender will detect this • Path-MTU-Discovery seems to be broken sometimes • Customer will blame you, not the broken PMTUD • “everything worked fine before we moved to your MPLS network!” • Workaround: • allow 1500+(x*4) Bytes packets over Ethernet (Baby Jumbo Frames) • Special config command: tag-switching mtu 1508 • oversized packets only allowed if oversize is caused by MPLS labels • Before you migrate to MPLS, make sure all your routers, interfaces and switches support Baby Jumbo Frames!

25. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

26. The LDP-ID • After a crash, a P router stopped forwarding any traffic... interface loopback99description test test testip address 222.222.222.222 255.255.255.255 • Like in OSPF, LDP uses highest loopback address as ID • Unlike in OSPF, LDP-ID has to be reachable LDP-ID not reachable  no LDP TCP-session  no labels! • So always configure the LDP-ID manually!

27. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

28. Traceroute stops at PE router ! Traceroute is no longer Your Friend

29. Traceroute stops at PE router ! A Multi ISP Environment Trans it ISP ISP B ISP A

30. Customer Complains about Strange Traceroute CPE-Router#traceroute www.cisco.com 1 accesslink.company.com (10.150.73.17) 0 msec 4 msec 0 msec <-- PE (Europe) 2 r16b01-pos0-0-0.bb.nextra.net (141.22.5.2) 104 msec 104 msec 100 msec <-- P (Europe) 3 r08b02-pos1-0-0. bb.nextra.net (141.22.1.6) 104 msec 104 msec 104 msec <-- P (Europe) 4 r08b01-fe4-1-0. bb.nextra.net (141.22.5.49) 148 msec 200 msec 200 msec <-- P (Europe) 5 r10b01-pos4-0-0. bb.nextra.net (141.22.5.34) 160 msec 200 msec 200 msec <-- PE (USA) 6 Pos4-0.GW8.NYC4.ALTER.NET (157.130.21.37) 100 msec 100 msec 100 msec 7 172.ATM2-0.XR2.NYC4.ALTER.NET (146.188.180.62) 104 msec 104 msec 104 msec 8 188.at-1-0-0.TR2.NYC8.ALTER.NET (152.63.21.106) 108 msec 104 msec 104 msec 9 124.at-6-0-0.TR2.SAC1.ALTER.NET (152.63.6.14) 180 msec 176 msec 180 msec 10 190.ATM6-0.GW8.SJC2.ALTER.NET (152.63.52.181) 184 msec 188 msec 184 msec 11 cisco.customer.alter.net (157.130.200.30) 188 msec 188 msec 188 msec 12 pigpen.cisco.com (128.107.240.170) 188 msec 184 msec 184 msec 13 * www.cisco.com (198.133.219.25) 184 msec *

31. What a Professional Troubleshooter Told Us :-) Hi,I think I can explain what is happening. Any IP packets received with IP options that require processing are process switched. That means that the packet is sent to the Route Processor for forwarding, rather than switched directly using CEF. If the RP is busy, then this may add to the latency.The traceroute command uses IP options in the header, so these packets will be process switched. Any other traffic, such as http, ftp etc will be switched using CEF. So when they run the traceroute command, they are seeing the latency of the packet being forwarded through the RP, rather than the true latency of the packets being fast switched by CEF.Regards,Your professional troubleshooter

32. The Real Reason • PE sends back ICMP Time Exceeded via appropriate Routing Table • P router has no routing information! • Only information he has is receiving LSP, but LSPs are unidirectional • So P router sends ICMP Time Exceeded via receiving LSP • ICMP packet travels through the MPLS backbone to the other PE, which then has the routing information to send the ICMP packet back through another LSP to the sender • So each traceroute probe travels through the whole MPLS backbone twice • TTL is high! • Customer can accept this because his productive traffic is not affected

33. But How to Explain this to a Customer? CPE-Router#traceroute www.cisco.com 1 accesslink.company.com (10.150.73.17) 0 msec 4 msec 0 msec <-- PE (Europe) 2 r16b01-pos0-0-0.bb.nextra.net (141.22.5.2) 104 msec 104 msec 100 msec<-- P (Europe) 3 r08b02-pos1-0-0. bb.nextra.net (141.22.1.6) 15 msec 12 msec 22 msec<-- P (Europe) 4 r08b01-fe4-1-0. bb.nextra.net (141.22.5.49) 16 msec 17 msec 14 msec<-- P (Europe) 5 r10b01-pos4-0-0. bb.nextra.net (141.22.5.34) 160 msec 200 msec 200 msec <-- PE (USA) 6 Pos4-0.GW8.NYC4.ALTER.NET (157.130.21.37) 100 msec 100 msec 100 msec 7 172.ATM2-0.XR2.NYC4.ALTER.NET (146.188.180.62) 104 msec 104 msec 104 msec 8 188.at-1-0-0.TR2.NYC8.ALTER.NET (152.63.21.106) 108 msec 104 msec 104 msec 9 124.at-6-0-0.TR2.SAC1.ALTER.NET (152.63.6.14) 180 msec 176 msec 180 msec 10 190.ATM6-0.GW8.SJC2.ALTER.NET (152.63.52.181) 184 msec 188 msec 184 msec 11 cisco.customer.alter.net (157.130.200.30) 188 msec 188 msec 188 msec 12 pigpen.cisco.com (128.107.240.170) 188 msec 184 msec 184 msec 13 * www.cisco.com (198.133.219.25) 184 msec * BUG!

34. Agenda • The Idea behind this Talk • Lessons Learned • The Service Architecture • The Use of Route Reflectors • Baby Jumbo Frames • The LDP-ID • Traceroute is no longer Your Friend • Some Security Considerations • no Religious Wars please :-)

35. Configuration Integrity • Sometimes config lines for a vrf routing enviromentappear on the global routing process level • Some buggy software versions • Typo in vrf name • Copy-Paste works too fast :-) • This can be very very very dangerous • Sometimes someone puts a interface into a wrong VRF :-( • The problem is not when you break connectivity (easy to spot) but when you add too much connectivity • Sanity Check Tools!

36. How it should look like router bgp 88 [usual iBGP stuff] no auto-summary ! address-family ipv4 vrf green redistribute connected redistribute rip neighbor 211.27.213.9 remote-as 65500 neighbor 211.27.213.9 activate default-information originate no auto-summary no synchronization exit-address-family What I found on a router router bgp 88 redistribute rip default-information originate Configuration Integrity

37. How it should look like router rip version 2 no auto-summary ! address-family ipv4 vrf blue version 2 redistribute bgp 88 metric 3 network 211.27.213.0 distribute-list 22 in no auto-summary exit-address-family What I found on a router router rip redistribute bgp 88 metric 3 network 211.27.213.0 Configuration Integrity • Imagine what happens if you use addresses from the 62/8 block! • Another reason for good filtering • (Preconfigure all possible redistri-butions with route-map “deny-all”)

38. Secure Your PE VTYs! line vty 0 4 access-class 99 in login authentication bbmethod access-list 99 permit 10.25.25.5 0.0.0.0 <-- NMS box access-list 99 permit 61.8.0.0 0.0.7.255 <-- BB-links • What happens, if customer sends hostroutefor your NMS box? • He will be able to telnet to your PE router!

39. Secure Your PE VTYs! • only use hostroutes for NMS • RIP, OSPF, EBGP has better admin distance than IBGP:-( • put hostroutes to Null0 for NMS into all VRFs • how to manage the CPEs? • filter routing updates from customer • do not accept all internal address ranges (NMS, BB) • secure, but could impact customer connectivity

40. Secure Your PE VTYs! • use better ACLs to secure the VTYs interface loopback0 ip address 1.1.1.1 255.255.255.255 access-list 199 permit 10.25.25.5 0.0.0.0 1.1.1.1 0.0.0.0 access-list 199 permit 61.8.0.0 0.0.7.255 1.1.1.1 0.0.0.0 access-list 199 permit 10.25.25.5 0.0.0.0 61.8.0.0 0.0.7.255 access-list 199 permit 61.8.0.0 0.0.7.255 61.8.0.0 0.0.7.255 • needs entries for BB interface addresses as telnetdestination (hop-to-hop telnet) • as BB interface addresses and loopback addresses arenot reachable within the vrfs, this should be secure • needs a nice IP addressing scheme (or a clever config generation tool)

41. That’s it! Thank you for your attention Now, it is time for • questions • comments • corrections • flames

42. How to Secure Your Job: Implement MPLS VPNs roger@gottsponer.ch NANOG24 / Miami, Florida February 10-12, 2002