Practical use of Ethernet OAM

Overview • A variety of Operations, Administration, and Management (OAM) protocols and tools were developed in recent years for MPLS, IP, and Ethernet networks. • These tools provide unparalleled power for an operator to proactively manage networks and customer Service Level Agreements (SLAs). • This session reviews the various OAM tools available in MPLS/IP/ Ethernet networks at various layers of the stack and recommends/reviews best practices for choosing the right OAM protocol to use in a network. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

OAM Tools Scope of this presentation Management Plane (NMS,EMS) OAM&P Network Plane (Network Elements) Scope of this presentation:OAM tools acrossnetwork elements Scope of this presentation is within network plane only(not management plane) © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

OAM Layers OAM Layering • OAM is layered… • Service Layer OAM • Network Layer OAM • Transport Layer OAM • ... and hierarchical • For example, service layer for Operator A is transport layer for theservice provider • Each layer supports its own OAM mechanisms • Operator A has an MPLS network and uses MPLS OAM tools • Operator B has an Ethernet network and uses Ethernet OAM tools Service Provider MPLS Ethernet CustomerNetwork CustomerNetwork Operator ANetwork Operator BNetwork CustomerLocation 1 CustomerLocation 2 Service OAM MPLS OAM(Operator A) Ethernet OAM(Operator B) Service Layer OAM Network Layer OAM Link OAM Link OAM Link OAM Transport Layer OAM © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

OAM Tools Each layer has its own best-suited OAM tools Brocade Solution • Standards-based, end-to-end OAM • Comprehensive/scalable MPLS, IP, and Ethernet OAM tools Business Problem • Fault detection, verification, and isolation at every level • Proactive detection of service degradation • Performance Monitoring (PM) and SLA verification VRF Ping and Traceroute (Layer 3 VPN) 802.1ag CFM for VPLS/VLLY.1731 PM for VPLS/VLL(Layer 2 VPN) VPN Ping and Traceroute BFD for OSPF and IS-IS IP LSP Ping and Traceroute BFD for RSVP-TE LSPs MPLS Layer 2 Trace Port Loop Detection UDLD Single-link LACPKeep-alive 802.1ag CFM/Y.1731 PM 802.3ahEFM OAM Layer 2 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Layer 2 OAM + Layer 2 VPN CFM/PM: 802.1ag CFM, Y.1731 PM VRF Ping and Traceroute (Layer 3 VPN) 802.1ag CFM for VPLS/VLLY.1731 PM for VPLS/VLL(Layer 2 VPN) VPN Ping and Traceroute BFD for OSPF and IS-IS IP LSP Ping and Traceroute BFD for RSVP-TE LSPs MPLS Layer 2 Trace Port Loop Detection UDLD Single-link LACPKeep-alive 802.1ag CFM/Y.1731 PM 802.3ahEFM OAM Layer 2 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag CFM Connectivity Fault Management (CFM) Service Provider Brocade Implementation • Support for minimum CCM timers (3.3 ms) using hardware offload • 3.3 ms, 10 ms, 100 ms, 1 s, 1 min, 10 min • Support for MIPs and up/down MEPs • Support for all eight MD levels (0-7) • Support for the following types of endpoints/services • VLANs and VPLS/VLL endpoints • Facilitates • Path discovery • Fault detection • Fault verification and isolation • Fault notification • Fault recovery • Supports • Continuity Check Messages (CCMs) • LinkTrace • Loopback messages CustomerNetwork CustomerNetwork Operator ANetwork Operator BNetwork Customerlocation 1 Customerlocation 2 Customer CFM MEP Service Provider CFM MIP Operator A CFM Operator B CFM © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag CFM Terminology Service Provider CustomerNetwork CustomerNetwork Operator ANetwork Operator BNetwork • ME (Maintenance Entity) • A point-to-point relationship between two MEPs within a single MA • MD Level • An integer from 0 to 7 in a field in a CFM PDU that is used, along with the VLAN ID, to identify which MIPs/MEPs would be interested in the contents of a CFM PDU • MD (Maintenance Domain) • The part of a network for which faults in Layer 2 connectivity can be managed • MEP (Maintenance End Point) • A Maintenance Point (MP) at the edge of a domain that actively sources CFM messages • Two types: up (inward*) MEP or down (outward) MEP • MIP (Maintenance Intermediate Point) • A maintenance point internal to a domain that only responds when triggered by certain CFM messages • MA (Maintenance Association) • A set of MEPs established to verify the integrity of a single service instance (a VLAN or a VPLS) Customerlocation 1 Customerlocation 2 Customer MA DownMEP ME MD level 5 (7, 6, or 5) Service Provider MA UPMEP ME MD level 3 (4 or 3) Operator A MA Operator B MA MEP ME ME MD level 1 (2, 1, or 0) MIP (*): “inward” in respect to the device © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag CFM Connectivity Check, LinkTrace, and Loopback Messages • Continuity Check Message (CCM) • A periodic hello message multicast by an MEP within the maintenance domain • LinkTrace Message (LTM) • A multicast message used by a source MEP to trace the path to other MEPs and MIPs in the same domain • All reachable MIPs and MEPs respond back with a Link Trace Unicast Reply (LTR) • The originating MEP can then determine the MAC addresses of all MIPs and MEPs belonging to the same Maintenance Domain • Loopback Message (LBM) • Used to verify the connectivity between a MEP and a peer MEP or MIP • A loopback message is initiated by a MEP with a destination MAC address set to the desired destination MEP or MIP (Unicast) • The receiving MIP or MEP responds to the Loopback message with a Loopback Reply (LBR) (Unicast) • A loopback message helps a MEP identify the precise location of a fault along a given path Periodic CCM (multicast) MEP Periodic CCM MEP LTR (Unicast) LTM (multicast) MEP MEP MIP LTR (Unicast) LBM (Unicast) MEP LBR MEP © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Hierarchical Fault Detection Example: fault in Operator B network (an MPLS Network) • Customer detects fault using Continuity Check and locates fault using Link Trace • Provider A detects fault using Continuity Check and locates fault using Link Trace • Provider B detects fault using Continuity Check, but isolates fault using MPLS OAM (see MPLS OAM section) • A service provider (not shown) would detect this fault in a similar way using Continuity Check and Link Trace from CPEs (Customer Premise Equipment) 1: Customer Continuity Check detects end-to-end fault 2: Customer Link Traces isolate fault past customer MIPs 3: Provider A’s Continuity Check detects end-to-end fault 4: Provider A Link Traces isolate fault inside Provider B’s network MIPs and MEPs at VPLS/VLL endpoints 5: Provider B’s Continuity Check detects service fault MPLS (VPLS/VLL) PE PE P Fault MEP MIP CustomerNetwork (Site 1) Operator A (Location A1) Operator B Operator A(Location A2) CustomerNetwork(Site 2) Fault Localized © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

IEEE 802.1ag Configuration Example To verify end-to-end connectivity between CE1 and CE2 Configure a down MEP on CE1 CE1(config)#cfm-enable CE1(config-cfm)#domain-name CUST_1 level 7 CE1(config-cfm-md-CUST_1)#ma-name ma_5 vlan-id 30 priority 3 CE1(config-cfm-md-CUST_1-ma-ma_5)#ccm-interval 10-second CE1(config-cfm-md-CUST_1-ma-ma_5)#mep 1 down vlan 30 port ethe 1/1 CE1(config-cfm-md-CUST_1-ma-ma_5)#remote-mep 2 to 2 MPLS VLL 7 7 7 7 2/1 1/1 1/1 2/1 CE1 PE1 PE2 CE2 Create a VLL instance (PE1) PE1(config)#router mpls PE1(config-mpls)vll pe1-to-pe2 30 PE1(config-mpls-vll)vll-peer 1.1.1.2 PE1(config-mpls-vll)untagged ethe 1/1 PE1(config-mpls-vll)vlan 30 PE1(config-mpls-vll-vlan)tagged ethe 1/1 Configure CFM on PE1 PE1(config)#cfm-enable PE1(config-cfm)#domain-name CUST_1 level 7 PE1(config-cfm-md-CUST_1)#ma-name ma_5 vll-id 30 priority 3 PE1(config-cfm-md-CUST_1-ma-ma_5)#ccm-interval 10-second In the above configuration, a MIP is created by default on the VLL port. Create a VLL instance (PE2) PE2(config)#router mpls PE2(config-mpls)vll pe2-to-pe1 30 PE2(config-mpls-vll)vpls-peer 1.1.1.1 PE2(config-mpls-vll)untagged ethe 2/1 PE2(config-mpls-vll)vlan 30 PE2(config-mpls-vll-vlan)tagged ethe 2/1 Configure CFM on PE2 PE2(config)#cfm-enable PE2(config-cfm)#domain-name CUST_1 level 7 PE2(config-cfm-md-CUST_1)#ma-name ma_5 vll-id 30 priority 3 PE2(config-cfm-md-CUST_1-ma-ma_5)#ccm-interval 10-second In the above configuration, a MIP is created by default on the VLL-endpoint. Configure a down MEP on CE2 CE2(config)#cfm-enable CE2(config-cfm)#domain-name CUST_1 level 7 CE2(config-cfm-md-CUST_1)#ma-name ma_5 vlan-id 30 priority 3 CE2(config-cfm-md-CUST_1-ma-ma_5)#ccm-interval 10-second CE1(config-cfm-md-CUST_1-ma-ma_5)#mep 2 down vlan 30 port ethe 2/1 CE1(config-cfm-md-CUST_1-ma-ma_5)#remote-mep 1 to 1 LSP ping and LSP traceroute tools would be used inside the MPLS network to detect and diagnose LSP failures © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

ITU-T Y.1731 Performance Management • Standards-based performance management for Ethernet networks • Interoperates in a multivendor environment • Supports high-precision, on-demand measurement of round-trip SLA parameters • Frame Delay (FD) • Frame Delay Variation (FDV) • Measurements done between MEPs Brocade MLX Brocade MLX MEP MEP ETH-DM Frame DelayFrame Delay Variation MEP: Management Enforcement Point ETH-DM: Ethernet Delay Measurement • Benefits • SLA monitoring and verification • Applicability • Aggregation, metro, and core networks • Delay-sensitive applications, such as voice • Differentiated services with SLA guarantees • Brocade differentiation • Hardware-based time-stamping mechanism • Measurements with microsecond granularity • Y.1731 PM for VPLS/VLL © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

ITU-T Y.1731 Performance Management Example Brocade MLX Brocade MLX MEP 2 MEP 3 ETH-DM NetIron# cfm delay_measurement domain md2 ma ma2 src-mep 3 target-mep 2 Y1731: Sending 10 delay_measurement to 0012.f2f7.3931, timeout 1000 msec Type Control-c to abort Reply from 0012.f2f7.3931: time= 32.131 us Reply from 0012.f2f7.3931: time= 31.637 us Reply from 0012.f2f7.3931: time= 32.566 us Reply from 0012.f2f7.3931: time= 34.052 us Reply from 0012.f2f7.3931: time= 33.376 us Reply from 0012.f2f7.3931: time= 31.501 us Reply from 0012.f2f7.3931: time= 33.016 us Reply from 0012.f2f7.3931: time= 32.537 us Reply from 0012.f2f7.3931: time= 32.492 us Reply from 0012.f2f7.3931: time= 32.552 us sent = 10 number = 10 A total of 10 delay measurement replies received. Success rate is 100 percent (10/10) ==================================================================== Round Trip Frame Delay Time : min = 31.501 us avg = 32.586 us max = 34.052 us Round Trip Frame Delay Variation : min = 45 ns avg = 839 ns max = 1.875 us ==================================================================== © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Link OAM IEEE 802.3ah Ethernet First Mile (EFM) OAM • Supports point-to-point (single) link OAM • Monitors and supports troubleshooting individual links • Standards-based for Ethernet networks • Interoperates in a multivendor environment • Supports • Fault detection and notification (alarms) • Discovery • Remote failure indication • Loopback testing 802.3ah OAM 802.3ah OAM NetIron#show link-oam info detail ethernet 1/1 OAM information for Ethernet port: 1/1 link-oam mode: active link status: up oam status: up Local information multiplexer action: forward parse action: forward stable: satisfied state: up loopback state: disabled dying-gasp: false critical-event: false link-fault: false Remote information multiplexer action: forward parse action: forward stable: satisfied loopback support: disabled dying-gasp: false critical-event: false link-fault: false © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Layer 2 OAM Summary Remember: OAM is layered and hierarchical(service OAM for an operator is transport OAM for a service provider) © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

MPLS OAM VRF Ping and Traceroute (Layer 3 VPN) 802.1ag CFM for VPLS/VLLY.1731 PM for VPLS/VLL(Layer 2 VPN) VPN   Ping and Traceroute BFD for OSPF and IS-IS IP   LSP Ping and Traceroute BFD for RSVP-TE LSPs MPLS Layer 2 Trace Port Loop Detection UDLD Single-link LACPKeep-alive 802.1ag CFM/Y.1731 PM 802.3ahEFM OAM Layer 2       © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

LSP Ping and LSP Traceroute MPLS OAM tools • LSP Ping and LSP Traceroute provide OAM functionality for MPLS networks based on RFC 4379. • LSP Ping and LSP Traceroute tools provide a mechanism to detect MPLS data plane failure. • MPLS echo requests follow the same data path that normal MPLS packets would traverse. • LSP Ping is used to detect data plane failure and to check the consistency between the data plane and the control plane. • LSP Traceroute is used to isolate the data plane failure to a particular router and to provide LSP path tracing. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

LSP Ping MPLS Network PE PE P LSP (LER) (LER) (LSR) • The basic idea is to verify that packets that belong to a particular Forwarding Equivalence Class (FEC) actually end their MPLS path on a Label Switching Router (LSR) that is an egress for that FEC. • LDP LSP Ping and RSVP LSP Ping are supported. Echo Request Echo Reply LSP Ping LDP LSP Ping NetIron# ping mpls ldp 22.22.22.22 Send 5 80-byte MPLS Echo Requests for LDP FEC 22.22.22.22/32, timeout 5000 msec Type Control-c to abort !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max=0/1/1 ms. Syntax: ping mpls ldp <ip-address | ip-address/mask-length> ... options RSVP LSP Ping NetIron# ping mpls rsvp lsp toxmr2frr-18 Send 5 92-byte MPLS Echo Requests over RSVP LSP toxmr2frr-18, timeout 5000 msec Type Control-c to abort !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max=0/1/5 ms. Syntax: ping mpls rsvp lsp <lsp-name> | session <tunnel-source-address> <tunnel-destination-address> <tunnel-id>... options © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

LSP Traceroute MPLS Network PE PE P LSP (LER) (LER) (LSR) • With LSP traceroute, an echo request packet is sent to the control plane of each transit LSR, which confirms that it is a transit LSR for this path. • Transit LSRs return echo replies. • LDP LSP Ping and RSVP LSP Ping are supported. Echo Request Echo Replies LSP Traceroute LDP LSP Traceroute NetIron# traceroute mpls ldp 22.22.22.22 Trace LDP LSP to 22.22.22.22/32, timeout 5000 msec, TTL 1 to 30 Type Control-c to abort 1 10ms 22.22.22.22 return code 3(Egress) Syntax: traceroute mpls ldp < ip-address | ip-address/mask-length> ... options RSVP LSP Traceroute NetIron # traceroute mpls rsvp lsp toxmr2frr-18 Trace RSVP LSP toxmr2frr-18, timeout 5000 msec, TTL 1 to 30 Type Control-c to abort 1 1ms 22.22.22.22 return code 3(Egress) Syntax: traceroute mpls rsvp lsp <lsp-name> | session <tunnel-source-address> <tunneldestination-address> <tunnel-id>... options © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

Practical use of Ethernet OAM