1 / 33

Efficient TCAM Utilization for Router Diversification

Explore TCAM usage and performance impacts in diversified router design for efficient lookup operations. Identify key factors such as key size, memory constraints, and lookup types.

barbaracollins
Download Presentation

Efficient TCAM Utilization for Router Diversification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Design of aDiversified Router: TCAM Usage John DeHartjdd@arl.wustl.edu http://www.arl.wustl.edu/arl

  2. TCAM Issues • Number of Databases to use • 16 available • Key size • Can be different for each Database. • Result Type • Absolute Index • Database Relative Index • Memory Pointer • TCAM Associated Data of width 32, 64 or 128 bits • Memory Constraints: • TCAM Associated Data • 512K x 36 bit ZBT SRAM • Supports 128K 128-bit Results • If used for Rx and Tx then 64K in each direction • IXP QDR SRAM Channel • 2 x 2Mx18 (effective 4M x 18b) • 4 times as much as the TCAM ZBT SRAM. • Supports 512K 128-bit Results • If used for Rx and Tx then 256K in each direction

  3. TCAM Issues • Types of Lookups supported: • Lookup (Direct) • Multi-Hit Lookup (Direct) • Simultaneous Multi-Database Lookup (Direct) • Multi-Database Lookup (Indirect) • Simultaneous Multi-Database Lookup (Indirect) • Re-Issue Multi-Database Lookup (Indirect)

  4. TCAM Issues • Performance Impacts • IXP/TCAM Interface • 16 bits wide • 200 MHz QDR • Effective 32bits per clock tick • Example: 128b results from ZBT SRAM via TCAM: 4 ticks  max of 50M results/sec • Key Size (200MHz, 16 bit Interface for commands) • Rates are max PER LA-1 Interface, up to CAM Core max rate • 32b (1W): 50M Lookups/sec (CAM Core constraint) • 36b (2W): 50M Lookups/sec (CAM Core constraint) • 64b (2W):100M Lookups/sec (Interface constraint) • 72b (3W): 67M Lookups/sec (Interface constraint) • 128b (4W): 50M Lookups/sec (Interface constraint)

  5. TCAM Issues • Performance Impacts (continued) • Result Type • Index or Pointer Types: • Key Size: Max CAM Core lookup rate • 36b: 50M/sec • 72b: 100M/sec • 144b: 100M/sec • 288b: 50M/sec • 576b: 25M/sec • Associated Data: CAM Core rates: • These rates are Total across both LA-1 Interfaces • Key Size: (32b Result Rate, 64b Result Rate, 128b Result Rate) • 36b: 50, 50, 25 • 72b: 100, 50, 25 • 144b: 100, 50, 25

  6. TCAM Performace Tables

  7. TCAM Performace Graphs

  8. TCAM Performace Graphs

  9. LC: Notes on TCAM Lookups • The following slides show a way to use the TCAM for the lookups. • Slight adjustments might be desirable depending on: • Ease of doing operations on non-byte length bit fields • What we learn about methods for using the TCAM. • Field and Identifier sizes: • MN id: 32 bits • MI id: 16 bits (64K Meta Interfaces per Meta Router) • MLI: 16 bits (64K Meta Links per Substrate Link) • Port: 8 bits (256 Physical Interfaces per Line Card) • QID: 17 bits (128K Queues per Queue Manager) • QM ID: 3 bits (8 Queue Managers per LC or PE.) • We probably can only support 4 QMs (2 bits) • (64 Q-Array Entries) / (16 CAM entries)  4 QMs per SRAM Controller.

  10. LC: Notes on TCAM Lookups • Lookup Key size options: • 32/36, 64/72, 128/144, 256/288, 512/576 (all in bits) • Lookup Result options: • Absolute Index: relative to beginning of TCAM array • Database Relative Index: relative to beginning of selected database • Memory Pointer Index: points to SRAM location of result data • Associated Data: 32, 64 or 128 bits of data associated with the lookup result. • Associated Data is stored in ZBT SRAM attached to TCAM. • TCAM Databases • How many to use? • 1 for TX and 1 for RX? • 1 for TX and 1 for each of the SL Types on Rx (5 types)? • Other…

  11. LC: TCAM Lookup Keys on RX • Each SL Type will use a different Database on RX • Thus we do not need an SL Type field in the Key • SL Type ID will be used to identify which database to use • P2P-DC(24b): • SL Type ID: 0x0 • Key Fields: Port(8)/MLI(16) • P2P-Tunnel(IPv4)(72b): • SL Type ID: 0x1 • Key Fields: Port(8)/EtherType(16)/IPSrc(32)/MLI(16) • P2P-VLAN0(72b): • SL Type ID: 0x2 • Key Fields: Port(8)/EthSAddr(48)/MLI(16) • MA(24b): • SL Type ID: 0x3 • Key Fields: Port(8)/MLI(16) • Legacy(24b): • SL Type ID: 0x4 • Key Fields: Port(8)/EtherType(16) • Key Fields: • Port(8b): Physical Interface number • MLI(16b): Meta Link Identifier • Ethertype(16b): Ethernet Header Type field • IPSrc(32b): IP Source Address • EthSAddr(48b): Ethernet Header Source Address

  12. DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) TCI (2B) DstAddr (6B) DstAddr (6B) Type=Substrate (2B) MLI (2B) SrcAddr (6B) SrcAddr (6B) LEN (2B) Type=802.1Q (2B) Meta Frame Type=802.1Q (2B) TCI≠VLAN0 (2B) TCI=VLAN0 (2B) Type=Substrate (2B) Type=Substrate (2B) MLI (2B) MLI (2B) LEN (2B) LEN (2B) PAD (nB) Meta Frame Meta Frame CRC (4B) PAD (nB) PAD (nB) CRC (4B) CRC (4B) P2P-VLAN0 Multi-Access LC: TCAM Lookup Keys on RX P2P-DC Port(8b) MLI(16b) 24 bits IPv4 Tunnel 72 bits Port (8b) EtherType (16b) 0x0800 IP SAddr (32b) MLI (16b) MA Port (8b) Ethernet SAddr (48b) MLI (16b) 72 bits P2P-VLAN0 Port(8b) MLI(16b) 24 bits Legacy 24 bits Port (8b) EtherType (16b) 0x0800 • Blue Shading: Determine SL Type • Black Outline: Key Fields from pkt DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) TCI ≠ VLAN0 (2B) TCI ≠ VLAN0 (2B) Type=IP (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) ID/Flags/FragOff (4B) TTL (1B) TTL (1B) Protocol=Substrate (1B) Protocol (1B) Hdr Cksum (2B) Hdr Cksum (2B) Src Addr (4B) Src Addr (4B) Dst Addr (4B) Dst Addr (4B) MLI (2B) IP Payload LEN (2B) Meta Frame PAD (nB) PAD (nB) CRC (4B) CRC (4B) P2P-DC Configured P2P-Tunnel Legacy

  13. DstAddr (6B) DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) Type=802.1Q (2B) DstAddr (6B) TCI (2B) TCI≠VLAN0 (2B) TCI=VLAN0 (2B) SrcAddr (6B) Type=Substrate (2B) Type=Substrate (2B) Type=Substrate (2B) MLI (2B) MLI (2B) MLI (2B) Type=802.1Q (2B) LEN (2B) LEN (2B) LEN (2B) TCI ≠ VLAN0 (2B) Meta Frame Meta Frame Meta Frame Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol=Substrate (1B) PAD (nB) PAD (nB) PAD (nB) Hdr Cksum (2B) Src Addr (4B) CRC (4B) CRC (4B) CRC (4B) Dst Addr (4B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC Packet RX: All SL Types P2P-VLAN0 Multi-Access P2P-DC P2P-Tunnel

  14. LC: TCAM Lookup Results on RX • Fields (68b/128b): • VLAN (16b) • We could probably drop this to 12b since our switch is supposed to only support 4K VLANs but we might want to leave this open to switches supporting larger numbers of VLANs. • MI (16b) • Blade Eth Hdr (8b) • Only needs to have the DAddr. • We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits are fixed and 8 bits are assigned and stored in the Lookup Result. Even 8 bits is overkill but it leaves us some flexibility. • SAddr can be configured and constant per LC • First EtherType can be constant: 802.1Q • Second EtherType can be constant: Substrate • QID (20b) • MnFlags(8b): • NhAddr(60b): If needed

  15. LC: TCAM Lookup Results on RX • Can we say there will be no Pass Through Meta Links where one side will be on a Multi Access and hence might need a NhAddr field? • If so then we can drop the MnFlags and NhAddr fields from result • Result size then becomes: 60b • Pass Through Meta Link Fields: • MnFlags(8b) • NhAddr(nB) • We have 60 bits left over that could be used for NhAddr • If we need more: • Do a second lookup with the following fields to retrieve the Next Hop bits from another Database for NH Bits? • Port (8b) • VLAN (16b) • MI (16b)

  16. LC: TCAM Lookups on TX • Key: • VLAN(16b) • TxMI(16b) • Result • The Lookup Result for TX will consist of several parts: • Lookup Result • Constant fields • Calculated fields • Fields that can be stored in Local Memory • Some of these are common across all SL Types • Other fields are specific to each SL Type • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-DC Hdr: • P2P-MA Hdr: • P2P-VLAN0 Hdr • P2P-Tunnel Hdr for IPv4 Tunnel

  17. LC: TCAM Lookups on TX • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-DC Hdr (64b) • Constant (16b) • EtherType (16b) = Substrate • Calculated (0b) • From Result (48b) • Eth DA (48b) • Lookup Result Total (Common From Result + Specific From Result): 96 bits • Total (Common + Specific) : 160 bits

  18. LC: TCAM Lookups on TX • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-MA Hdr (64b) : • Constant (16b) • EtherType (16b) = Substrate • Calculated (0b) • From Result (48b) • Eth DA (48b) • Lookup Result Total (Common From Result + Specific From Result): 96 bits • Total (Common + Specific) : 164 bits

  19. LC: TCAM Lookups on TX • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-VLAN0 Hdr (80b): • Constant (16b) • EtherType1 (16b) = 802.1Q • EtherType2 (16b) = Substrate • Calculated (0b) • From Result (64b) • Eth DA (48b) • TCI (16b) • Lookup Result Total (Common From Result + Specific From Result): 112 bits • Total (Common + Specific) : 176 bits

  20. LC: TCAM Lookups on TX • Result (continued) • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-Tunnel Hdr for IPv4 Tunnel (224b): • Constant (64b) • Eth Hdr EtherType (16b) = 0x0800 • IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b): All can be constant? • IP Hdr Flags(3b)/FragOff(13b) (16b) : what is our stance on Fragments? If never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result • IP Hdr TTL (8b): Constant • IP Hdr Proto (8b) = Substrate • Calculated (48b) • IP Hdr Len(16b) : needs to be calculated for each packet sent, so shouldn’t be in Result. • IP Hdr ID(16b): should be unique for each packet sent, so shouldn’t be in Result. • IP Hdr Checksum (16b): Needs to be calculated, so shouldn’t be in Result. • Local Memory (32b) • IP Hdr Src Addr (32b) : tied to port • From Result (80b) • Eth Hdr DA (48b) • IP Hdr Dst Addr (32b) • Lookup Result Total (Common From Result + Specific From Result): 128 bits • Total (Common + Specific) : 320 bits

  21. LC: TCAM Lookups on TX • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • Ignored for Legacy Traffic • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • Legacy (IPv4) with VLAN Hdr (96b): • Constant (16b) • EtherType1 (16b) = 802.1Q • Calculated (0b) • ARP Lookup on NhAddr (48b) • Eth DA (48b) • From Result (32b) • EtherType2 (16b) = IPv4 • TCI (16b) • Lookup Result Total (Common From Result + Specific From Result): 80 bits • Total (Common + Specific) : 192 bits

  22. LC: TCAM Lookups on TX • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (96b): • From Result (48b) • SL Type(4b) • Port(8b) • MLI(16b) • Ignored for Legacy Traffic • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • Legacy (IPv4) without VLAN Hdr (64b): • Constant (0b) • Calculated (0b) • ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b) • Eth DA (48b) • From Result (16b) • EtherType (16b) = IPv4 • Lookup Result Total (Common From Result + Specific From Result): 64 bits • Total (Common + Specific) : 160 bits

  23. SUMMARY: LC: TCAM Lookups • Rx Key: 72 bits • Rx Result Size: 68/128 bits • Depends on whether we need to support MA on other end of pass through MetaLink and on what size the NhAddr needs to be. • If we can limit NhAddr to 60 bits, then it fits in 128 bits. • Otherwise we will have to do something else… • Tx Key Size: 32 bits • Tx Result Size: 128 bits • We need to watch out for the Tx Result for Tunnels. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory.

  24. Databases on LC • Rx: • 0000: DC • 0001: IPv4 Tunnel • 0010: VLAN0 • 0011: MA • 0100: Legacy (non-substrate) with or without VLAN • TX: • 1000: all TX lookups use one database

  25. Extra • The next set of slides are for templates or extra information if needed

  26. Text Slide Template

  27. Image Slide Template

  28. OLD • The rest of these are old slides that should be deleted at some point.

  29. LC: TCAM Lookup Keys on RX • P2P-DC(28b): SL_Type(4)/Port(8)/MLI(16) • P2P-Tunnel(IPv4)(76b): SL_Type(4)/Port(8)/EtherType(16)/IPSrc(32)/MLI(16) • P2P-VLAN0(76b): SL_Type(4)/Port(8)/EthSAddr(48)/MLI(16) • MA(28b): SL_Type(4)/Port(8)/MLI(16) • Legacy(28b): SL_Type(4)/Port(8)/EType(16) • Fields: • SL_Type (4b): Substrate Link Type • 0000: DC • 0001: IPv4 Tunnel • 0010: VLAN0 • 0011: MA • 0100: Legacy (non-substrate) without VLAN • 0101: Legacy (non-substrate) with VLAN • Port(8b): Physical Interface number • MLI(16b): Meta Link Identifier • Ethertype(16b): Ethernet Header Type field • IPSrc(32b): IP Source Address • EthSAddr(48b): Ethernet Header Source Address

  30. Databases on LC • Rx: • 0000: DC • 0001: IPv4 Tunnel • 0010: VLAN0 • 0011: MA • 0100: Legacy (non-substrate) without VLAN • 0101: Legacy (non-substrate) with VLAN • TX: • 1000: all TX lookups use one database • Do we really need to use the TCAM for TX Lookups? • VLAN(16b)/TxMI(16b) is the lookup key. • Really only 4K VLANs support: 12 bits • How many VLANs (i.e. MRs) should be supported per LC? • How many MIs should be supported per MR? • 8MB per SRAM/(16B per Lookup)  512K Entries

  31. LC: TCAM Lookup Results on RX • Standard Fields (64b/128b): • Type (4b): Probably don’t need this since there is a size in MnFlags. • 0000: Default, not Pass Through, ignore MnFlags • 1000: Pass Through with no extra lookup needed for NhAddr(nB), MnFlags(8b) should be 0x00 • 1001: Pass Through with extra lookup needed for NhAddr(nB) • VLAN (16b) • MI (16b) • Blade Eth Hdr (8b) • Only needs to have the DAddr. • We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits are fixed and 8 bits are assigned and stored in the Lookup Result. Even 8 bits is overkill but it leaves us some flexibility. • SAddr can be configured and constant per LC • First EtherType can be constant: 802.1Q • Second EtherType can be constant: Substrate • QID (20b) • MnFlags(8b): If needed • NhAddr(56b): If needed

  32. SUMMARY: LC: TCAM Lookups • Rx Key Size: 128 bits • If We dropped the SL Type field and just made each SL Type a separate Database then we could get the RX Key down to 72 bits • Rx Result Size: 64/128 bits • Depends on whether we need to support MA on other end of pass through MetaLink and on what size the NhAddr needs to be. • Tx Key Size: 32 bits • Tx Result Size: 128 bits • We need to watch out for the Tx Result for Tunnels. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory. • If we find we can’t use the 128 bit Result size for Tx, we might as well include the Local Memory result in the TCAM lookup. • The Local Memory result was going to be the data that was only dependent on the Physical Interface id and was going to go into Local memory to try to keep the TCAM result below 128 bits.

  33. SL(4b) 0000 Port(8b) MLI(16b) PAD(100b) SL (4b) 0001 Port (8b) EtherType (16b) 0x0800 IP SAddr (32b) MLI (16b) PAD (52b) SL (4b) 0010 Port (8b) Ethernet SAddr (48b) MLI (16b) PAD (52b) DstAddr (6B) SL(4b) 0011 Port(8b) MLI(16b) PAD(100b) SrcAddr (6B) Type=802.1Q (2B) TCI (2B) SL (4b) 0100 Port (8b) EtherType (16b) 0x0800 PAD (100b) Type=Substrate (2B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookup Keys on RX • Blue Shading: Determine SL Type • Black Outline: Key Fields from pkt DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) TCI ≠ VLAN0 (2B) TCI ≠ VLAN0 (2B) DstAddr (6B) DstAddr (6B) Type=IP (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) Ver/HLen/Tos/Len (4B) SrcAddr (6B) SrcAddr (6B) ID/Flags/FragOff (4B) ID/Flags/FragOff (4B) TTL (1B) Type=802.1Q (2B) TTL (1B) Type=802.1Q (2B) Protocol=Substrate (1B) TCI≠VLAN0 (2B) Protocol (1B) TCI=VLAN0 (2B) Hdr Cksum (2B) Hdr Cksum (2B) Type=Substrate (2B) Type=Substrate (2B) Src Addr (4B) Src Addr (4B) MLI (2B) MLI (2B) Dst Addr (4B) LEN (2B) Dst Addr (4B) LEN (2B) Meta Frame MLI (2B) Meta Frame IP Payload LEN (2B) Meta Frame PAD (nB) PAD (nB) PAD (nB) PAD (nB) CRC (4B) CRC (4B) CRC (4B) CRC (4B) P2P-DC Configured P2P-VLAN0 Multi-Access P2P-Tunnel Legacy

More Related