1 / 92

Design of a Diversified Router: Lookup Block

This presentation explores the architecture and hardware details of the Lookup Block in a diversified router, including database organization, key/entry sizes, and lookup commands.

mdrew
Download Presentation

Design of a Diversified Router: Lookup Block

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Design of aDiversified Router: Lookup Block John DeHartjdd@arl.wustl.edu http://www.arl.wustl.edu/arl

  2. Overview • These slides are as much a definition of what is NOT in the Lookup Block as they are what is. • In defining what is not in the Lookup Block I am putting some requirements on other blocks. • These requirements have to do with where fields are added to frame headers. • Not everything can or needs to be kept in the TCAM. • There are also: • Constants • Fields that have to be calculated for each frame • Fields that are configurable per Blade or per physical interface. • Etc. • Also, there is a lot of information about the TCAM here.

  3. Architecture Review • First lets review the architecture of the Promentum™ ATCA-7010 card which will be used to implement our LC and NP Blades: • Two Intel IXP2850 NPs • 1.4 GHz Core • 700 MHz Xscale • Each NPU has: • 3x256MB RDRAM, 533 MHz • 4 QDR II SRAM Channels • Channels 1, 2 and 3 populated with 8MB each running at 200 MHz • 16KB of Scratch Memory • 16 Microengines • Instruction Store: 8K 40-bit wide instructions • Local Memory: 640 32-bit words • TCAM: Network Search Engine (NSE) on SRAM channel 0 • Each NPU has a separate LA-1 Interface • Part Number: IDT75K72234 • 18Mb TCAM

  4. NP Blades

  5. TCAM HW Details • CAM Size: • Data: 256K 72-bit entries • Organized into Segments. • Mask: 256K 72-bit entries • Segments: • Each Segment is 8k 72-bit entries • 32 Segments • Segments are not shared between Databases. • Minimum database size is therefore 8K 72-bit entries. • Do databases wider than 72-bits use multiple parallel segments or do they use sequential entries in a segment to make up the long entries? • Sequential entries are used to comprise one longer entry. • 36b DB has 16K entries per segment • 72b DB has 8K entries per segment • 144b DB has 4K entries per segment • 288b DB has 2K entries per segment • 576b DB has 1K entries per segment • Can a segment be dynamically added to a Database as it grows? • Yes, more on this feature in a future issue of the IDT User Manual…

  6. TCAM HW Details • Number of Databases available: 16 • Database Core Sizes: 36b, 72b, 144b, 288b, 576b • Coresize indicates how many CAM core entries are used per DB entry • Key/Entry size • Can be different for each Database. • Key/Entry size <= Database Core Size • Key/Entry size indicates how many memory access cycles it will take to get the Key into the TCAM across the 16-bit wide QDR II SRAM interface.

  7. TCAM HW Details • Result Type • Absolute Index: relative to beginning of CAM • Database Relative Index: relative to beginning of Database • Memory Pointer: Translation based on database configuration registers • Base address • Result size • TCAM Associated Data of width 32, 64 or 128 bits • Memory Usage: • Results can be stored in TCAM Associated Data SRAM or IXP SRAM. • TCAM Associated Data • 512K x 36 bit ZBT SRAM (4 bits of parity) • Supports 256K 64-bit Results • If used for Ingress and Egress then 128K in each direction • Supports 128K 128-bit Results • If used for Ingress and Egress then 64K in each direction • Results deposited directly in Results Mailbox • IXP QDR II SRAM Channel • 2 x 2Mx18 (effective 4M x 18b) • 4 times as much as the TCAM ZBT SRAM. • Supports 1024K 64-bit Results • If used for Ingress and Egress then 512K in each direction • Supports 512K 128-bit Results • If used for Ingress and Egress then 256K in each direction • Read Results Mailbox to check Hit bit and to get Index or Memory Pointer • Then read SRAM for actual Result.

  8. TCAM HW Details • Lookup commands supported: • Direct: Command is encoded in 2b Instruction field on Address bus • Indirect: Instruction field = 11b, Command encoded on Data bus. • Lookup (Direct) • 1 DB, 1 Result • Multi-Hit Lookup (Direct) • 1 DB, <= 8 Results • Simultaneous Multi-Database Lookup (Direct) • 2 DB, 1 Result Each • DBs must be consecutive! • Multi-Database Lookup (Indirect) • <= 8 DB, 1 Result Each • Simultaneous Multi-Database Lookup (Indirect) • 2 DB, 1 Result Each • Functionally same as Direct version but key presentation and DB selection are different. • DBs need not be consecutive. • Re-Issue Multi-Database Lookup (Indirect) • <= 8 DB, 1 Result Each • Search Key can be modified for each DB being searched. • First 32 bits of search key can be specified for each • Rest of key is same for each.

  9. TCAM HW Details • Mask Registers Notes (mostly for reference) • When are these used? • I think we will need one of these for each database that is to be used in a Multi Database Lookup (MDL), where the database entries do not actually use all the bits in the corresponding core size. • For example: a 32-bit lookup would have a core size of 36 bits and so would need a GMR configured as 0xFFFFFFFF00 to mask off the low order 4 bits when it is used in a MDL where there are larger databases also being searched. • 64 72-bit Global Mask Registers (GMR) • Can be combined for different database sizes • 36-bit databases have access to 31 out of a total of 64 GMRs • A bit in the configuration for a database selects which half of the GMRs can be used • A field in each lookup command selects which specific GMR is to be used with the lookup key. • Value of 0x1F (31) is used in command to indicate no GMR is to be used. Hence, 36-bit lookups cannot use all 32 GMRs in its half. • 72-bit databases have access to 31 out of a total of 64 GMRs • A bit in the configuration for a database selects which half of the GMRs can be used • A field in each lookup command selects which specific GMR is to be used with the lookup key. • Value of 0x1F (31) is used in command to indicate no GMR is to be used. Hence, 72-bit lookups cannot use all 32 GMRs in its half. • 144-bit lookups have 32 GMRs available to it. • 288-bit lookups have 16 GMRs available to it. • 576-bit lookups have 8 GMRs available to it. • Each lookup command can have one GMR associated with it.

  10. TCAM Usage Notes • Database Types are defined and managed by the IMS Software. • The Type of the Database is defined in the software only. • It tells the software how to define and use masks and priorities (weights). • Allows the software to provide to the user a more flexible way to specify entries. • Types of Databases: • Longest Prefix Match (LPM): • Mask matches length of prefix • Exact Match (EM) • Mask matches full Entry size • Best/Range Match: What we typically call General Match. • Mask is completely general. • Priority: • Priority within a database is done by order of the entries. • Exact Match should not need priority within the database since only one Entry should match a supplied Key. • LPM and Best/Range Match do use priority within the databases. • So, the order in which the entries are stored in these databases is important. • For LPM DBs we would want to group prefixes by length in the TCAM. • And this is almost certainly what the IMS software does. • Changing priorities on existing entries may cause us some problems. • It appears that the only way to change the priority of a Best/Range Match entry might be to write a new entry in a different location (different priority) and then delete the old entry. • Changing the priority of an LPM entry really would mean changing its prefix. • IMS uses a weight assigned to Entries as they are added for LPM and Best/Range Match • I believe this weight is just used to group entries of the same weight together and to ensure that entries are ordered based on their weights as they are added.

  11. TCAM Performance • Three Factors that affect performance: • Lookup Size (Key) • Associated Data Width (Result) • CAM Core Lookup Rate • IXP/TCAM LA-1 Interface • 16 bits wide • 200 MHz QDR II SRAM Interface • Effectively 32bits per clock tick • So getting Key in is 32bits/tick • Example: 128b Key would take 4 ticks to get clocked into TCAM. • Max of 50 M Lookups/sec • Table on next slide shows some of the performance numbers for some Sizes that are of interest to us. • What we’ll see a little later is that in the worst case, we need a TOTAL Lookup rate of 12.5 M/sec (6.25 M/sec on each LA-1 interface)

  12. TCAM Performance (Rates in M/sec)

  13. TCAM Software • Several software components exist, enough to be really confusing. • IDT Libraries: • MicroEngine Libraries: • NSE-QDR Data Plane Macro (DPM) API • Iipc.uc and Iipc.h • IIPC: Integrated IPCo-processor • Microengine Lookup Library (MLL) • IipcMll.uc • XScale: • Lookup Management Library (LML) • Control Plane: • Initialization Management and Search (IMS) Library • Simulation: • NSE with Dual QDR Interfaces IDT75K234SLAM • Intel Libraries: • TCAM Classifier Library • Microengine and XScale support for using TCAM. • Requires installation of MLL and LML. • Is geared toward a very specific application of NSE to IPv4 Forwarding App. • May be useful as an example of code to look at but probably not useful for us to use directly. • IXA SDK 4.0 Location: • src/library/microblocks_library/microcode/idt_tcam_classifier

  14. Lookup Block • Three Lookup Blocks Needed: • All the Lookup Blocks will use the TCAM • LC-Ingress • All Databases for Ingress will be Exact Match • LC-Egress • All Databases for Egress will be Exact Match • MR • There will probably be multiple versions of this: • Shared • Dedicated • IPv4 • MPLS • But lets think of it as one for now and focus on IPv4. • Discussion later on what combination of the three types of DB we might use. • Base functionality should be the same for all three • A lot of code should be common • LC-Ingress and LC-Egress will share a TCAM • There will be two MR Lookup engines sharing a TCAM • ARP on the LC might need/want to use the TCAM. • The aging properties of the TCAM might be very useful for ARP. • So, we should leave some room for ARP on the LC TCAM. • We will need to think more about ARP when we get into the details of the control plane.

  15. QM QM MR Lookup Block Control TCAM XScale XScale Rx DeMux Parse Parse DeMux Rx Lookup Lookup Tx Tx HeaderFormat HeaderFormat MR (NPUA) MR (NPUB)

  16. LC Lookup Block Ingress (NPUB) R T M S W I T C H Lookup Hdr Format QM/Schd Switch Tx Phy Int Rx Key Extract XScale LC TCAM ARP XScale Phy Int Tx QM/Schd Hdr Format Lookup Rate Monitor Key Extract Switch Rx Egress (NPUA)

  17. Lookup Block Requirements • General: • Number of Packets per second required to handle? • Line Rate: 10Gb/s • Assume an average IP Packet Size of 200 Bytes (1600 bits) • (10Gb/s)/(1600 bits/pkt) = 6.25 Mpkt/s • Ethernet Header of 14 Bytes • Average Frame Size of 214 Bytes (1712 bits) • (10Gb/s)/(1712 bits/pkt) = 5.841 Mpkt/s • Ethernet Inter-Frame Spacing: 96 bits • Average Frame Size with Inter-Frame Spacing: 1808 bits • (10Gb/s)/(1808 bits/pkt) = 5.53 Mpkt/s • LC: Number of Lookups per second required: • 1 Ingress and 1 Egress lookup required per packet • If we assume 6.25 MPkts/sec then we need 12.5 M Lookups/sec. • MR/NPU: Number of Lookups per second required: • 5 Gb/s per MR/NPU: 3.125 M Lookups/sec • Total of 6.25 M Lookups/sec • Total Number of Lookup Entries to be supported? • Dependent on Size of Entries • Size of Entries and Keys? • Dependent on type of Lookup: MR, LC-Ingress, LC-Egress • Size of Results? • Dependent on type of Lookup: MR, LC-Ingress, LC-Egress

  18. Keys and Results for Ingress LC and Egress LC • Keys: • Ingress (Link  Router): • What fields in the External Frame Formats uniquely identify the MetaLink? • First we have to identify the Substrate Link Type • Then we can Identify the Substrate Link and MetaLink • Egress (Router  Link): • What fields in the Internal Frame Format uniquely identify the MetaLink? • Results: • We need to identify what fields are needed to build the appropriate frame headers. • The fields needed may consist of several parts: • Constant fields: constants in the LC Rx/Tx Code • Calculated fields: • Things like TTL and Checksums • Statically configured Fields that can be stored in Local Memory • Things like per physical interface or Blade Ethernet Src Addresses • ARP results for Ethernet DAddr on a Multi-Access link • Lookup Result from TCAM • Everything else… • Ingress (Link  Router): Details later • Egress (Router Link): Details later

  19. Field Sizes in Keys and Results • Field and Identifier sizes: • MR id: 16 bits (64K Meta Routers per Substrate Router) • MR ID == VLAN (Defined locally on a Substrate Router) • Note: We can probably shorten this to 12 bits since our switch only supports 4K VLANs which is 12 bits. • MI id: 16 bits (64K Meta Interfaces per Meta Router) • This seems like a lot. What level of flexibility do we need to support? • MLI: 16 bits (64K Meta Links per Substrate Link) • This seems safe and should not changed. • Port: 4 bits (16 Physical Interfaces per Line Card) • Note: I originally had this defined as 8 bits but since the RTM only supports 10 physical interfaces, 4 bits is enough. There were some places where the extra 4 bits pushed us to a larger size. • QID: 20 bits (QM_ID:Queue_ID) • Queue_ID: 17 bits (128K Queues per Queue Manager) • QM_ID: 3 bits (8 Queue Managers per LC or PE.) • We probably can only support 4 QMs, which could be encoded in 2 bits. • (64 Q-Array Entries) / (16 CAM entries)  4 QMs per SRAM Controller.

  20. DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) VLAN (2B) VLAN (2B) Type=Substrate (2B) Type=Substrate (2B) TxMI (2B) RxMI (2B) MnFlags (1B) MnFlags (1B) NhAddr (nB) NhAddr (nB) LEN (2B) LEN (2B) Meta Frame Meta Frame PAD (nB) PAD (nB) CRC (4B) CRC (4B) LC … LC: Internal Frame Formats Internal Frame Leaving Ingress LC Internal Frame Arriving at Egress LC Packet arriving On Port N Packet leaving On Port M LC MR Switch Switch … IXP PE

  21. DstAddr (6B) DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) Type=802.1Q (2B) TCI (2B) TCI ≠ VLAN0 (2B) TCI ≠ VLAN0 (2B) DstAddr (6B) DstAddr (6B) Type=Substrate (2B) Type=IP (2B) Type=IP (2B) MLI (2B) Ver/HLen/Tos/Len (4B) Ver/HLen/Tos/Len (4B) SrcAddr (6B) SrcAddr (6B) LEN (2B) ID/Flags/FragOff (4B) ID/Flags/FragOff (4B) Meta Frame TTL (1B) Type=802.1Q (2B) Type=802.1Q (2B) TTL (1B) Protocol=Substrate (1B) Protocol (1B) TCI=VLAN0 (2B) TCI≠VLAN0 (2B) Hdr Cksum (2B) Hdr Cksum (2B) Type=Substrate (2B) Type=Substrate (2B) Src Addr (4B) Src Addr (4B) MLI (2B) MLI (2B) Dst Addr (4B) LEN (2B) LEN (2B) Dst Addr (4B) PAD (nB) Meta Frame Meta Frame MLI (2B) IP Payload CRC (4B) LEN (2B) P2P-VLAN0 Meta Frame P2P-Tunnel PAD (nB) PAD (nB) PAD (nB) PAD (nB) CRC (4B) CRC (4B) CRC (4B) CRC (4B) LC: External Frame Formats P2P-DC (Configured) Multi-Access Legacy

  22. DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) VLAN (2B) VLAN (2B) Type=Substrate (2B) Type=Substrate (2B) RxMI (2B) TxMI (2B) MnFlags (1B) MnFlags (1B) NhAddr (nB) NhAddr (nB) LEN (2B) LEN (2B) Meta Frame Meta Frame PAD (nB) PAD (nB) DstAddr (6B) DstAddr (6B) DstAddr (6B) CRC (4B) CRC (4B) SrcAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) Type=802.1Q (2B) TCI ≠ VLAN0 (2B) TCI (2B) TCI ≠ VLAN0 (2B) DstAddr (6B) Type=IP (2B) Type=Substrate (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) MLI (2B) Ver/HLen/Tos/Len (4B) SrcAddr (6B) ID/Flags/FragOff (4B) LEN (2B) ID/Flags/FragOff (4B) DstAddr (6B) TTL (1B) Meta Frame TTL (1B) Type=802.1Q (2B) Protocol=Substrate (1B) Protocol (1B) SrcAddr (6B) TCI=VLAN0 (2B) Hdr Cksum (2B) Hdr Cksum (2B) Type=Substrate (2B) Src Addr (4B) Type=802.1Q (2B) Src Addr (4B) MLI (2B) Dst Addr (4B) TCI≠VLAN0 (2B) Dst Addr (4B) LEN (2B) PAD (nB) Type=Substrate (2B) MLI (2B) Meta Frame IP Payload MLI (2B) LEN (2B) CRC (4B) LEN (2B) Meta Frame Meta Frame PAD (nB) PAD (nB) PAD (nB) CRC (4B) CRC (4B) CRC (4B) PAD (nB) CRC (4B) LC: TCAM Lookup Keys Internal Frame Leaving Ingress LC Internal Frame Arriving at Egress LC Ingress LC Egress LC • Blue Shading: Determine SL Type • Black Outline: Key Fields from pkt P2P-DC (Configured) P2P-Tunnel Legacy P2P-VLAN0 Multi-Access

  23. SL(4b) 0000 Port (4b) MLI(16b) SL(4b) 0001 Port (4b) EtherType (16b) 0x0800 IP SAddr (32b) MLI (16b) SL(4b) 0100 Port (4b) Ethernet SAddr (48b) MLI (16b) SL(4b) 0011 Port (4b) MLI(16b) SL(4b) 0010 Port (4b) EtherType (16b) 0x0800 DstAddr (6B) DstAddr (6B) SrcAddr (6B) SrcAddr (6B) Type=802.1Q (2B) Type=802.1Q (2B) TCI (2B) TCI ≠ VLAN0 (2B) DstAddr (6B) DstAddr (6B) Type=Substrate (2B) Type=IP (2B) MLI (2B) Ver/HLen/Tos/Len (4B) SrcAddr (6B) SrcAddr (6B) LEN (2B) ID/Flags/FragOff (4B) Type=802.1Q (2B) Meta Frame TTL (1B) Type=802.1Q (2B) TCI≠VLAN0 (2B) Protocol (1B) TCI=VLAN0 (2B) Hdr Cksum (2B) Type=Substrate (2B) Type=Substrate (2B) Src Addr (4B) MLI (2B) MLI (2B) LEN (2B) Dst Addr (4B) LEN (2B) PAD (nB) Meta Frame Meta Frame IP Payload CRC (4B) PAD (nB) PAD (nB) PAD (nB) CRC (4B) CRC (4B) CRC (4B) P2P-VLAN0 Multi-Access Legacy LC: TCAM Lookup Keys on Ingress P2P-DC 24 bits IPv4 Tunnel 72 bits Legacy 24 bits P2P-VLAN0 24 bits MA 72 bits DstAddr (6B) • Blue Shading: Determine SL Type • Black Outline: Key Fields from pkt SrcAddr (6B) Type=802.1Q (2B) TCI ≠ VLAN0 (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol=Substrate (1B) Hdr Cksum (2B) Src Addr (4B) Dst Addr (4B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) P2P-DC (Configured) P2P-Tunnel

  24. LC: TCAM Lookup Results on Ingress • We need the Ethernet Header fields to get the frame to the blade that is to process it next. • We also need a QID and RxMI • Ethernet header fields that are constants can be configured and do not need to be in the TCAM Lookup Result. • Ethernet Header fields: • DAddr: Depends on MetaLink • SAddr: Can be constant and configured per LC • EtherType1: Can be a constant: 802.1Q • VLAN(TCI): Different for each MR • EtherType2: Can be a constant: Substrate • TCAM Lookup Result (60b) • VLAN (16b) • RxMI (16b) • DAddr (8b) • We can control the MAC Addresses of the Blades, so lets say that 40 of the 48 bits of DAddr are constant across all blades and 8 bits are assigned and stored in the Lookup Result. • Will 8 bits be enough to support multiple chasses? • We could go up to 12 bits and still use 64bit Associated Data • QID (20b) • What about Ingress  Egress Pass Thru MetaLinks? • We will define a special Substrate VLAN for this use • We will also define a special set of MIs

  25. Pass Thru MetaLinks and Multi-Access SLs • When going MR  LC-Egress the MR may provide a Next Hop MN Address for the LC to use to map to a MAC address. • This is particularly used when the destination Substrate Link is Multi-Access and there may be multiple MAC addresses used on the same Multi-Access MetaLink. • When going LC-Ingress  LC-Egress for a pass through MetaLink, do we need to do something similar? • This could arise when a MetaNet has hosts on a multi-access network but the first Substrate Router that these hosts have access to does not have a MR for that MN. • However, I contend that if there is no MR on that access SR, then there is nothing there to discriminate between the multiple MN addresses on the single MA MetaLink and hence it cannot be supported.

  26. Pass Thru MetaLinks and Multi-Access SLs Host1 No way to communicate Next Hop addresses from MR to distant LC Host2 Host3 Host4 LC LC LC ARP ML MR MA Network Host5 P2P SL MA SL Host6 Substrate Router1 Substrate Router2 • Implications: • We will not extend MA links across Substrate Routers and other Substrate Links. • MetaNets must place a MR in the substrate router that terminates a MA Substrate Link on which they want to support hosts. Host7 Host8 … HostN

  27. LC: TCAM Lookups on Egress DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) VLAN (2B) Type=Substrate (2B) TxMI (2B) MnFlags (1B) NhAddr (nB) LEN (2B) Meta Frame PAD (nB) CRC (4B) • Key: • VLAN(16b) • TxMI(16b) • Result • The Lookup Result for Egress will consist of several parts: • Lookup Result • Constant fields • Calculated fields • Fields that can be stored in Local Memory • Some of these are common across all SL Types • Other fields are specific to each SL Type • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-DC Hdr: • P2P-MA Hdr: • P2P-VLAN0 Hdr • P2P-Tunnel Hdr for IPv4 Tunnel

  28. DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) TCI (2B) Type=Substrate (2B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-DC Hdr (64b) • Constant (16b) • EtherType (16b) = Substrate • Calculated (0b) • From Result (48b) • Eth DA (48b) • Lookup Result Total (Common Result + Specific Result): 92 bits • Total (Common + Specific) : 156 bits

  29. DstAddr (6B) SrcAddr (6B) Type=Substrate (2B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (92b): • From Result (48b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • MA Hdr (64b) : • Constant (16b) • EtherType (16b) = Substrate • Calculated (0b) • ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b) • Eth DA (48b) • From Result (0b) • Lookup Result Total (Common From Result + Specific From Result): 44 bits • Total (Common + Specific) : 156 bits

  30. DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) TCI≠VLAN0 (2B) Type=Substrate (2B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • MA with VLAN Hdr (96b) : • Constant (32b) • EtherType1 (16b) = 802.1Q • EtherType2 (16b) = Substrate • Calculated (0b) • ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b) • Eth DA (48b) • From Result (16b) • TCI (16b) • Lookup Result Total (Common From Result + Specific From Result): 60 bits • Total (Common + Specific) : 188 bits

  31. DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) TCI=VLAN0 (2B) Type=Substrate (2B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-VLAN0 Hdr (80b): • Constant (16b) • EtherType1 (16b) = 802.1Q • EtherType2 (16b) = Substrate • Calculated (0b) • From Result (64b) • Eth DA (48b) • TCI (16b) • Lookup Result Total (Common From Result + Specific From Result): 108 bits • Total (Common + Specific) : 172 bits

  32. DstAddr (6B) SrcAddr (6B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol=Substrate (1B) Hdr Cksum (2B) Src Addr (4B) Dst Addr (4B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Result (continued) • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-Tunnel Hdr for IPv4 Tunnel without VLANs (224b): • Constant (64b) • Eth Hdr EtherType (16b) = 0x0800 • IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b): All can be constant? • IP Hdr TTL (8b): Initialized to a contant when sending. • IP Hdr Proto (8b) = Substrate • Calculated (48b) • IP Pkt Len(16b) : Calculated for each packet. • IP Hdr ID(16b): should be unique for each packet sent, so shouldn’t be in Result. • IP Hdr Checksum (16b): Needs to be calculated, so shouldn’t be in Result. • IP Hdr Flags(3b)/FragOff(13b) (16b) : If fragments are never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result • Local Memory (32b) • IP Hdr Src Addr (32b) : tied to physical interface (10 entry table in Egress LC Tx) • From Result (80b) • Eth Hdr DA (48b) • IP Hdr Dst Addr (32b) • Lookup Result Total (Common From Result + Specific From Result): 124 bits • Total (Common + Specific) : 316 bits

  33. DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) TCI ≠ VLAN0 (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol=Substrate (1B) Hdr Cksum (2B) Src Addr (4B) Dst Addr (4B) MLI (2B) LEN (2B) Meta Frame PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Result (continued) • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • P2P-Tunnel Hdr for IPv4 Tunnel with VLANs (240b): • Constant (64b) • First Eth Hdr EtherType (16b) = 802.1QS • Second Eth Hdr EtherType (16b) = 0x0800 • IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b): All can be constant? • IP Hdr TTL (8b): Initialized to a contant when sending. • IP Hdr Proto (8b) = Substrate • Calculated (48b) • IP Pkt Len(16b) : Calculated for each packet. • IP Hdr ID(16b): should be unique for each packet sent, so shouldn’t be in Result. • IP Hdr Checksum (16b): Needs to be calculated, so shouldn’t be in Result. • IP Hdr Flags(3b)/FragOff(13b) (16b) : If fragments are never used, these are constants, if it is possible we will have to use them, then this has to be calculated. Either way, shouldn’t be in Result • Local Memory (32b) • IP Hdr Src Addr (32b) : tied to physical interface (10 entry table in Egress LC Tx) • From Result (96b) • Eth Hdr DA (48b) • IP Hdr Dst Addr (32b) • VLAN (16b) • Lookup Result Total (Common From Result + Specific From Result): 140 bits (This is a PROBLEM!) • Total (Common + Specific) : 332 bits

  34. DstAddr (6B) SrcAddr (6B) Type=802.1Q (2B) TCI ≠ VLAN0 (2B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol (1B) Hdr Cksum (2B) Src Addr (4B) Dst Addr (4B) IP Payload PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • Ignored for Legacy Traffic • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • Legacy (IPv4) with VLAN Hdr (96b): • IP Header provided by MR! • Constant (16b) • EtherType1 (16b) = 802.1Q • Calculated (0b) • ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b) • Eth DA (48b) • From Result (32b) • EtherType2 (16b) = IPv4 • TCI (16b) • Lookup Result Total (Common From Result + Specific From Result): 76 bits • Total (Common + Specific) : 188 bits

  35. DstAddr (6B) SrcAddr (6B) Type=IP (2B) Ver/HLen/Tos/Len (4B) ID/Flags/FragOff (4B) TTL (1B) Protocol (1B) Hdr Cksum (2B) Src Addr (4B) Dst Addr (4B) IP Payload PAD (nB) CRC (4B) LC: TCAM Lookups on Egress • Key: • VLAN(16b) • TxMI(16b) • Result • Common across all SL Types (92b): • From Result (44b) • SL Type(4b) • Port(4b) • MLI(16b) • Ignored for Legacy Traffic • QID (20b) • Local Memory (48b) • Eth Hdr SA (48b) : tied to Port • SL Type Specific Headers • Legacy (IPv4) without VLAN Hdr (64b): • IP Header provided by MR! • Constant (0b) • Calculated (0b) • ARP Lookup on NhAddr (Is ARP cache another database in TCAM?) (48b) • Eth DA (48b) • From Result (16b) • EtherType (16b) = IPv4 • Lookup Result Total (Common From Result + Specific From Result): 60 bits • Total (Common + Specific) : 156 bits

  36. LC: Lookup Block Parameters • All lookups will be Exact Match. • Ingress: • # Databases: 1 • 4 bits in Key identify the SL Type • 0000: DC • 0001: IPv4 Tunnel • 0010: Legacy (non-substrate) with or without VLAN • 0011: VLAN0 • 0100: MA (with or without VLAN) • Core Size: 72b • Key Size: 24b - 72b • AD Result Size: 64b of which we’ll use 60 bits • Egress: • # Databases: 1 • Core Size: 36b • Key Size: 32b • AD Result Size: 128b of which we’ll use different amounts per SL Type • With one problem to still work out.

  37. SUMMARY: LC: TCAM Lookups • Ingress Key Size: 24 bits or 72 bits • Ingress Result Size: 60 bits • Egress Key Size: 32 bits • Egress Result Size: 128 bits • The IP Tunnel with VLANs Substrate Link option is a problem. • Discussion of ways to handle them are on next slide • We also need to watch out for the Egress Result for Tunnels w/o VLANs. If we introduce anything else we want in there then we go beyond the 128 bits supportable through the TCAM’s Associated memory.

  38. Handling IP Tunnel SL with VLANs • Result Fields (144 bits): • SL Type(4b) • Port(4b) • MLI(16b) • QID (20b) • Eth Hdr DA (48b) • IP Hdr Dst Addr (32b) • VLAN (16b) • 128 bits is maximum size of a Result stored in TCAM Associated Data SRAM • Options for handling this Result • Not allow this type of SL • Might be ok for short term but almost certainly not ok for long term. • Find 16 bits we don’t really need in Result • Do a second lookup when we find a SL like this. • Do a Multi-Hit lookup and put two entries in for these SLs and only one entry for all others. • Then concatenate the two results when we get them. • Only allow a small fixed number of this type of SL: • Store an index in the 4 bits we have left • store the extra bits we need in a table in Local memory. • However, this is a little tricky since we would then need to get the extra bits from the control plane into Local Memory and we will want Substrate Links to be able to be added dynamically.

  39. QM QM MR Lookup Block Control TCAM XScale XScale Rx DeMux Parse Parse DeMux Rx Lookup Lookup Tx Tx HeaderFormat HeaderFormat MR (NPUA) MR (NPUB)

  40. Buf Handle(32b) Buffer Handle(32b) MR-1 . . . MR-n DRAM Buf Ptr(32b) DRAM Buf Ptr(32b) Buffer Offset(16b) Buffer Offset(16b) MR Id(16b) MR Id(16b) Input MI(16b) MR Mem Ptr(32b) Lookup Result(Nb) MR Mem Ptr(32b) QM MR Lookup Key(16B) Common Router Framework (CRF) Functional Blocks Parse HeaderFormat Lookup Tx Rx DeMux MR-1 . . . MR-n • Lookup • Function • Perform lookup in TCAM based on MR Id and lookup key • Result: • Output MI • QID • Stats index • MR-specific Lookup Result (flags, etc. ?) • How wide can/should this be?

  41. MR Lookup Block Requirements • Shared NP Lookup Engine specific: • Number of Lookups per second required: • 1 lookup required per packet • 5Gb/s per NP on a blade • If we assume 6.25 MPkts/sec for 10Gb/s then for 5Gb/s would be 3.125 MPkt/s • We would want 3.125 M Lookups/sec per LA-1 Interface, total of 6.25 M Lookups/sec for the TCAM Core. • Number of MRs to be supported? • Will each get its own database? No. This would limit it to 16 which is not enough. • How many keys will each MR be limited to? • How much of Result can be MR-specific? • How much of Key can be MR-specific? • How are masks to be supported? • Mask core is same size as Data core. One mask per Entry • Global Mask Registers also available for masking key to match size of Entry during Multi Database Lookups where the multiple databases have different sizes. • How will multiple hits across databases be supported? • How will priorities be supported? • Priorities within a database are purely by the order of the keys. • For example, in a GM filter table if Keys 4 and 7 both match, Key 4 is selected. • Priorities across databases will have to be included in the Entries • Do we need support for non-exclusive (make a copy) filters? • Later? • How are GM with fields with ranges supported? • The IDT libraries support this by adding multiple entries, each with its own mask, to the DB to cover the range of the field.

  42. IPv4 MR Lookup Entry Examples • Route Lookup: Longest Prefix Match • Entry (64b): • MR ID (16b) • MI (16b) • DAddr (32b) • Mask: (32 + Prefix length) high order bits set to 1 • GM Match Lookup • Entry (148b): • MR ID (16b) • MI (16b) • If we could shorten the MR/MI fields by a total of 4 bits this would fit in a 144 bit core size. • SAddr (32b) • DAddr (32b) • Protocol (8b) • Sport(16b) • Dport(16b) • TCP_Flags (12b) • Mask: Completely general, user defined. • EM Match Lookup • Entry (136b): • MR ID (16b) • MI (16b) • SAddr (32b) • DAddr (32b) • Protocol (8b) • Sport(16b) • Dport(16b) • Mask: 136 high order bits set to 1

  43. IPv4 MR Lookup Databases • How many databases to use? • Three Options: • 3: a separate DB for each • 2: one DB for GM and one for RL and EM • 1: RL, GM and EM all in one DB • 3 Databases: • Means we would use Multi Database Lookup (MDL) command • More efficient use of CAM core entries as each DB could be sized closer to its Entry size • Guaranteed at least one Result from each Database if an existing match existed in each database. • 2 Databases: • We could use MDL command • Guaranteed one Result from GM and one from either EM or RL but not both! • Order is important: • EM filters would all go first in EM/RL DB, with full masks. • At most one entry would match • EM filters would always be higher priority than Routes. • This seems ok. That is what is typically implied by an Exact match filter. • If no EM filter match, we would get the best RL match. • RL entries would be sorted by prefix length so first match was the longest. • We could use two separate commands: Lookup or MHL for GM and MHL for EM/RL • Guaranteed at least one Result from each {GM,EM,RL} if an existing match existed in each. • Price: Two lookups per packet. • 1 Database: • Use Multi Hit Lookup (MHL) command • Efficient use of the CAM core entries is a potential problem. • Would not be as bad, if we could get the GM filters down to 144 bits by making the MR/MI fields a combined 4 bits shorter. • Order is important • EM Filters first • GM Filters second • Ordering within GM filters defines priority to a certain extent • With Result of 64 bit AD, we can get back at most 4 Results • So, we could end up with 3 GM filter results and then arbitrate priority between them based on a priority field in Result. • RL Entries last • EM and GM always take priority over RL. • Seems ok, that is typically what is implied by the filters. • Priority field in Results could be used to arbitrate between matched EM and GM filters

  44. MR ID (16b) MI (16b) DAddr (32b) SAddr (32b) Protocol (8b) DPort (16b) Sport (16b) TCP_Flags (12b) IPv4 MR Lookup Key Examples • Order matters: • Same Key will be applied to all Databases(MDL) • Multi-Database Lookup (MDL) • Each Database will use the number of bits it was configured for, starting at the MSB. • DAddr field needs to be first • TCP_Flags field needs to be last • Route Lookup: Longest Prefix Match • Key (64b): • MR ID (16b) • MI (16b) • DAddr (32b) • GM Match Lookup: Best/Range Match • Key (148b): • MR ID (16b) • MI (16b) • DAddr (32b) • SAddr (32b) • Protocol (8b) • Sport(16b) • Dport(16b) • TCP_Flags (12b) • MASK/Ranges • How will we handle • Masks for Addr fields • Ranges for Port fields • Wildcard for Protocol field • EM Match Lookup: Exact Match • Key (136b): • MR ID (16b) • MI (16b) • DAddr (32b) • SAddr (32b) • Protocol (8b) • Sport(16b) • Dport(16b)

  45. Mask Mask Mask Data Data Data Core Entries For RL DB 64 bits Core Size: 72 bits GMR=0xFFFFFFFFF 0xFFFFFFF00 Core Entries for EM DB 136 bits Core Size: 144 bits GMR=0xFFFFFFFFF 0xFFFFFFFFF 0xFFFFFFFFF 0xFFFFFFF00 IPv4 MR Lookup Key Examples Lookup Key: 148 bits out of 5 32-bit words transmitted with Lookup command. MR ID (16b) MI (16b) DAddr(32b) SAddr(32b) DPort(16b) SPort(16b) Proto (8b) TCP_Flags (12b) Pad (12b) W1 W2 W3 W4 W5 MDL GMR GMR GMR Core Entries for GM DB 148 bits Core Size: 288 bits GMR=0xFFFFFFFFF 0xFFFFFFFFF 0xFFFFFFFFF 0xFFFFFFFFF 0xF00000000 0x000000000 0x000000000 0x000000000

  46. IPv4 MR Lookup Result Examples • Result: • QID(20b) • Output MI (16b) • Priority(8b): range 0-255 • Drop(1b) • Stats Index (16b): 65535 indices for stats • If we need to put anything else in Result, bits should be taken from here. • Total of 45 bits • Each Database will have 64 bits of associated data, of which we will use the low order 61 bits. • And for MDL lookups only 61 of 64 bits of Associated Data is returned. • RTN=1b • ADSP=1b • AD WIDTH=01b • Results Mailbox: • D: Done (1b): set to 1 when ALL searches are completed. • H: Hit (1b): set to 1 if the search was successful and result is valid, 0 otherwise • MH: MHit (1b): set to 1 if search was successful and there were additional hits in database. • R: Reserved bits. • AD (Associated Data): 61 of the 64 bits of Associated Data from the Associated Data ZBT SRAM attached to TCAM. Results Mailbox AD[60:32] (1st Search) D H MH AD[31: 0] (1st Search) AD[60:32] (2nd Search) R H MH AD[31: 0] (2nd Search) AD[60:32] (3rd Search) R H MH AD[31: 0] 3rd Search) Not Used Not Used

  47. IPv4 MR Database Core Sizes • Route Database • Core Size: 72b • Entries per Segment: 8K • Number of Entries needed per route: 1 • Number of Routes per Segment: 8K • GM Database • Core Size: 288b • Entries per Segment: 2K • Number of Entries needed per filter: dependent on filter • Number of Filters per Segment: <= 2K • EM Database • Core Size: 144b • Entries per Segment: 4K • Number of Entries needed per filter: 1 • Number of Filters per Segment: 4K • Configuration used in our FPX-based Router: • 32 GM filters • 10K Ingress EM Filters • 10K Egress EM Filters • ~ 40K Route Entries • Configuration needed to achieve approximately the same numbers: • 5 Segments for Route Database • 1 Segment for GM Database • 5 Segments for EM Database • Total of 11 Segments (out of a total of 32 in TCAM)

  48. IPv4 MR Database AD Usage • Each Segment can be configured with a Base Address and a result size for calculating an address into the Associated Data. • The Associated Data is stored in a 512K x 36 bit ZBT SRAM • Using 64bit Results will give us 256K slots in the AD SRAM. • 48K Route DB Entries • <= 2K GM DB Entries • 20K EM DB Entries • Max Total of 70K Results needed. • Plenty of room in the AD for the IPv4 MR Results

  49. MPLS Lookup • MPLS uses a 20 bit Label • Key (52 bits): • MR ID (16b) • MI (16b) • MPLS_Label (20b) • Use an Exact Match Database • MPLS Label Database • Core Size: 72b • Entries per Segment: 8K • Number of Entries needed per label: 1 • Number of Labels per Segment: 8K • Drop Bit: • Does MPLS need a Drop Bit? • Perhaps it would use a Miss as the same thing as Drop. That is, the fact that a label is not entered in the Database is an indication that frames using that label should be dropped. • But, if we explicitly have a drop bit than Hits on those Entries could be counted separately from Misses. • What will MPLS Label Lookup Result look like? • New Label (20 bits) • QID(20b) • Output MI (16b) • Stats Index(16b) • Drop Bit (1b) • Total of 73 bits (128 bit wide Associated Data) • NOTE: We could use a 64 bit AD if we did not use the Drop bit and only supported 8-bit Stats Index.

  50. MPLS Lookup Result Examples • Result: • Reserved (3b): Don’t uses these, they will not show up in Results Mailbox. • New Label (20b) • QID(20b) • Output MI (16b) • Stats Index(16b) • Drop bit (1b) • Total of 73 bits (76 counting reserved bits) • DB will use 128 bits of associated data and will return the Associated Data followed by the Absolute Index. • We don’t need the Absolute Index and we don’t need the top 3 bits of the AD. With this ordering we just have to read the first 4 words on the results Mailbox instead of 5. • RTN=1b • ADSP=1b • AD WIDTH=10b • Results Mailbox: • D: Done (1b): set to 1 when search is completed. • H: Hit (1b): set to 1 if the search was successful and result is valid, 0 otherwise • MH: MHit (1b): set to 1 if search was successful and there were additional hits in database. • Absolute Index: Index offset from beginning of TCAM array. • Associated Data: 128 bits of Associated Data from the Associated Data ZBT SRAM attached to TCAM. D H MH Associated Data [124:96] Associated Data [95:64] Associated Data [63:32] Associated Data [31:0] Results Mailbox Reserved[31:22] Absolute Index[21:0] Not Used Not Used Not Used

More Related