FPX/SPC Integration

1. FPX/SPC Integration Fred Kuhns and Alex Chandra Applied Research Laboratory

2. Rule • if GM filter: Exclusive, Ingress and Non-multicast • if OVIN is invalid use result of RL • if QID valid use it, else use dgram QID • else • both OVIN and QID must be valid • In all cases insert resulting OVIN in inter-port shim • if packet from SPC and RC == 0, then OVIN and QID must be valid. The SPC may change these fields. • if GM filter: Exclusive, Egress and Non-multicast • Always use OVIN from Shim • if QID is valid use it, else use dgram QID • If packet from SPC and RC == 0, then OVIN and QID in shim must be valid – SPC may alter these values.

3. Notes: mtg [09/16/02] and subsequent discussions • TTL Processing Note: • Ingress ISAR: If TTL = {0,1} and packet from line-card then send to SPC, bypassing CARL. Otherwise process normally (TTL is not changed or inspected further). • Egress OSAR: If TTL > 0 and packet to line-card, then decrement. Always send regardless of TTL value (TTL is not changed or inspected further). • ICMP messages: We will support sending and receiving the following • Time exceeded for datagram: FPX ingress port sends packet to SPC if TTL = 1/0. The SPC will decide whether to deliver locally or generate ICMP message. • Destination unreachable: if there is no route entry then send to SPC – we already do this. SPC responds with ICMP msg if it is unable to resolve route. • ICMP echo and reply: This will have the routers local IP address so we “can” implement at CP – need to look into how we do this at the CP. Alternatively, we can define a route table entry for the local interfaces IP address to send those packets to the SPC. • Filter on IP header Source Address: • Invalid Source addresses: • The high order 3 bits equal 111. Note class D (Multicast) has 1110, class E (Reserved) has 1111 and broadcast is of course all 1’s. • Address is all 0’s • Address high order byte equals 127d (0x7f) – i.e. the loopback interface • In an invalid Source Address then silently drop. • Note: while we “should” drop directed broadcasts and filter for IP spoofing we will not do this due to the difficulty of implementation. Discussion: ??

4. Notes (2) • Filter on IP header Destination Address: • Invalid Destination addresses for forwarding (IP lookup): • {-1, -1}, {0, 0}, {prefix, 0}, Class E (240.0.0.0 – 247.255.255.255) • If valid unicast then forward: high order 3 bits do not equal 111 and address is not all 0’s. • If valid broadcast address then send to CP • there are two broadcast address of concern: 0xffffffff and local “subnet” broadcast. • Can we use the General Match filter for this? • If multicast address (first 4 bits = 1110) then ISSUE?? • If class E, all zeros or any other case then silently drop. • IP Fragmentation: • Assume MTU is same on all ports, so will not need to fragment • if receive an IP fragment, then silently drop. Rationale: describe?? • Path MTU discovery: • support the IP option form: describe approach?? • do not support the ICMP based approach: describe approach?? • Other Packet Processing Errors: • IP header parameter errors other than version: Silently drop. • Errors: Checksum error, length error, invalid source address, Others?. • Rationale: Since the IP header may be corrupted or counterfeit we can not and should not rely on any of the field values. • AAL5 checksum or length errors – silently drop packet. Rationale same as for IP parameter errors. • Filter packets looking for source addresses – is this “doable” in current design. Protects against certain types of DOS attacks and required by rfc.

5. Notes(3) • Will support GM filtering on output port selectable via a configuration flag. Default case does not enable this feature. • How do we deal with Multicast and Broadcast? • with IGMP we could use a GM filter to send these multicast packet to the CP – can/should we do this?? • General Match Filters and default routes: • Alex will add an additional bit to the GM filter indicating OVIN validity. If OVIN is not valid then the results of the FIPL lookup are used. If the fipl lookup fails then treat as you would any datagram without a valid route. • ICMP Redirect • Should we generate an ICMP redirect if the resulting OVIN == IVIN??

6. System Architecture ControlProcessor Switch Fabric ATM Switch Core IPP OPP IPP OPP IPP OPP IPP OPP IPP OPP IPP OPP FPX FPX FPX FPX FPX FPX Port Processors SPC SPC SPC SPC SPC SPC LC LC LC LC LC LC Line Cards

7. CP Ingress Traffic (Input Port) Egress Traffic (Output Port) SPC SPC SPC/FPX EG_VCI IN_VCI VCI = IBase + SP VCI = IBase + SP FPX FPX LC LC VCI = OBase + Out_PN VCI = OBase + In_PN Logical view of SPC/FPX • Fast Path: Core IP processing in FPX (classification, packet scheduling, distributed queuing etc.) • Slow Path: Exceptional and “active” processing in SPC • Control Path: messages from the CP

8. Switch Line Card Ingress I only care about the resulting behavior. So some details are missing if they do not directly support the “story” SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX ISAR – receive packet from Line card or SPC • Reassemble packet, verify AAL5 checksum. • If invalid checksum, silently drop packet. • From LC: • If bad IP parameters or invalid checksum then silently discard packet {ip hlen < 5, ip cksum invalid, AAL5 cksum invalid, ip tlen + shim_len != AAL5 len} • If IP options or not IPv4, Set EX flag and send to SPC • If TTL = 0/1, send to SPC • otherwise “send to” CARL • From SPC: • Drop packet if: • Not HO: if hlen < 5, IP/AAL5 cksum invalid, AAL5 len != hlen+16 • HO: if hlen < 5, IP/AAL5 cksum invalid, AAL5 len != 120+16, DP=1 • Note: for HO=0, ISAR expect SPC to drop packet if necessary. • if RC = 1, “send to” CARL for classification • Fill in Shim values: • InVIN using PN register, what about sub-port? • Flags: • RC = 1 unless SPC returns RC = 0, DP = 1 or EX = 1. • NM = 0 in all cases (set by CARL) • EX = 1 when IP options, 0 otherwise • HO = 0, CARL may reset • HR = 1 when HO = 1 and returned from SPC, 0 otherwise • FM = 1 from LC otherwise 0 from SW • TO = 1 => LC and 0 => SW. redefined by CARL 40 41 42 50 43 51 44 52 45 53 46 47

9. in: <5-tuple> GM Switch Line Card Ingress Notes: MTP = Multicast Tree Position, 8 bit # SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX Classify Packet out: <E: priority, HO, DP, QID, OVIN> <N:HO, QID, OVIN> CARL out: <Flags (FT, HO, LR, TO), OVIN0, QID0, Rate0, OVIN1, OVIN1 Rate1, MTP?> in: <FM, 5-tuple, MTP> EM (priority) out: <OVIN> QID = SW_DGRAM_BASE (440) + OVIN(4:2) fipl (priority) in: <destination> 40 • in parallel, GM, EM and dgram lookup • update OVIN & QID with highest priority matching filter/route • if packet from SPC and RC = 0 and DP = 0 then assume TO and OVIN are correct (TO assume = 0/SW, erroneous for it to = 1/LC since route lookup performed at ingress port) • if (QID = 0)then QID = SW_DGRAM_BASE (440) + OVIN(4:2) • else QIDassumed valid and not modified by CARL • QMGR remaps QID = QID + 128 • Update TO and DP (GM Exclusive may require we drop packet) • Send to OSAR 41 ISAR 42 50 43 51 44 52 45 53 46 47

10. Switch Line Card Ingress SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX QM/OSAR/PSM • QM • Place packet in correct queue, schedule for transmission • When scheduled, send to OSAR • OSAR • encapsulate in AAL5 frame • To SW • update shim: flags (all zero), IVIN, OVIN, PPN, MTP • encapsulate in AAL5 frame and determine VC from OutVIN • To SPC (ingress and egress same) • Encapsulate on AAL5 frame, set VC • Insert Intra-port shim 40 ISAR/ CARL 41 42 50 43 51 44 52 45 53 46 47

11. ... ... QID 448 – 511 Datagram Queues (64) ... Switch Line Card QID 440 – 447 datagram queues to switch (TO = 0) QID 256 – 439 Reserved Flows (184) ... FPX: Queue assignment after classification *Note on QIDs to the SPC 1 .. 7 are reserved for SPC bound traffic QID Port Description Flag Supported 1 Ingress No match in CARL NM=1 Yes 2 Ingress Not IPv4 EX=1 Yes 3 Ingress TTL = 0/1 EX=1 Yes 4 Ingress IP Options EX=1 Yes 5 Egress Output link down EX=1 No 6 Not defined 7 Not defined SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX After CARL classifies packet it is placed into a queue and sent on a VC based on TO, SR, QID and OutVIN. Does not show reclassification. Software assigns all QIDs except 1-7. The range 128 – 255 is calculated internally. *QID1-127 SR = 0, TO = 1/0 QID128-255 (qid = qid + 128) SR = 1, TO = 1/0 OutVIN (port) used to calculate VC RC = 1 TO = 0 TO = 1 40 CARL 41 42 50 43 51 44 52 45 53 46 47 OutVIN (sub-port) used to calculate VC TO = 1 TO = 0

12. Switch Line Card Egress SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX ISAR – receive packet from switch 40 41 42 50 • Receive packet from switch; Set FM = 0 (SW) • IP len + 8 != AAL5 length, then silently drop • otherwise same as ingress ISAR 43 51 44 52 45 53 46 47

13. in: <FM, 5-tuple, “depth”> EM pkt ref Switch Line Card Egress SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX Classify Packet CARL out: <Flags (FT, HO, LR, TO), OVIN0, QID0, Rate0, OVIN1, OVIN1 Rate1, MTP?> ISAR • Exact Match only • out 1 or 2 tupples {TO, OVIN, QID, Rate, MB} • From SPC and RC = 0 and DP = 0 (bypass pkt), then TO flag and OVIN are assumed correct • if TO = 1/LC • if QID = 0 QID = LC_DGRAM_BASE(448) + OVIN(1:0):SA(1:0):DA(1:0) • else QID not modified by CARL QMGR set QID = QID + 128 • else same as ingress case • Send to QM include pkt pointer, rates, shim fields. 40 41 42 43 50 44 51 45 52 46 53 47

14. Switch Line Card Egress SPC VC = 56 TO = 0 ingress VC = 57 TO = 1 egress FPX QM/OSAR • QM • send pkt, rates and shim fields to OSAR • OSAR • Sending to LC: • remove shim, • update LFS option (rate2) • decrement TTL if > 0 (always send even if TTL = 0) • update IP hdr checksum • encapsulate in AAL5 frame and calculate VC from OutVIN • Sending to SPC: • Encapsulate on AAL5 frame, set VC • Insert Intra-port shim ISAR 40 41 42 43 50 44 51 45 52 46 53 47

15. FPX to SPC – Intra-Port Shim • Flag Values • DP (Drop Packet): SPC directive to FPX. Will always be zero coming into SPC. • RC (Reclassify Packet): SPC directive to FPX. May be 0 or 1 going to SPC. SPC must redefine. • NM (No Match): FPX can not classify (no route), FPX to SPC. Set to 0 before returning to FPX. • EX (Exception Packet): Requires non-fast path; IP options, Non IPv4. FPX to SPC. Set to 0 before returning to FPX. • HO (Header Only): FPX to SPC; Set if packet size > 120B and only first chunk sent to SPC. SPC must not modify. SPC can not drop internally if this bit is set. • HR (Header Only Return): Internal FPX flag. SPC must not modify. • FM (From LC/SW): Set by FPX. SPC can use but not modify. • TO (To LC/SW): Set by FPX. SPC can use but not modify. Input VIN Output VIN PPN Flags Queue Identifier (QID) Port # Sub # Port # Sub # Port # DP RC NM EX HO HR FM TO TTL Chunks Queue Length Packet Pointer Second Chunk Pointer

16. VIN PN SP FPX to SPC, Inter-port Shim • Inter-port Shim – expect the Input and Output VINs to be correct. No other information needed – in particular the PPN is not used by the SPC. Flags Input VIN Output VIN PPN

17. Looking at the Shim Fields (Intra-Port) • Flags: Input to SPC → indicates required processing {NM, EX, HO} • NM → FPX lookup failed, perform lookup • If SPC can resolve route then • set {Output VIN, QID} • set NM = 0, • return to FPX. • else • send ICMP to source and drop packet (internal to SPC). • EX → Implies IP Options, IP “parameter error”, or Non IPv4. • If IP option, process options and return to FPX with EX = 0. Further iff an EM filter has been added/removed then set RC = 1. • If Non IPv4 then SPC either “consumes” or process as if NM is set. • HO → Only the header (first 120 bytes) of packet. • if need to drop packet, then set DP = 1 and return to FPX

18. Scenario Overview • Default system configuration (routes, FPX register file) • FPX, SPC and CP • Basic datagram forwarding, no GM or EM filters • valid packets • various error conditions • get traffic statistics (counts) • Forwarding with GM and EM filters • CP adds default GM and EM filters • traffic: no matching filters • traffic: match GM and EM. Try mix of traffic (dgram, reserved) and sending to SPC • get traffic statistics (counts) • Control FPX from SPC • add/remove EM filter • get traffic statistics (counts) • Add/remove filters and route entries under load

19. Default Configuration for Scenarios • Initial, Basic Configuration • SPC configuration – none for datagram forwarding test • Will use NCHARGE where possible. • NID image stored in PAL on FPX. Must be configured (VC translation table) then correct RAD image selected and downloaded. • RAD Configuration • FPX register values (separate slide) • Download default FIPL table (separate slide) • Default general match and exact match filters - none for initial datagram forwarding test • Other FPX specific configuration issues? • None that I know of

20. RAD Configuration • CP configuration module will handle basic configuration issues • RAD configuration • initialize and configure SPCs (indicate if FPX present) • use NCHARGE to download and start RAD. • configure NID VC translation table • RAD Initialization – see separate slide with register file defaults • Set/verify default VCI values: CP Control VCI • Registers SPC Control VCI, DQ Control VCI, Ibase VCI, Obase VCI, SPC IN VCI andSPC EG VCI • Set port number: • register PN • Set/verify default route and exact match priorities • registers RL Priority and EM Priority • Set/verify default link rate, SPC rate and speed advantage • registers QM Speed, QM Link and QM SPC • Does software set or just read the registers RL RootNodePtr and EM Offset? • Configure CARL for default tests • add our usual default routes – see separate slide with default routes • For classifier tests, download default GM and EM filters – see separate slide

21. FPX Register File • Configuration registers • We need to have test cases where we read, modify and verify. • Default values are correct for testing.

22. Default Routes and Interface Addresses • Download default routing table at initialization. • SPC represents the VIN differently: This needs to be fixed. • At some point we need to have a better (i.e. more convenient) way of specifying routes and dynamically changing them using a standard routing protocol such as OSPF.

23. Scenario 1: Datagram Forwarding, No G/E filters • Datagram Forwarding, Valid packet, Ingress and Egress Ports • Valid route, valid IP header. Forward packet • Datagram Forwarding, Error/Exceptional processing, Ingress Port • IP option other than LFS, send to SPC with EX flag = 1 (see LFS processing below) • Valid packet, no route, send to SPC with NM flag = 1 and QID = 1 • SPC has route, update OutVIN and QID. Send back to FPX, NM = 0 • SPC does not have route, send ICMPdestination unreachable to source if HO = 0 then drop packet else set DP flag = 1, return packet to FPX. • Valid route, TTL = 0 or 1. Send to SPC with EX flag = 1 and QID = 3?. • If packet destination address is local then deliver to CP. • If not local then send ICMP Time Exceeded message to source, if HO = 0 then drop packet else set DP flag = 1 and return to FPX. • Checksum error, invalid length or other parameter error, FPX silently drops packet. • If the IP version is not valid then set EX flag = 1 and send to SPC • if IP version not supported then if HO = 0 drop pkt else set DP flag = 1 and return packet to FPX for dropping. Otherwise SPC processes packet – though it is not clear what this means. • Buffer overrun – silently drop packet. • Datagram Forwarding, Error/Exceptional processing, Egress Port • IP options are not checked, Route and GM filters are not checked, Assume SHIM correct • Checksum or other parameter error, drop silently • If indicated interface or VC not available?, set EX flag = 1 and send to SPC with QID = 4?. SPC will send ICMP destination unreachable message to source, set DP flag = 1 and return to FPX. • Buffer overrun – silently drop packet.

24. Scenario 2: Matching GM or EM Filters • Install GM and EM filters from CP • See word document for filter specification • Dynamically add and remove EM filters • Dynamically add and remove GM filters • Dynamically add and remove routes • Read packet counters for each filter type • Packet Matches GM – repeat for both Ingress and Egress ports • Non-Exclusive filter • Exclusive filter • Drop Packet, Send to SPC, Header Only • Both Exclusive and Non-Exclusive filter • Exclusive sent to SPC, Non-Exclusive send header only to SPC • Packet Matches EM filter – repeat for both Ingress and Egress ports • Reserved flow – do not send to SPC but set OVIN and QID • Send to SPC, complete packet • Send to SPC, Header Only • Packet Matches EM and GM Filters at Ingress port • Alter priorities to verify CARL operation (which lookup engine “wins”) • EM and GM exclusive, EM has higher priority, reserved • EM and GM exclusive, GM has higher priority, drop packet • Should we try with RL having the highest priority?

25. Scenario X: LFS Option Processing • Case 1: new LFS flow • FPX send packet with IP option to SPC before it is classified. • SPC processes IP option and send control cell to FPX to install new EM filter for flow. • SPC assigns an unused QID • Does the SPC have to also assign the OutVIN? Is so then the SPC has a complete routing table available. • SPC waits for FPX to reply to control message before sending packet back with the RC flag = 1. • The FPX classifies the packet and places in correct queue. • Case 2: Existing LFS reservation, no change required • RC flag = 1, packet returned to SPC. • Case 3: Existing LFS reservation, decrease allocation • Increase available count, send control message to FPX (updated reservation), wait for reply, RC flag = 1, send packet to FPX, look for any partial allocations, allocate freed bandwidth, send any necessary control cells to FPX with updates • Case 4: Existing LFS reservation, increase allocation • if request can be satisfied then send control cell to FPX updating entry, wait for reply, set RC flag = 1 and return to FPX. • if request can be partially satisfied then add entry to Partial Allocation Table, send control cell to FPX updating entry, wait for reply, set RC flag = 1 and return to FPX. • if request can not be satisfied then add entry to Partial Allocation Table, set RC flag = 1 and return to FPX. • Case 5: Existing LFS reservation, Release request • Increase available count, send control message to FPX removing EM filter, set the QID and OutVIN in shim to be same as the released EM filter, (RC = 0), send packet to FPX. • Case 6: No existing LFS reservation, Release request • do nothing, send packet to FPX with RC flag = 1.