Maintaining Packet Order in Two-Stage Switches

1. Maintaining Packet Order in Two-Stage Switches Isaac Keslassy, Nick McKeown Stanford University

2. OQ routers: • + work-conserving • - memory bandwidth = (N+1)R R R R R R R • IQ routers: • + memory bandwidth = 2R • - arbitration complexity Bipartite Matching Two Successive Scaling Problems

3. Today: 64 ports at 10Gbps, 64-byte cells. • Arbitration Time = = 51.2ns • Request/Grant Communication BW = 17.5Gbps 64bytes 10Gbps IQ Arbitration Complexity • Scaling to 160Gbps: • Arbitration Time = 3.2ns • Request/Grant Communication BW = 280Gbps • Two main alternatives for scaling: • Increase cell size (Kar et al., 2000) • Eliminate arbitration (Chang et al., 2001)

4. Desirable Characteristics for Router Architecture • Ideal: OQ • 100% throughput • Minimum delay • Maintains packet order • Necessary: able to regularly connect any input to any output • What if the world was perfect? Assume Bernoulli iid uniform arrival traffic...

5. 1 1 N N Cyclic Shift? Uniform Bernoulli iid traffic: 100% throughput

6. 1 1 N N Problem: real traffic is non-uniform Cyclic Shift? q(t) b(t) (t) Long-term service opportunities exceed arrivals:

7. 1 2 2 1 1 1 1 N N N Two-Stage Switch External Inputs Internal Inputs External Outputs Load-balancing cyclic shift Switching cyclic shift 100% throughput for broad range of traffic types (C.S. Chang et al., 2001)

8. External Inputs Internal Inputs External Outputs q(t) 2(t) 1(t) 1 a(t) b(t) 1 1 N • First cyclic shift: N N • Traffic rate: • Long-term service opportunities exceed arrivals: Two-Stage Switch • (2 = 1 possible)

9. Two-Stage Switch Characteristics • Eliminates arbitration • 100% throughput • Conventional router packaging

10. 1 1 1 N N N Using a Single Stage Twice Linecards Lookup Phase 1 Buffer 1 Lookup Buffer 2 Phase 2 Lookup Buffer 3

11. Optical Switch Fabric Two-Stage Switch Characteristics Optical links Racks of linecards

12. 1 2 2 1 1 1 1 N N N Two-Stage Switch Characteristics External Inputs Internal Inputs External Outputs Cyclic Shift Cyclic Shift Problem: unbounded mis-sequencing

13. t External Inputs Internal Inputs External Outputs 2 1 1 1 2 2 1 1 1 N N N Cyclic Shift Cyclic Shift Full Frames First (FFF):Intuitive Idea • Idea: • Spread cells evenly across all linecards

14. External Inputs Internal Inputs External Outputs 2 1 2 3 1 1 1 1 N N N Cyclic Shift Cyclic Shift Full Frames First (FFF):Intuitive Idea • Idea: • Spread cells evenly across all linecards • Read them in order

15. External Inputs Internal Inputs External Outputs 1 1 2 1 1 1 2 3 N N N Cyclic Shift Cyclic Shift Full Frames First (FFF):Intuitive Idea • Idea: • Spread cells evenly across all linecards • Read them in order

16. t External Inputs Internal Inputs External Outputs 2 1 1 2 2 1 3 3 1 1 1 N N N Cyclic Shift Cyclic Shift First Problem Problem: if two packets don’t arrive consecutively, there may be a hole in the reading sequence

17. t 3 1 2 Flow Load Balancing Coordination Buffer (VOQ) 1 1 2 1 1 1 2 3 N N N Cyclic Shift Cyclic Shift Coordination Buffer Solution: collect cells from a flow in a coordination buffer, and load-balance them among linecards

18. t Input 1: Input 2: a b t Flow Load Balancing Coordination Buffer (VOQ) 2 3 1 b a 1 3 1 2 1 1 1 2 N N N Cyclic Shift Cyclic Shift Second Problem Problem: No access to cell 2 because of head-of-line blocking

19. a b 1 3 2 Expanding VOQ Structure Solution: expand VOQ structure by distinguishing among switch inputs

20. FFF: Guarantees Theorem 1: for any arrival process for which OQ has 100% throughput, so does FFF Theorem 2: for any arrival process, Davg (FFF)  Davg(OQ) + (4N2 - 2) Theorem 3: FFF maintains packet order

21. R R 1 1 1 1 1 1 2 2 2 2 2 2 Cyclic Shift Cyclic Shift R/N R/N R/N 3 3 3 3 3 3 Passive mesh 2R/N Passive mesh Two-Stage Switch in Optics

22. Summary • FFF: practical algorithm that solves mis-sequencing • Same throughput as OQ, and average delay within a bound • New approach to optical switching