Using optics to scale Internet Routers Computer Forum, May 2003

1. Using optics to scale Internet Routers Computer Forum, May 2003 Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford.edu www.stanford.edu/~nickm

2. Problems facing routers • The problem: • Capacity scales slower than user traffic. • Power limits capacity. • All-optical routers are infeasible. • Our approach • Explore how optics can be used inside routers to reduce power, and therefore scale capacity. • Design a high capacity router that exposes the problems, and leads to interesting research questions.

3. Internet Routers Line Capacity 2x / 7 months User Traffic 2x / 12months Router Capacity 2.2x / 18months Moore’s Law 2x / 18 months DRAM Random Access Time 1.1x / 18months

4. Power consumption of single-rack Internet routers

5. Multi-rack distributed routers reduce power density Optical links 100s of metres Switch Core Linecards

6. Motivating Design: 100Tb/s Optical Router Optical Switch Electronic Linecard #1 Electronic Linecard #625 160-320Gb/s 160-320Gb/s 40Gb/s • Line termination • IP packet processing • Packet buffering • Line termination • IP packet processing • Packet buffering 40Gb/s 160Gb/s Arbitration 40Gb/s Request 40Gb/s Grant (100Tb/s = 625 * 160Gb/s)

7. Research Groups • Mark Horowitz horowitz@ee.stanford.edu • Nick McKeown nickm@ee.stanford.edu • Olav Solgaard olav@ee.stanford.edu • David Miller dabm@ee.stanford.edu • 8-10 PhD students

8. 100Tb/s optical router • Objective • To determine the best way to incorporate optics into routers. • Push technology hard to expose new issues. • Photonics, Electronics, System design • Motivating example: The design of a 100 Tb/s Internet router • Challenging but not impossible (~100x current commercial systems) • It identifies some interesting research problems

9. Research Problems • Linecard • Memory bottleneck: Address lookup and packet buffering • Architecture • Arbitration: Computation complexity • Switch Fabric • Optics: Fabric scalability and speed • Optics: Optical modulators • Electronics: Low power optical links • Electronics: Optical switch control • Electronics: Clock recovery for intra-system links • Packaging.

10. Outline • Load-Balanced Switch Overview • Passive Mesh Paradigm • WGR-based Switch Fabric • Hybrid Optical-Electrical Switch Fabric

11. The Arbitration Problem • A packet switch fabric is reconfigured for every packet transfer. • At 160Gb/s, a new IP packet can arrive every 2ns. • The configuration is picked to maximize throughput and not waste capacity. • Known algorithms are too slow.

12. Load Balancing 1 1 1 N N N Load-Balanced Switch External Inputs Internal Inputs External Outputs Load-balancing cyclic shift Switching cyclic shift • First stage load-balances incoming flows • Second stage is the usual switching cyclic shift

13. 1 2 2 1 1 1 1 N N N Load-Balanced Switch External Inputs Internal Inputs External Outputs Load-balancing cyclic shift Switching cyclic shift 100% throughput for broad range of traffic types (C.S. Chang et al., 2001)

14. Outline • Load-Balanced Switch Overview • Passive Mesh Paradigm • WGR-based Switch Fabric • Hybrid Optical-Electrical Switch Fabric

15. 1 1 1 1 1 1 R/N R/N R/N 2 2 2 2 2 2 3 3 3 3 3 3 Passive mesh 2R/N Passive mesh Passive Mesh R R Cyclic Shift Cyclic Shift No more arbitrations, no more reconfigurations!

16. Outline • Load-Balanced Switch Overview • Passive Mesh Paradigm • WGR-based Switch Fabric • Hybrid Optical-Electrical Switch Fabric

17. 1 1 1 l l l … , N 1 2 AWGR (Arrayed Waveguide Grating Router) A Passive Optical Component • Wavelength i on input port j goes to output port (i+j-1) mod N • Can shuffle information from different inputs 1 l Linecard 1 Linecard 1 1 Linecard 2 1 l Linecard 2 2 NxN WGR 1 l Linecard N Linecard N N

18. Fixed Laser/Modulator Detector l l 1 1 1 N 1 1 l l l l , , l l 1 2 1 2 Linecard 1 Linecard 1 2 2 2 1 l l … … N N l l N N l l 1 1 2 1 l l 2 2 , l l , l l Linecard 2 Linecard 2 1 2 1 2 2 2 2 3 … l l … NxN WGR N N l l N N l l 1 1 N N-1 N N l l l l , , l l 1 2 1 2 Linecard N Linecard N 2 2 1 N l l … … N N l l N N WGR Based Solution

19. Switch fabric design • Design a switch fabric • For load-balancing and switching stages • 625 ports of 2x160 Gbps • Features: • Flexibility: arbitrary addition and deletion of linecards (due to upgrades/failures) • Scalability

20. Outline • Load-Balanced Switch Overview • Passive Mesh Paradigm • WGR-based Switch Fabric • Hybrid Optical-Electrical Switch Fabric

21. From Linecard Mesh to Group Mesh 2R/6 Linecard 1 Linecard 1 Linecard 2 Linecard 2 Linecard 3 Linecard 3 Linecard 4 Linecard 4 Linecard 5 Linecard 5 Linecard 6 Linecard 6

22. From Linecard Mesh to Group Mesh Linecard 1 Linecard 1 Group 1 Group 1 3R Linecard 2 Linecard 2 Linecard 3 Linecard 3 3R 3R Linecard 4 Linecard 4 Group 2 Group 2 3R Linecard 5 Linecard 5 Linecard 6 Linecard 6

23. 2R/3 Linecard 1 Linecard 1 Linecard 2 Linecard 2 Linecard 3 Linecard 3 Example

24. Example 8R/3 Linecard 1 Crossbar Crossbar Linecard 1 Linecard 2 Linecard 2 4R/3 4R/3 Linecard 3 Crossbar Crossbar Linecard 3 2R/3

25. StaticMEMS 4R/3 Linecard 1 2x3Crossbar 2x3Crossbar Linecard 1 4R/3 4R/3 Linecard 2 Linecard 2 2R/3 Linecard 3 2x3Crossbar 2x3Crossbar Linecard 3 4R/3 Example

26. Hybrid Switch Fabric Static MEMS Electronic Switches Fixed Lasers Optical Receivers Electronic Switches 1 1 GxG MEMS Linecard 1 Linecard 1 2 2 LxM Crossbar MxL Crossbar Linecard 2 3 3 Linecard 2 1 M M Linecard L Linecard L GxG MEMS Group 1 Group 1 1 1 2 Linecard 1 Linecard 1 2 2 LxM Crossbar MxL Crossbar Linecard 2 3 3 Linecard 2 GxG MEMS M M Linecard L Linecard L 3 Group 2 Group 2 1 1 Linecard 1 Linecard 1 2 2 LxM Crossbar MxL Crossbar Linecard 2 3 3 Linecard 2 GxG MEMS M M Linecard L Linecard L M Group G Group G

27. Conclusion • Power: 100Tb/s optical switch fabric consumes almost no power. • Optics: No optical components reconfigured on packet-by-packet basis. • Capacity: No centralized arbitration and scheduling algorithms. • Throughput: 100% throughput guarantee.