The Limits of Switch Bandwidth

1. The Limits of Switch Bandwidth Scott Kipp February 28, 2011

2. Increasing Bandwidth • Switch designs are limited by: • ASICS that are getting faster with process improvements • 45 nm processes can support 25Gbps lanes • Backplanes and connectors are getting faster and support 25 Gbps • FR4 will probably need to be replaced with Nelco13 • Front panel is set by 19” rack width and limits the port count • The height is optimized on 1.75” (1RU) increments and is a differentiator • Power of blade – optics usually only dissipate 50W of power max • Higher speed designs will need new optical module and electrical interface standards • 25-28 Gbps electrical signals is needed to increase the bandwidth of the switch

3. 10GbE ASIC and Module Design XENPAK XPAK XFP 300 Pin MSA SFP+ X2 QSFP – 4 lanes of 10GbE

4. 100GbE ASIC and Module Design Under Development By CFP MSA CXP CXP2? QSFP+ CFP4 CFP CFP2

5. Switch Design • Switches are typically designed for a 19” rack and this allows about 16-17” of port space for ports • Highest densities so far are 64 mini SFP+ (mSFP) ports at 10GE or 48 SFP+ and 4 QSFP ports • Blades inserted horizontally or vertically Modular Switch with 48 SFP+ ports Modular Switch with 64 mSFP+ ports 1U Switch

6. Switch Designs with 10Gbps interfaces 48 SFP+ Design - 480 Gbps of throughput, 48 Watts* of power Limited by size of SFP+ with 15mm pitch 4 CFP Design - 400 Gbps of throughput, 60 Watts** of power Limited by ASICs now, but soon limited by size of CFP Port with 84mm pitch CFP CFP CFP CFP 40 QSFP+ Design - 1600 Gbps of throughput, 80 Watts*** of power Limited by ASICs now, size of QSFP+ with 21mm pitch 32 CXP Design - 3200 Gbps of throughput, 80 Watts**** of power Limited by ASICs, CXP has 27mm pitch *Based on 1 Watt of power/SFP per SFF-8431 *** Based on 2 Watts of power/QSFP+ **Based on 15 Watts/CFP for 10km links ****Based on 2.5 Watts of power / CXP

7. Summary of 10 Gbps Switch Designs • Support of single-mode applications requires larger modules because of higher power dissipation • Multimode modules support less power and are denser

8. Switch Designs with 25-28 Gbps interfaces 48 SFP+ Design - 1.3 Tbps of throughput at 32GFC, 48 Watts* of power Limited by size of SFP+ with 15mm pitch 8 CFP2 Design - 800 Gbps of throughput, 96 Watts** of power Limited by ASICs now, but soon limited by size of CFP2 Port with 42mm pitch CFP2 CFP2 CFP2 CFP2 CFP2 CFP2 CFP2 CFP2 40 QSFP+ Design - 4 Tbpsof throughput, 80 Watts*** of power Limited by ASICs and power, QSFP+ pitch is 21mm 16 CFP4 Design - 1.6Tbpsof throughput, 96Watts**** of power Limited by ASICs, CXP pitch is 27mm *Based on 1 Watt of power/SFP per SFF-8431 **Based on 12 Watts/CFP2 for 10km links ***Based on 2 Watt of power/QSFP ****Based on 6Watts/CFP2 for 10km links

9. Summary of 25Gbps Switch Designs • Support of single-mode applications requires larger modules because of higher power dissipation • Multimode modules support less power and are denser

10. Tbps Bandwidths require faster speeds • 10Gbps lanes will limit the throughput of the switch/blade in the next few years • Difficult to expand beyond 128 ports / ASIC • Module size limits bandwidth and causes long trace lengths • In the next few years, 25-28Gbps signaling needed to support Tbps switch/blade bandwidths • 4X25Gbps easier to support than 10X10Gbps in near future • Cost of 25-28G signaling compared to 10Gbps will be an interesting battle in the next few years • Terabit Ethernet will require major innovation • 100x10G or 40x25G not practical for pluggable modules • 10X100G not practical from signaling perspective

11. Thank You