Firefly: Illuminating Future Network-on-Chip with Nanophotonics

1. Firefly: Illuminating Future Network-on-Chip with Nanophotonics Yan Pan, Prabhat Kumar, John Kim†, Gokhan Memik, Yu Zhang, Alok Choudhary EECS DepartmentNorthwestern University Evanston, IL, USA {panyan,prabhat-kumar,g-memik,yu-zhang,a-choudhary}@northwestern.edu † CS DepartmentKAIST Daejeon, Korea jjk12@cs.kaist.ac.kr

2. Motivation On-Chip Network Topologies • Network-on-chip is critical for performance. Mesh[MIT RAW] [TILE64] [Teraflops] C-Mesh[Balfour’06][Cianchetti’09] Crossbar[Vantrease’08][Kirman’06] Others: Torus[Shacham’07], Flattened Butterfly[Kim’07], Dragonfly[Kim’08], Hierarchical(Bus&Mesh)[Das’08], Clos[Joshi’09], Ring[Larrabee], ……

3. Motivation Signaling technologies • Electrical signaling • Repeater insertion needed • Bandwidth density (up to 8 Gbps/um) [Chang HPCA‘08] • Nanophotonics • Bandwidth density ~100Gbps/ μm !!! [Batten HOTI’08] • Generally distance independent power consumption • Speed of light  low latency • Propagation • Switching [Cianchetti ISCA’09]

4. Motivation Nanophotonic components resonant detectors Ge-doped • Basic components coupler waveguide off-chip laser source resonant modulators

5. Motivation Resonant Rings • Selective • Couple optical energy of a specific wavelength • Radius r  Baseline Wavelength • Temperature t  Manufacturing error correction • Carrier density d Fast tuning by charge injection

6. Motivation Putting it together • Modulation & detection • ~100 Gbps/μm bandwidth density [Batten HOTI’08] 10001011 11010101 • 64 wavelengths DWDM • 3 ~ 5μm waveguide pitch • 10Gbps per link 10001011 11010101 ~100 Gbps/μmbandwidth density

7. Motivation What’s the catch? • Power Cost • Ring heating • Laser Power • E/O & O/E conversions • Distance insensitive • For short links (2.5mm) • Nanophotonics • Electrical • RC lines with repeater insertion • For long links • Nanophotonics • Cost stays the same • Electrical • Cost increases [Batten HOTI’08] [Cheng ISCA’06]

8. Motivation Here is the idea …… • Design an architecture that differentiates traffic. • Use electrical signaling for short links. • Use nanophotonics only for long range traffic. • What do we gain? • Low latency • High bandwidth density • High power efficiency • Localized arbitration • Scalability

9. Architecture of Firefly Outline • Motivation • Architecture of Firefly • Evaluation • Conclusion

10. Architecture of Firefly Layout View of 64-core Firefly • Concentration • 4 cores share a router • 16 routers

11. Architecture of Firefly Layout View of 64-core Firefly • Concentration • Clusters • Electrically connected • Mesh topology • 4 routers per cluster • 4 clusters Cluster 0(C0) Cluster 1(C1) Cluster 2(C2) Cluster 3(C3)

12. Architecture of Firefly Layout View of 64-core Firefly • Concentration • Clusters • Assemblies • Routers from different clusters • Optically connected • Logical crossbars

13. Architecture of Firefly Layout View of 64-core Firefly • Clusters • Electrical CMESH • Assemblies • Nanophotonic crossbars Nanophotonic Crossbars Efficient nanophotonic crossbars needed!

14. Architecture of Firefly Nanophotonic crossbars • Single-Write-Multiple-Read (SWMR) [Kirman’06] (CMXbar†) • Dedicated sending channel • Multicast in nature • Receiver compare & discard • High fan-out  laser power † [Joshi NOCS’09] SWMR Crossbar

15. Architecture of Firefly Nanophotonic crossbars • Multiple-Write-Single-Read (MWSR)[Vantrease’08] (DMXbar†) • Dedicated receiving channel • Demux to channel • Global arbitration needed! † [Joshi NOCS’09] MWSR Crossbar

16. Architecture of Firefly Reservation-assisted SWMR • Goal • Avoid global arbitration • Reduce power • Proposed design • Reservation channels • Narrow • Multicast to reserve • Destination ID • Packet length • Uni-cast data packet R-SWMR Crossbar

17. Architecture of Firefly Router Microarchitecture • Virtual-channel router • Added optical link ports and extra buffer. Separate receiving channels from other clusters. Dedicated sending channel for all traffic.

18. Architecture of Firefly Routing (FIREFLY_dest) • Routing • Intra-cluster routing • Traversing optical link

19. Architecture of Firefly Firefly – another look • Clusters • Short electrical links • Concentrated mesh • Assemblies • Long nanophotonic links • Partitioned crossbars • Benefits • Traffic locality • Reduced hardware • Localized arbitration • Distributed inter-cluster bandwidth

20. Evaluation Outline • Motivation • Architecture of Firefly • Evaluation • Conclusion

21. Evaluation Evaluation Setup • Cycle-accurate simulator (Booksim) • Firefly vs. CMESH, Dragonfly† and OP_XBAR • Synthetic traffic patterns and traces Electrical Hybrid Optical Hybrid [† Kim et al, ISCA’08]

22. Evaluation Load / Latency Curve • Throughput • Up to 4.8x over OP_XBAR • At least +70% over Dragonfly 4.8x 70% Bitcomp, 1-cycle Uniform, 1-cycle

23. Evaluation Energy Breakdown • Reduced hardware by partitioning • Reduced heating • Throughput impact • Locality • 34% energy reduction over OP_XBAR with locality

24. Evaluation Technology Sensitivity • α is heating ratio and β is laser ratio. • Firefly favors traffic locality. bitcomp taper_L0.7D7

25. Conclusion Conclusion • Technology impacts architecture • New opportunities in nanophotonics • Low latency, high bandwidth density • Tailored architectures needed • Firefly benefits from nanophotonics by providing • Power Efficiency • Hybrid signaling • Partitioned R-SWMR crossbars Reduced hardware/power • Scalability • Scalable inter-cluster bandwidth • Low-radix routers/crossbars