COM S 614 Advanced Systems

1. COM S 614Advanced Systems Novel Communications U-Net and Active Messages

2. Problem Domain • System Area Networks • Cluster Computing • Rack-and-stack builds • COTS hardware

3. Presentation Parts • U-Net — say it fast • Active Messages — keep it simple

4. Cluster Computing • File server clusters • Database servers • Storage Area Networks • Web Servers • Scientific Applications

5. Parallel Applications • Communications Intensive • Small messages • 50% < 200 bytes • RPC style programming

6. Traditional Network Interface • NIC on IO bus • DMA to kernel buffers • Interrupt driven

7. Performance Barriers • Multiple data copies • Kernel processing bottleneck • Context switch expensive

8. Goals • Remove kernel • Zero-copy send/receive • Simple API

9. User Level Network Interface • Application thinks it owns the adapter • Application owns its buffer pool • Application handles retries, scheduling

10. ULNI Pieces • Virtual interface • Large buffer pool for data • Ring buffers for send/receive descriptors • Memory mapped status registers

11. ULNI API • Kernel used to establish VI • Common functions for descriptor and pool management • DMA direct to user space

12. Virtual Interface Architecture • Compaq, Intel, Microsoft • Standard for communications within a cluster • MS provides similar WinSock Direct • WinSock Disaster • 10X latency over NIC • 60-70% max thruput

13. ULNI Lessons • Network transfers through DMA • Programmed IO can be better • Polling is simple, but not efficient • Alternative is interrupts, worse! • Unreliable, no flow feedback • Point-to-point • Context switch VI slow

14. ULNI Future • Interface mapped to coherent memory bus Note: might waste a page of SRAM for status registers, almost all of it is wasted • Polling code inserted by compiler • Multicast traffic managed by switches, spanning tree • Multiple NI’s an interesting research direction

15. How to say it… • Don’t wait for the response • Do simple things • Messages look like long jumps

16. Procedure call?! • Identical code on all nodes • Really small messages (16 bytes) • Not allowed to block • “Deadlock avoidance is a rather tricky issue…”

17. Active Message Applications • Parallel computation • Data flow • Flow control, lock management • Virtual Mapped Memory Communication

18. AM Lessons • Simplify buffer management • High caliber initial implementation • Willingness to distribute and support the code • Provided a common vocabulary for developers • Efficient RPC methods

19. Fini!