Design and implementation  Main features  Socket API

1. A Scalable High-performance Communication Library for Wide-area EnvironmentsHideo Saito Ken HironakaKenjiroTauraThe University of Tokyo{h_saito, kenny, tau}@logos.ic.i.u-tokyo.ac.jp Overview ScalableSockets (SSOCK) A communication library for parallel computation Solves the connectivity issues involved with WANs Achieves high scalability and performance Related work SOCKS (Leech et al. ‘96) UDP hole punching (Rosenberg et al. ’03) TCP splicing (Denis et al. ‘04) IPOP (Ganguly et al. ‘06) SmartSockets (Maassen et al. ‘07) DHTs • Background • Increase in the bandwidth of WANs • SINET3 (JP), SURFnet (NL), etc. • Realistic to perform parallel computation using multiple clusters connected via a WAN • InTrigger (JP), DAS-3 (NL), Grid5000 (FR), etc. • Need for a “smart” communication library that automatically adapts to the environment and offers: • Connectivity • Deal with FWs and NATs • Scalability • Need to limit the number of wide-area connections in • order to avoid resource allocation limits • Communication performance • “Establishing connections whenever possible” can result in • poor communication performance • Meanwhile, mindlessly limiting the number of connections • and forwarding messages does not work either • Experimental results • Experimental setup • 1,264 cores in 11 clusters • Connectiity and scalability • Brought up a process on each of the 1,264 cores • Socket library • Connections could not be established between the two • clusters behind NAT gateways • Reached the limit on the number of file descriptors that • each process could use • Reached the limit on the number of connections that • could be handled by a single NAT gateway • SSOCK (w/ 1 ssockd per cluster) • Connections could be established between all 1,264 • processes • Simultaneous connects • Brought up the same number of • processes in each cluster • Every process tried to connect to • every other process simultaneously • using non-blocking connects • The Socket library perfomed poorly • because many SYN packets were dropped • Point-to-point performance Design and implementation Main features Socket API No need to modify existing applications/middleware Overlay network FW/NAT traversal (any node pair can connect to ea. other) High scalability (all node pairs can connect to ea. other) High performance with minimal configuration Optimizations based on latency measurements Only necessary configuration is on endpoint Bootserv The means by which libssocks and ssockds learn of each other (libssocks and ssockds must be configured with the endpoint of bootserv) Libssock Library component of SSOCK From the application’s point of view, calling libssock’s connect() establishes a directconnection Ssockd Daemon component of SSOCK Connects to other ssockds to construct an overlay (uses the overlay construction algorithm of MC-MPI (Saito et al. ‘07)) Forwards data (libssock -> ssockd, ssockd -> ssockd) Bootserv (One per system) Overlay Network Libssock Virtual connection Ssockd (≥1 per LAN) Real connection Intra-cluster (kototoi) ping-pong performance Inter-cluster (hongo-okubo) ping-pong performance Future work A study on the effects of bringing up multiple ssockds per LAN