VLBI & Protocols vlbi_udp Multiple Flow Tests

1. VLBI & Protocolsvlbi_udp Multiple Flow Tests Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

2. Setting up the Tests • Thank you to the people at all the sites for your help and making facilities available • Including: • Onsala • JIVE • Torun and Pioner Poznan • GARR Bologna • Jodrell and Manchester Network Group • Dante ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

3. vlbi_udp: Multi-site Streams Gbit link Chalmers University of Technology, Gothenburg Metsähovi OnsalaSweden Jodrell BankUK Gbit link TorunPoland DedicatedGbit link Dwingeloo DWDM link MedicinaItaly ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

4. vlbi_udp: UDP on the WAN • iGrid2002 monolithic code • Convert to use pthreads • control • Data input • Data output • Code branch for Simon’s file transfer tests • Work on vlbi_recv: • Output thread polled for data in the ring buffer – burned CPU • Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput • Output thread uses sched_yield() when no work to do • Multi-flow Network performance – being set up Nov/Dec06 • 3 Sites to JIVE: manc UKLight; Manc production; Bologna GEANT PoP • Measure: throughput, packet loss, re-ordering, 1-way delay • To do • Add code for:MarkV card and PCEVN interface ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

5. vlbi_udp: Some of the Problems • JIVE made Huygens, mark524 (.54) and mark620 (.59) available • Within minutes of Arpad leaving the Alteon NIC of mark524 lost the data network! • OK used mark623 (.62) – faster CPU • Firewalls needed to allow vlbi_udp ports • Aarrgg (!!!) Huygens is SUZE Linux • Routing – well this ALWAYS needs to be fixed !!! • AMD Opteron did not like sched_getaffinity() sched_setaffinity() • Comment out this bit • udpmon flows Onsala to JIVE look good • udpmon flows JIVE mark623 to Onsala & Manc UKL don’t work • Firewall down stops after 77 udpmon loops • Firewall up udpmon cant communicate with Onsala • CPU load issues on the markV systems • Don’t seem to be able to keep up with receiveing UDP flow AND emptying the ring buffer • Torun PC / Link lost during the tests ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

6. 2.2GHz AMD Athlon • Udpmon OK but single thread • Vlbi_udp uses recvmsg() & iovectors • No not quite !!! • Asked for a flow of 600Mbit/s (-w20 us) got ~300 Mbit/s (-w58 us) • 1.8% packet loss • 1.1% re-ordered (mainly by 1 or 2 packets) • Onsala gig7 (Academic internet to Ams then UKLight)Onsala  gig8 (all Academic internet) • ~4.5% lost and ~3.5% re-ordered vlbi_udp Onsala  mark623 • It worked ! ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

7. Multiple vlbi_udp Flows • Gig7  Huygens UKLight 15 us spacing 800 Mbit/s • Gig8  mark623 Academic Internet 20 us spacing 600 Mbit/s • Bologna  mark620 Academic Internet 30 us spacing 400 Mbit/s SJ5 Access link SURFnet Access link GARR Access link ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

8. Multiple vlbi_udp Flows • Gig7  Huygens UKLight 15 us spacing • 816 Mbit/s del <1Mbit/sstep 1 Mbit/s • Zero packet loss • Zero re-ordering • Gig8  mark623 Academic Internet 20 us spacing • 612 Mbit/s • 0.6 falling to 0.05% packet loss • 0.02 % re-ordering • Bologna  mark620 Academic Internet 30 us spacing • 396 Mbit/s • 0.02 % packet loss • 0 % re-ordering ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

9. Any Questions? ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester

10. Backup Slides ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester