1 / 22

GETB – a Gigabit Ethernet Application Platform Its Use in the ATLAS TDAQ Network

GETB – a Gigabit Ethernet Application Platform Its Use in the ATLAS TDAQ Network. Matei Ciobotaru 1,4 Micheal LeVine 3. Stefan Stancu 2 Brian Martin 1. 1 CERN 2 University of California, Irvine 3 Brookhaven National Laboratory 4 “Politehnica” University of Bucharest. Outline. Motivation

karis
Download Presentation

GETB – a Gigabit Ethernet Application Platform Its Use in the ATLAS TDAQ Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GETB – a Gigabit Ethernet Application PlatformIts Use in the ATLAS TDAQ Network Matei Ciobotaru1,4 Micheal LeVine3 Stefan Stancu2 Brian Martin1 1CERN 2University of California, Irvine 3Brookhaven National Laboratory 4“Politehnica” University of Bucharest

  2. Outline • Motivation • The ATLAS Network and its requirements • The Gigabit Ethernet Testbed (GETB) • Hardware: FPGA-based PCI card • Firmware: Handel-C language • Software: Python-based control system • Applications and sample results • The Network Tester • The ATLAS ROB Emulator Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  3. ATLAS TDAQ Network • ATLAS TDAQ Network • Gigabit Ethernet • Layer 2 only • Only switches • ~ 1000 end-nodes • Sustained rates • 5 Gb/s on average • Requirements • Minimal packet loss • Minimal latency • High Performance Switches • Try before you buy Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  4. Motivation • What we needed • A system to evaluate switches for the ATLAS TDAQ network • A development platform for other networking applications • The market offered the commercial testers • Examples: Ixia, Smartbits • Built for standard RFC benchmarks • Not easily customizable • Too many features we don’t need • Expensive • Finally a new platform was created Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  5. GETB Platform – Hardware • GETB = Gigabit Ethernet Testbed • PCI card with an FPGA and 2 Ethernet ports Configuration Flash GPS Clock Input SDRAM 2 x 64Mb 2 x Gigabit Ethernet SRAM 2 x 512Kb Altera Stratix EP1S25 FPGA 3.3V PCI Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  6. GETB Platform – Firmware • 90% Handel-C, 10% VHDL • FPGA advantage  massive parallelism • Code shared among projects • Commercial IP cores (IP = Intellectual Property) • Gigabit Ethernet MAC , PCI Controller • Logic utilization ~ 85 - 90% • Single FPGA controls 2 ports Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  7. GETB Platform – Control • Based on the Python scripting language • Distributed system – 64 cards in 15 hosts (128 ports in total) • Common control infrastructure for all applications Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  8. Connect Configure Run Analyze Report Python Test Description (configure tester and DUT) Network Tester – Features • Traffic Generator and Measurement System • 1 Gigabit/sec for all packet sizes (send and receive simultaneously) • User-defined traffic patterns • Deterministic (packet descriptors) • Client-Server (request-response) • Real-time measurements • Throughput, packet loss, latency and inter-packet time (IPG) • Histograms for latency, IPG, packet size • ATLAS Specific • Requirements and test procedures documented • Pre-defined scripts to run all tests Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  9. Network Tester – ATLAS Traffic • Performance for ATLAS Traffic (funnel-shaped traffic pattern) • Depends on the number of buffers • Determine the maximum load the clients can receive without loss Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  10. ATLAS ROB Emulator – Context • ROB = Read Out Buffer • Intelligent event fragment buffering card • Data from detectors enter on fiber links • TDAQ requests event fragments from the ROB to do the event filtering • Connected to TDAQ via GE ports or PCI • 10% of total system needed for tests • Production ROB card not available in sufficient quantity • Need a ROB Emulator to test candidate switches with the TDAQ software applications Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  11. ATLAS ROB Emulator – Features • One GETB  2 x ROB emulators • Fully compliant with the protocols used by TDAQ applications • Seamless integration in a TDAQ testbed • Real-time access to relevant statistics • Ready to be installed in the pre-series testbed Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  12. Summary • GETB = Flexible development platform • FPGA-based • Gigabit line-speed performance • Applications • Network Tester • Test switches according to ATLAS requirements • ATLAS ROB Emulator • Can be integrated in the full ATLAS TDAQ infrastructure • Work ongoing on other applications (Network Emulator) • The GETB is currently being used to evaluate network equipment for the ATLAS TDAQ network Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  13. End of Presentation • Backup slides Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  14. Network Tester – Packet Generation • Client Server • ATLAS-like request-reply traffic • Clients send data request packets to servers; Servers send back replies • Clients use a token-based system to control network load • Independent Generators • Packet descriptors • Modify MAC addresses, packet size, inter-packet time, IP headers, etc. • Wide range of traffic patterns • Ethernet and IP packets, VLANs Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  15. Network Tester – Full-Mesh Traffic • Fully-meshed traffic performance Switch A Switch B Maximum loss rate ~ 1% Maximum loss rate > 20% Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  16. Switch Quality of Service • 8 transmitters, each sending traffic with a different priority • The switch uses Weighted Round Robin Scheduling • Result obtained using data collected by one GETB tester port Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  17. Buffer Sizes • Custom method for measuring buffer sizes • 3 devices  3 different memory management techniques • Size of buffers important for the ATLAS traffic pattern Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  18. MAC Address Table • Switch advertises 16000 entries in the MAC table • Measurement reveals problems for more than 5000 addresses Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  19. Latency Measurement • Latency for a fully meshed traffic pattern Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  20. Sample Histogram • Example of histogram of Inter-packet Time for a Negative Exponential distribution • Features • Packets can be classified according to source or VLAN ID • Latency, IPG and packet size can be histogrammed • User defined resolution and histogram window (start offset, length) Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  21. SequentialBlock ParallelBlock // 3 Clock Cycles { a=1; b=2; c=3; } // 1 Clock Cycle par{ a=1; b=2; c=3; } Handel-C • Hardware description language • Like VHDL, but with syntax similar to C • The result of the compilation is the description of an electrical circuit • Contains built-in parallel constructs • Special features • Arbitrary widths on variables • Enhanced bit manipulation operators • Simple timing model • Each assignment is one clock cycle • Support for hardware constructs • Multiple clock domains, on-chip memories, external interfaces • Synchronization primitives: channels, semaphores Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

  22. classStack: "A well-known data structure" def __init__(self): # constructor self.items = [] def push(self, x): self.items.append(x) def pop(self): x = self.items[-1] del self.items[-1] return x def empty(self): return len(self.items) == 0 Python Language • Features • Object oriented • Easy to learn, read, use • Extremely portable • Extensible (new modules) • What is it used for? • rapid prototyping • scientific applications • extension language • web programming Matei Ciobotaru – The GETB Platform – Real Time 2005, Stockholm

More Related