1 / 42

Computer Networks & Digital Lab project.

Computer Networks & Digital Lab project. In cooperation with Mellanox Technologies Ltd. Guided by: Crupnicoff Diego & Gurewitz Omer. Students: Cohen Erez , Gindi Nimrod & Krig Amit. Thanks !!!!!. תודות

oki
Download Presentation

Computer Networks & Digital Lab project.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computer Networks & Digital Lab project. In cooperation with Mellanox Technologies Ltd. Guided by: Crupnicoff Diego & Gurewitz Omer. Students: Cohen Erez , Gindi Nimrod & Krig Amit.

  2. Thanks !!!!! תודות ברצוננו להודות לכל האנשים שבלעדיהם לא היה פרויקט זה יוצא לפועל ובעזרת הדרכתם וסיועם הגענו עד כאן : יורם אור-חן (מעבדה לרשתות מחשבים) – על התמיכה והלבביות לאורך כל הדרך . אלי שושן ( מעבדה ספרתית ) – על ההכוונה להתמקד בכיוונים ובנושאים המעניינים באמת . יורם יחיה (מעבדה לרשתות מחשבים) – על העזרה הרבה בתיאום כל שלבי הפרוייקט . עומר גורביץ ודייגו קופרניקוף– על ההדרכה, ההנחיה והעזרה הרבה בכל שלבי הפרוייקט . ברצוננו להודות באופו מיוחד לשי כהן ,סמנכ"ל תפעול בחברת מלאנוקס טכנולוגיות, על הזמן והמשאבים הרבים (והיקרים) שאפשר לנו על מנת שנוכל להשלים את הפרוייקט

  3. AGENDA • Projects Motivation . • Projects objective . • InfiniBand short preview (10 min). • Projects System overview : • Hardware Project description (10 min). • Software Project description (10 min). • Projects Demonstration Tools and process explanation (10 min) • Agilent IB tracer usage demonstration . • Specific InfiniBand Terms to be used . • Specific patterns to be sent explanation . • Mellanox InfiniBand Development tool to be used .

  4. AGENDA – Cont’ • Projects Demonstration (20 min) • Installation Process of the System in ‘Virgin’ system . • Sending Various InfiniBand Packets from the • Ibgenerator to Agilent’s IB tracer. • Demonstration of full InfiniBand system flow • ( Sending InfiniBand Packets from the Ibgenerator • through Agilent’s IB tracer to Mellanox Infinibridge)

  5. Projects Motivation The InfiniBand world is still in his early days – and as such it requires all sorts of supporting equipment . As we will show you, analyzing equipment already existed when we started thinking on this project , what we could not locate is dedicated device which purpose is transmitting InfiniBand packets ( and we still can not find such device in the market !!!! – although several companies declared almost a year ago that they are working on this issue ). That , and our desire to learn more about the InfiniBand standard , brought us to the idea of this project .

  6. Projects objective • Learning and understanding deeply, the new InfiniBand™ Architecture. • Implementation of packet generator / transmitter focusing on InfiniBand™ protocol (release 1.0a).

  7. InfiniBand short preview(Refreshing our memory) In order to refresh our memory without saying what was already presented in former presentations of these projects we will review here the InfiniBand architecture by going over it in ‘title’ level . The InfiniBand Standard

  8. End Node End Node End Node End Node Switch Switch Switch End Node Switch End Node End Node End Node End Node End Node InfiniBand Overview Switch fabric Concurrent data transfer No mechanical constrains Performance High bandwidth (2.5 to 30Gbit/sec) Fast I/O access by application QOS Packet-granularity b/w allocation Packet-granularity latency decision

  9. Target Channel Adapters for Specialized Subsystems Host Channel Adapters for computing platforms Subnets consist of Links & Switches Routers enable inter-subnet communications while providing subnet isolation The InfiniBand Architecture Model Target TCA IB Link CPU Switch TCA Target MemCntlr HCA IB Link IB Link Host Interconnect CPU IB Link SysMem Router Router Network or IB IB Link

  10. InfiniBand System Architecture Decouples CPU and I/O OS-independent. Scope – from “PCI” to WAN Same ‘look and feel’ for local or remote nodes. Wide cost/performance range for implementations.

  11. InfiniBand Overview cont’ Reliability Reliable transport service in HW Automatic path migration in HW (fault tolerance) Scalability/flexibility Up to 64K nodes in subnet, up to 2128 nodes in network Multiple link width/trace (Cu, Fiber) Auto-negotiation of link width and transfer rate

  12. InfiniBand architecture. InfiniBand is a Layered architecture. Similar to an IP network, each layer supply services to the higher layer. An IB packet is build from headers added by each layer. The layers responsibilities are: Insure correct routing of a packet. Insure correct data. Insure QOS. And more ….

  13. Application Application Upper Layer protocols Upper Layer protocols Transport Layer Transport Layer Network Layer Network Layer Packet relay Packet relay Link Layer Link Layer Link Link Link Link Packet relay Physical Layer Physical Layer PHY PHY PHY PHY PHY PHY IB End node IB Switch IB Router Legacy Router InfiniBand architecture cont’.

  14. InfiniBand architecture features. Multiple transport services Reliable and unreliable Connected and datagram Enables memory exposure to remote node RDMA-read and RDMA-write Enables network partitioning Partition key and routing programming Enables user-level access to I/O Adapter validates access rights Adapter translates memory address

  15. InfiniBand architecture features cont’. Enables dynamic load balancing Within end-node or in the fabric Multiple levels of QOS decisions

  16. Hardware description

  17. Hardware system description JTAG connector I2C connector IB port 2.5 GB/sec Power unit Oscillator Agilent’s SerDes 10  1 JTAG I2C interface Mictor Xilinx xcv400e FPGA (125 MHz DDR) transmitter PCI connector for power and reset

  18. System interface • The board is a standard PCI form factor. • The board contain the following interfaces: • I2C – software interface: This interface is used to load data (10 bit for each byte, by InfiniBand spec), commands and to control the Ibgenerator. We have used this interface because it’s a very simple and cheap solution. • InfiniBand connector – This is the interface to the InfiniBand fabric. We use 1x connector according to the InfiniBand spec.

  19. System interface • JTAG – This is a common interface to connect to FPGA’s. In our board we used the JTAG interface to program the Xilinx FPGA and to debug the Verilog code. • PCI interface – We didn’t use a “real” PCI interface. We have used this interface only to get power from a PC and to get reset signal.

  20. System flow • Data is received from the I2C interface to the FPGA unit. • Data can be written to 3 different location in the FPGA: • Data array – This is a 256X32 bit array, it’s holding the data to be transmitted to the IB fabric. • Command array – This is a 32X32 bit array, it’s holding in each row (each 32 bit) a command to execute. Each command hold the following information: • Pointer to the start address in the data array. • Pointer to the end address in the data array.

  21. System flow con’t • Times to transmit this data. • Pointer to the next command to execute. • Status register – This is the “go” command of the system. To start a transmission we should write the address of the command to be execute and the FPGA will start sending this data.

  22. System flow con’t – Send TS1 Status Command Array Data Array To SerDes TS1 Data

  23. System flow con’t • Once the FPGA starts to work on a command, it will send 10 bits of data in a rate of 125 MHz DDR to the SerDes. It will also send TBC (Transmit Byte Clock) signal. • The SerDes will send this data in a rate of 2.5 Gbit/sec in a differential lines.

  24. Signal integrity

  25. Software description

  26. SW description The software project contains 3 major libraries MPGA library I2C library IB generator library

  27. MPGA library The MPGA(Management Packet Generator Analyzer) library provides software for generation and analysis of InfiniBand packets. This library is especially important when using the IB generator for sending and receiving all kinds of InfiniBand packets. Refer to mpga.h and the related files packet_append.h, packet_utilities and ib_opcodes.h for further details.

  28. MPGA structure • This library has 3 hierarchical levels: • The first level is the upper level containing the user interface functions. • The second level of the library is in charge of the InfiniBand packet generation building blocks. • The last level in MPGA is a hidden part of the library containing all of the internal functions used only by this library.

  29. Hק'Headers in Level 1 Headers in level 2 Headers in level 3 MPGA structure User level Building blocks Internal functions

  30. Application Upper Layer protocols Transport Layer Network Layer Link Layer Physical Layer IB End node PAYLOAD BTH PAYLOAD MPGA cont’ - Generating packet flow Raw data to send Building a transport packet

  31. Application Upper Layer protocols Transport Layer Network Layer Link Layer Physical Layer IB End node LRH BTH PAYLOAD VCRC ICRC MPGA cont’ - Generating packet flow Building a Link layer packet .

  32. ???? LRH ???? ICRC VCRC LRH BTH DETHH Payload ICRC VCRC Payload pointer Packet size MPGA cont’ - Analyzing packet flow Link layer of the incoming packet. Analyzing the transport layer of the incoming packet.

  33. Mpga cont’ – special features • Endian proof • ICRC/VCRC calculation (Cyclic Redundancy Code) • Error generation

  34. IB generator library • This library is the major driver for the Ibgenerator. • The library uses the I2C library to communicate with the Ibgenerator through the I2C master card (CALIBRE). • All of the basic structures of the Ibgenerator Lib is based on the FPGA I2C interface defined in InfiniBand project HW section.

  35. IB generator library The main features are : 8 to 10 Link phy section Sending TS1/TS2 (Tranning sequenc one/two). Logical link Sending Flow controls init and normal state. Packets Sending a regular IB packet. sending big buffers 4K MTU.

  36. I2C library The I2C library is based on the API of CALIBRE Company The library provides I2C interface to the IB generator board.

  37. Demonstration preview

  38. Demonstration preview • IB tracer • Init the system. • Ibgenerator device installation. • Caliber card installation. • Installation of Ibgenerator software package. • Connect to the Agilent IB tracer. • Start the Ibgenerator GUI: • Send TS1. • Send TS2. • Send idle data. • Send credit packet (init and normal state).

  39. Demonstration preview • Send the IB golden packet. • Send RDMA write. • Send Ack. • Send errors: • ICRC error • VCRC error • Packet length error. • Send big packet (4k MTU).

  40. Demonstration preview • Connect to Mellanox Infinibridge device through Agilent IB tracer. • Send the following packets: • TS1 • TS2. • Idle data. • Credit packet (init and normal state). • Show physical and logical link state. • Send data packets to Mellanox device.

  41. Demonstration preview

  42. Demonstration

More Related