Design and Characterization of TMD-MPI Ethernet Bridge

1. Kevin Lam Professor Paul Chow Design and Characterization of TMD-MPI Ethernet Bridge

2. Background • In embedded development, Field Programmable Gate Arrays (FPGAs) allow mixing of hard and soft elements • Issue: Communication between components • MPI is a standard for inter-process communication in software parallel programming • TMD-MPI implements a hardware interface for MPI • Unified method of communication between all components • Abstracts the communication medium

3. TMD-MPI Application FPGA Embedded CPU Hardware Engines

4. Research Objective • Attempt to create a module that routes MPI messages through on-board Ethernet hardware • Design Objectives: • Reliability • High Bandwidth • Low Latency

5. Multi-Board TMD-MPI system (your desk!)

6. Design Specification • Behaviour • FSL interface to on-board network (via NetIF module) • Transparent inter-board communication via Ethernet • FIFO behaviour • Assumptions • Only two systems connected, by short Ethernet cable • Reliable, in-order delivery of packets

7. Design Specification FPGA FPGA Logical FSL FSL-Ethernet FSL-Ethernet FSL FSL

8. Design Specification Packet Format • Basic Ethernet framing • Source/Destination MAC address • Type code – set to an unused value • CRC checksum performed by Ethernet hardware • Length header to account for smaller-than-minimum packets • Set to 0 if data is longer than minimum packet length • Followed by data payload, up to maximum Ethernet frame length • No handshake protocol implemented

9. Implementation • Base design around Xilinx echo example • First send/receive data independently • Test using PC as second ‘board’ • Develop FSL interface, test with MicroBlaze • Merge FSL interface to Ethernet module • Add protocol headers, implement data packing

10. Testing • Initial testing done using MicroBlaze processors • High speed testing done using custom cores Board 1 Board 2 MicroBlaze 1 FSL-Enet FSL-Enet MicroBlaze 2 UART UART PC

11. Design Evaluation • Reliability testing using two dedicated tester cores • Transmitting sequence numbers, checking for mismatch/out of order Board 1 Board 2 Test Core 1 FSL-Enet FSL-Enet Test Core 2 GPIO GPIO MicroBlaze MicroBlaze UART UART PC

12. Design Evaluation Bandwidth • Uses 2 dedicated hardware cores • Core A writes data to the FSL-Ethernet core whenever the FSL is not full • Core B reads data from the FSL-Ethernet core whenever the FSL has data • Core A counts the number of clock cycles elapsed to write a certain amount of data to the FSL • Result read by MicroBlaze and displayed via UART • Measured constant 962.5 Mbit/s data throughput rate

13. Design Evaluation Latency • Uses 2 dedicated hardware cores • Core A sends 1 word of data to Core B via FSL-Ethernet • Core B replies with 1 word • Core A measures the clock cycles elapsed between sending its data and receiving the response • Result read by MicroBlaze and displayed via UART • Several trials yield 598-599 cycle round trip (125 MHz) • One-way latency is approximately 300 cycles or 2.4 microseconds

14. Conclusions • Bandwidth sufficient for VGA video streaming • Latency may be an issue for applications that pass data back and forth frequently • No handshaking protocol • Receiver in streaming applications must not be slower than the sender • Messages cannot be sent until both boards are powered on and connected • Future versions should attempt to implement handshaking • Affect complexity and bandwidth • Much more robust system

15. Acknowledgements Professor Paul Chow Vince Mirian Sami Sadaka Tony Zhou