Implementing White Rabbit in your design

1. Implementing White Rabbit in your design Peter Jansweijer

2. Outline: • My background: • KM3NeT / FPGA / Serial Communication • White Rabbit introduction • Technology / Implementations • WR PTP Core in detail • How to implement WR in your design

3. KM3NeT Artist impression & detection method 640 strings 18 DOM/string 11.520 DOMs 357.120 PMTs ~ 860m Cherenkov m n 100m Volume: ~5 km3 11.520 DOMs in the deep sea at 3-5 km depth

4. The very beginning… “Measuring propagation delay over a coded serial communication channel using FPGA.” 011101011000001010110111010110000010101101110101 3 1 0 RxRecClk BitSlide(4:0) 0 0001 = 1 0 0000 = 0 0 0011 = 3 Start/Stop delay Presented at the VLVnT09, October 15, 2009 in Athens Elsevier Nuclear Instruments and Methods: http://dx.doi.org/10.1016/j.nima.2010.04.126FPGA transceivers propagation delay study: http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf

5. White Rabbit Introduction

6. Layer 1 Syntonization • Syntonization = “The adjustment of two electronic circuits or devices in terms of frequency” • All network devices use the same physical layer clock. • Clock is encoded in the Ethernet carrier and recovered by the receiver chip. • Phase detection allows sub-ns delay measurement

7. Precision Time Protocol (IEEE 1588) • Frame-based synchronization protocol. • Synchronizes local clock with master clock. • Link delay evaluated by measuring and exchanging frames with tx/rx timestamps

8. Digital Dual Mixer Time Difference (DDMTD)

9. Using the DDMTD as phase detector and phase shifter

10. White Rabbit: an extension of Ethernet • Two separate services • (enhancements to Ethernet) provided by WR: • Synchronization: accuracy better than 1 ns precision (tens of pssdev skew max) • Deterministic, reliable and low-latency Control Data delivery

11. Standardization • Ongoing standardization effort to include WR into PTPv3 • Extension to PTP (IEEE1588) – defined as PTP profile • Compatible with “standard” PTP • According to ISPCS Plug Fest results… • “… White Rabbit is the most accurate PTP implementation in the world!”

12. White Rabbit Switch http://www.ohwr.org/projects/white-rabbit/wiki/Switch • Central element of WR network • 18 1000BASE-BX10 ports • Open design (H/W and S/W) • Commercially available

13. White Rabbit Node example 1: SPEC http://www.ohwr.org/projects/spec/wiki • FMC-based Hardware Kit • All carrier cards are equipped with a White Rabbit port. • Mezzanines can use the accurate clock signal and “TAI” (synchronous sampling clock, trigger, time tag, …). • Starting kit: http://www.ohwr.org/projects/wr-starting-kit/wiki

14. Other White Rabbit Node examples: CUTE-WR (LHAASO) Central Logic Board (KM3NeT) CRIO-WR (CERN) SVEC (CERN) SPEXI (CERN)

15. White Rabbit PTP Core http://www.ohwr.org/projects/wr-cores/wiki/Wrpc_core T-Sense / Unique ID

16. White Rabbit PTP Core in detail

17. White Rabbit PTP Core in detail T-Sense / Unique ID PHY: Interface to the physical network, with deterministic phase relationship between gigabit clock and system clock

18. White Rabbit PTP Core in detail T-Sense / Unique ID Endpoint: Ethernet MAC, with precision time stamping capabilities of in- and out-bound Ethernet frames

19. White Rabbit PTP Core in detail “Normal” packets Redirector: Redirects all PTP Ethernet packets to the Mini-NIC T-Sense / Unique ID Receives the payload of the PTP packets and stores payload data in memory

20. White Rabbit PTP Core in detail Embedded Processor system (LatticeMico32 + Bus interface + Memory) running a PTP-daemon T-Sense / Unique ID

21. White Rabbit PTP Core in detail Now the CPU has “knowledge” of time. It can adjust the local time and update the proper TAI time and control the Pulse Per Second (PPS) generation T-Sense / Unique ID

22. White Rabbit PTP Core in detail The local oscillator that is used for TX is locked onto the RX receive clock. A PLL is made using Digital Dual Mixer Time Difference (DDMTD) technique and software on the LatticeMico32 (hence Soft-PLL) to tune the oscillator. DDMTD allows for < 1 ns phase adjustment under control of the LatticeMico32 => a result of calculations of the round trip delay (Timing Synchro) T-Sense / Unique ID

23. White Rabbit PTP Core in detail T-Sense / Unique ID 1-Wire is used to fetch a unique MAC addressUART is used for debug only

24. How to implement WR in your design

25. How to implement White Rabbit in your design FPGA DATA (1 Gbps) SFP PHY Unique-ID (MAC-addr) (optional) GTREFCLK SPI Your own stuff VCXO DAC 125 MHz CLK125M_PLLREF REF clock generator CLK_DMTD CLK20M_VCXO VCXO DAC PLL 20+ MHz DMTD clock generator Timing (TAI[ns],REFCLK) http://www.ohwr.org/projects/white-rabbit/wiki/WRReferenceDesign

26. White Rabbit & CLBv2(KM3NeT) FPGA IP/UDP Packet Buffer Stream Selector (IPMUX) 31 TDCs Start Time Slice UTC & Offset counter since Fifo TDC0 Time Slice Start RxPort 1 RxPacket Buffer 64KB 31 PMTs RxPort 2 Rx Stream Select Rx_mac2buf Rx_buf2data Fifo TDC 30 Flags RxPort_m Management & Control S State Machine Management & Config. Pause Frame Fifo ADC Hydrophone TxPacket Buffer 32KB TxPort 1 TxPort 2 Tx Stream Select Tx_pkt2mac Tx_data2buf Management & Control Flags TxPort_m S S Nano Beacon M M M S M M M M M Debug LEDs WB Crossbar (1x7) WB Crossbar (3x2) S S S S S M M M I2C GPIO I2C 2nd CPU LM32 SPI UART Xilinx Kintex-7 S MEM S M S M Data UTC time & Clock (PPS, 125 MHz) Compass Debug RS232 Temp Tilt SPI Flash Control Point to Point interconnection Wishbone bus

27. White Rabbit Group: http://www.ohwr.org/projects/white-rabbit/wiki Picture taken @ 4th WR workshop (2011), GSI Darmstadt • WR Users http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers Many slides in this presentation orginate from the WR group: Thanks!

28. Thank you