White Rabbit Sub-Nanosecond timing over Ethernet

1. White RabbitSub-Nanosecond timing over Ethernet H.Z. Peek ... ... on behalf of the White Rabbit collaboration “Oh dear! Oh dear! I shall be too late!” 1) 1) Alice in Wonderland, LewisCarroll (1865)

2. Outline • What is White Rabbit • Network • Building blocks • Measurement results • Applications • Conclusion

3. What is White Rabbit • Networking and timing • Based on well-known technologies / standards • Open Hardware and Open Software • International collaboration • Main features: • Transparent, high-accuracy time distribution, • Low-latency, deterministic data delivery, • Designed for high reliability, • Plug & play.

4. Timing over EthernetHistory • 1985 • NTP (RFC 1305) sub milli-second • 2002 • PTP (IEEE 1588) sub micro-second • 2012 • White Rabbit (?) sub nano-second

5. White Rabbit timing distribution • Synchronization with sub-ns accuracy over fiber • A combination of: • Synchronous Ethernet (SyncE) syntonization • Digital Dual-Mixer Time Difference (DDMTD) phase detection (j) • Precision Timing Protocol (PTP) synchronization

6. Timing distribution Master Slave Reference Clock j SFP SFP Ordinary serial data communication channels Ordinary serial data communication channels! Tx Rx Tx Rx High Precision Low jitter High Precision Low jitter Slave j SFP SFP Rx Rx Tx Tx • Averaging all clock edges • “mean phase” • Syntonization • “The adjustment of two electronic circuits or devices in terms of frequency”

7. PTP (IEEE1588) Master Slave t1 t1 t2 t4 t4 SFP SFP t3 j j Rx Tx Ref Clk Cnt Cnt =0 t1 t2 t4 t3 toffset Rx Tx t1 Sync message Time Stamp t2(= Cnt + SlaveBitSlide) Data (Packets) Follow_Up Message (t1) Clock t1 t2 t3 t1 t2 Delay_Req Message t4 (= Cnt + MasterBitSlide + j) Delay_Response Message (t4) t4 t3 t1 t2 t2 –t1=offset + MSdelay t4 –t3=-offset + SMdelay MSdelay=SMdelay Master Clock Time Slave Clock Time

8. White Rabbit network

9. White Rabbit Switch V2 • Central element of White Rabbit network • Fully custom design, designed from scratch • 10 SFP ports (1000Base-LX) • Capable of driving long distance Single Mode fiber

10. White Rabbit Node • Simple PCIe FMC Carrier (SPEC) • Currently available http://www.ohwr.org/projects/spec/wiki

11. White Rabbit Switch V3 • 2 uplink ports, 16 downlink ports • Hardware just assembled. • Hardware & Software currently being tested. • Production expected Q1-2012 http://www.ohwr.org/attachments/741/White_Rabbit_Technical_Spec_05.pdf

12. Measurement test setup Hot-air gun demo: http://www.youtube.com/watch?v=ZSRQEExbdq8

13. Measurement results

14. Applications • Digital to Time Converter (DTC) = Alarm Clock • Time to Digital Converter (TDC) = Time stamping

15. Conclusions • White Rabbit enables measurement and control applications which are using distributed system technologies. • Such applications may be spread over large distances. • Data transmission delay changes are continuously measured and compensated. • System timing of White Rabbit nodes aresynchronized with high precision.

16. White Rabbita multi-laboratory, multi-company effort Coming Soon! Thank you

18. How do we know the time offset between Master and Slave? Clock & Data coded into one stream DC-Balance Special code-groups / Word Alignment • Toffset = Total delay(Master>Slave>Master) / 2 • Can we measure propagation delay using existing serial communication channels? Serial Communication Coding Properties: 1 2 3

19. Measure propagation delay using FPGA SerDes technology Regenerate system clock at the receiver 1 Using a barrel shifter for word alignment delay known with bit clock resolution 3 Xilinx ML507 Board Receiver x 20 System Clock Transmitter Lattice LFSCM25 Xilinx Virtex-5 SerDes SerDes SFP SFP 100 Km Fiber Lattice SC PCI Expressx1 Evaluation Board System Clock LEDs Start Stop

20. Time offset measurement test setup 10 Km fiber Loopback the recovered clock with a Voltage ControledXtalOscillator Clock Loopback (DPLL) Stop Stop DAC Start Master VCXO Slave

21. Resynchronization + Barrel shifting action 011101011000001010110111010110000010101101110101 3 1 0 RxRecClk BitSlide(4:0) Unit Interval (UI) 0 0001 = 1 0 0000 = 0 0 0011 = 3 Start/Stop delay Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI • Details : VLVnT09, October 15, 2009 in Athens • Presentation: http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt • Paper: http://dx.doi.org/10.1016/j.nima.2010.04.126

22. Measure time offset • Bidirectional + Loopback the recovered clock • 1.25 Gbps (IEEE802.3 1000BASE-X = Gigabit Ethernet) • Time offset is determined by: Tx Rx Master Slave Reference Clk RxUsrClk TxUsrClk SFP SFP Start Stop RxUsrClk TxUsrClk j Rx Tx Count toffset 1.25 Gpbs Fine time Coarse time 20 x #bit clocks • # of system clocks • # bit clocks (i.e. barrel shifts) • Phase between Master node Tx and Rx clock 800 ps 10 .. 100 ps For details please see Technical Report “ETR2010-01”: http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf

23. Time offset and fibre dispersion Master time Slave time toffset l1 tpdl1 dmTx dsRx l2 dmRx tdisp tpdl1 dsTx (tdisp = 15150 ps over 10.7 km => D(l) = l416 ps/km)