1 / 30

The TrainBuilder ATCA Data Acquisition Board for the European-XFEL

The TrainBuilder ATCA Data Acquisition Board for the European-XFEL John Coughlan, Chris Day, James Edwards, Ed Freeman, Senerath Galagedera and Rob Halsall Science & Technology Facilities Council Rutherford Appleton Laboratory Oxfordshire, United Kingdom E-mail : john.coughlan@stfc.ac.uk.

nola
Download Presentation

The TrainBuilder ATCA Data Acquisition Board for the European-XFEL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The TrainBuilder ATCA Data Acquisition Board • for the European-XFEL • John Coughlan, Chris Day, James Edwards, Ed Freeman, • Senerath Galagedera and Rob Halsall • Science & Technology Facilities Council • Rutherford Appleton Laboratory • Oxfordshire, United Kingdom • E-mail: john.coughlan@stfc.ac.uk Presented by John Coughlan. TWEPP Oxford. September 2012.

  2. Topics • European-XFEL DAQ • TrainBuilder demonstrator board • FPGA Firmware • System Applications Presented by John Coughlan. TWEPP Oxford. September 2012.

  3. XFEL DAQ Data Flow Common Systems XFEL DAQ 10G 10G 10G Detector 2D Camera Train Builder 10G Switch 10G servers PC Layer • 3 Large 2D Pixel Detectors • 1M Pixel, 2B/Pixel, 512 Images @10Hz (X-Ray pulse Train) • 10 Gbyte per second per Megapixel rate • 10G Ethernet using UDP protocol • Modularity : 16 x 10G Links off detector per Megapixel • Scalable • Full Data Rate to PC Layer Presented by John Coughlan. TWEPP Oxford. September 2012.

  4. Image Train Building Full Images 2 MB Partial Images 128 KB N= 16 10Gbps 10Gbps Train Builder FEMs Train Builder PC SWITCH PC N x 512 N x 512 FEMs PC SWITCH PC Train Builder 10G servers PC Layer Deep Buffers Switch 10G Links Detector 2D Camera Detector Specific Common XFEL DAQ Presented by John Coughlan. TWEPP Oxford. September 2012.

  5. TrainBuilder Demonstrator • Train Builder Demonstrator • ATCA form factor • 5 x Xilinx V5FXT100 FPGAs • Analogue X-point 80x80 • 4 x FMCs dual SFP+ 10Gb • 8 x 2GB DDR2 SODIMM VLP Modules • 4 x QDRII SRAMs • Links Configured as Input or Output by software • Connects to either • Detectors or PC Farm Presented by John Coughlan. TWEPP Oxford. September 2012.

  6. TrainBuilder FE x 4 SRAM PPC code Virtex5 FX100T, Dual PPC 2 x DDR2 VLP SODIMMs 2 GByte DESY FMC 2 x 10Gbps SFP+ QDRII SRAM 8 MByte Presented by John Coughlan. TWEPP Oxford. September 2012.

  7. TrainBuilder FE x 4 10G Inputs from Detector QDRII SRAM Pixel reordering To Xpoint DDR2 Partial Images train buffer Virtex5 FX100T, Dual PPC 2 x DDR2 VLP SODIMMs 2 GByte DESY FMC 2 x 10Gbps SFP+ QDRII SRAM 8 MByte Configured as INPUT : Data from Detectors Presented by John Coughlan. TWEPP Oxford. September 2012.

  8. TrainBuilder FE x 4 Virtex5 FX100T, Dual PPC 2 x DDR2 VLP SODIMMs 2 GByte DESY FMC 2 x 10Gbps SFP+ QDRII SRAM 8 MByte 10G Outputs to Farm From Xpoint QDRII SRAM Image Tiling DDR2 Full Images train buffer Configured as OUTPUT : Data to PC Farm Presented by John Coughlan. TWEPP Oxford. September 2012.

  9. TrainBuilder 10G link FMC • FPGA Mezzanine Card FMC • ANSI/VITA 57 • Developed by DESY Electronics Group • Dual 10 Gbps SFP+ • Vitesse 10G PHYs • XAUI 4 x 3.125 Gbps Presented by John Coughlan. TWEPP Oxford. September 2012.

  10. TrainBuilder Demonstrator • Crosspoint Switch • Mindspeed 80x80 6.5 Gbps • 64 Tx & Rx @ 2.5 Gbps • Switch changes @ ~ 10 Hz • Master FPGA Virtex5 FXT100 • Manages data switching • PPC GbE TCP/IP server • Python scripts on client PC • FPGA Configuration CF card • Boot Spartan3 with FLASH Xpoint Master FPGA GbE Ctrl Boot MMC Spartan3AN FPGA GbE Ctrl RJ45 Presented by John Coughlan. TWEPP Oxford. September 2012.

  11. TrainBuilder ATCA Interfaces JTAG • Zone 3 Custom RTM • 32 Tx & Rx data lanes to Crosspoint • + clock&controls • Zone 2 • Std interfaces • Zone 1 • Power 48V • MMC 3V3 ATCA Zone 3 Cable/Rear Transition Module ATCA Zone 2 GbE Ctrl PCIe ATCA 48V DC-DC ATCA Zone 1 JTAG Presented by John Coughlan. TWEPP Oxford. September 2012.

  12. TrainBuilder Data Rates • Detector (512 bunches/train) • ~ 640 MB/sec per 10G LINK • 10G • XAUI 4 x 3.125 Gbps • > 1 GB/sec • Crosspoint • Aurora 4 lanes 64B/66B @ 2.5 Gbps • > 1 GB/sec • DDR2 (1 per link) 2 GB • Dual PPC Hard IP Embedded • ~ 1 GB/sec Read & Write (Long bursts) • QDRII SRAM (1 per 2 links) 8 MB • 32 bit R & W @ 250 MHz • ~ 1 GB/sec per link (2 word burst) • Total B/W per TB board 8 Links • ~ 8 GB/sec Presented by John Coughlan. TWEPP Oxford. September 2012.

  13. Firmware Presented by John Coughlan. TWEPP Oxford. September 2012.

  14. FE FPGA Firmware DDR2 SDRAM QDRII SRAM TB FPGA XPOINT DUAL 10G PHY FMC Power PC Memory Controller Subsystem ‘XCTRL’ XAUI Core 0 Aurora Core 0 10G UDP Core 0 QDRII Interface 4 TX PAIRS 4 TX PAIRS PHY SFP+ 4 RX PAIRS 4 RX PAIRS XTAL PLL XTAL PLL LOCAL LINK Protocol Power PC Memory Controller Subsystem QDRII Interface Aurora Core 1 XAUI Core 1 10G UDP Core 1 4 TX PAIRS 4 TX PAIRS PHY SFP+ 4 RX PAIRS 4 RX PAIRS XAUI DDR2 SDRAM Image Buffers Storage QDRII SRAM Image Manipulation Aurora 64B/66B Presented by John Coughlan. TWEPP Oxford. September 2012.

  15. FE FPGA Firmware 10G Inputs from Detector QDRII SRAM Pixel reordering To Xpoint DDR2 Partial Images train buffer DDR2 SDRAM QDRII SRAM TB FPGA XPOINT DUAL 10G PHY FMC Power PC Memory Controller Subsystem ‘XCTRL’ XAUI Core 0 Aurora Core 0 10G UDP Core 0 QDRII Interface 4 TX PAIRS 4 TX PAIRS PHY SFP+ 4 RX PAIRS 4 RX PAIRS XTAL PLL XTAL PLL LOCAL LINK Protocol Power PC Memory Controller Subsystem QDRII Interface Aurora Core 1 XAUI Core 1 10G UDP Core 1 4 TX PAIRS 4 TX PAIRS PHY SFP+ 4 RX PAIRS 4 RX PAIRS XAUI DDR2 SDRAM Image Buffers Storage QDRII SRAM Image Manipulation Aurora 64/66B Presented by John Coughlan. TWEPP Oxford. September 2012.

  16. FE FPGA Firmware 10G Outputs to Farm From Xpoint QDRII SRAM Image Tiling DDR2 Full Images train buffer DDR2 SDRAM QDRII SRAM TB FPGA XPOINT DUAL 10G PHY FMC Power PC Memory Controller Subsystem ‘XCTRL’ XAUI Core 0 Aurora Core 0 10G UDP Core 0 QDRII Interface 4 TX PAIRS 4 TX PAIRS PHY SFP+ 4 RX PAIRS 4 RX PAIRS XTAL PLL XTAL PLL LOCAL LINK Protocol Power PC Memory Controller Subsystem QDRII Interface Aurora Core 1 XAUI Core 1 10G UDP Core 1 4 TX PAIRS 4 TX PAIRS PHY SFP+ 4 RX PAIRS 4 RX PAIRS XAUI DDR2 SDRAM Image Buffers Storage QDRII SRAM Image Manipulation Aurora 64/66B Presented by John Coughlan. TWEPP Oxford. September 2012.

  17. TB RAM RAM FPGA 5 ASIC RX UDP XAUI PHY SFP AUR PPC UDP XAUI PHY SFP PPC AUR PPC UDP XAUI PHY SFP RAM RAM FPGA 6 AUR PPC UDP XAUI PHY SFP AUR PPC UDP XAUI PHY SFP RAM RAM FPGA 7 AUR PPC UDP XAUI PHY SFP AUR PPC PE XAUI PHY SFP RAM RAM RAM RAM FPGA 8 FPGA 4 AUR PPC PE XAUI PHY SFP PHY XAUI UDP PPC AUR RAM ASIC RX AUR PPC PE XAUI PHY SFP SFP PHY XAUI UDP PPC AUR UDP XAUI PHY SFP PPC RAM RAM TrainBuilder Firmware System TrainBuilder FEMs FEM RAM FPGA FPGA 1 SFP PHY XAUI UDP PPC AUR SFP PHY XAUI UDP PPC AUR RAM RAM FPGA 2 SFP PHY XAUI UDP PPC AUR SFP PHY XAUI UDP PPC AUR RAM XP RAM FPGA 3 SFP PHY XAUI UDP PPC AUR SFP PHY XAUI UDP PPC AUR SFP FPGA FPGA GIGE Presented by John Coughlan. TWEPP Oxford. September 2012.

  18. Applications Presented by John Coughlan. TWEPP Oxford. September 2012.

  19. Status • First ATCA boards manufactured in Q1/2012 • 4 Boards all passed Boundary SCAN first time. • Basic Tests working • 10G links Tx and Rx • DDR2 • QDRII • Crosspoint • GbE controls • Performance testing at RAL. Data Rates 1 GB/s/link? Soak Tests. • Board in Hamburg for XFEL DAQ PC Farm tests since July. Presented by John Coughlan. TWEPP Oxford. September 2012.

  20. First Detector Application Large Pixel Detector Prototype Quadrant. 4 FEMs 10G links x4 1 x TB configured as 4 Inputs and 4 Outputs Presented by John Coughlan. TWEPP Oxford. September 2012.

  21. 2 x TBs : ½ Mpixel 2 x TB System : 8 Links ½ Mpixel x8 Passive Cables Tx & Rx INPUT OUTPUT Presented by John Coughlan. TWEPP Oxford. September 2012.

  22. FUTURE 4 x TBs : 1 Mpixel 2013 1 Mpixel with 4 x ATCA TB Demonstrators. 16 LINKs NEW Optical RTMs x16 NEW ATCA Switch Board Parallel Opto Links SNAP 12 Tx / Rx @ 2.5 Gbps Scaleable to larger systems Presented by John Coughlan. TWEPP Oxford. September 2012.

  23. Summary & Plans • Developed an ATCA board for common XFEL DAQ • Firmware infrastructure already operating incl. 10G UDP • 2 x Demonstrator ATCA can instrument ½ Mpixel today • Baseline is 1 Mpixel. Scheme exists to scale to larger detectors. • Plans • Complete Performance Tests on prototypes • Develop Optical RTM and Switch card 2013 • Integrate with 1 Mpixel detectors 2014 • Install XFEL in 2015 • Data taking at XEFL in 2016 • Acknowledgements: • M. Zimmer, I. Sheviakov (DESY Electronics) • C. Youngman (XFEL DAQ) Presented by John Coughlan. TWEPP Oxford. September 2012.

  24. Questions Presented by John Coughlan. TWEPP Oxford. September 2012.

  25. Spare Slides Presented by John Coughlan. TWEPP Oxford. September 2012.

  26. European-XFEL Hamburg Xray Free Electron Laser 2D Camera captures 512 Images per Train Train repetition rate = 10 Hz ~5,000 fps Presented by John Coughlan. TWEPP Oxford. September 2012.

  27. Crosspoint Switch Operation DDR2 DDR2 FEEs PC Farm Full Image Trains 512 x 2 MB Partial Images 128 KB Time Ordered Switch Protocol : Xilinx Aurora 64/66B @ 2.5 Gbps Xpoint Switch changes only @ 10 Hz Presented by John Coughlan. TWEPP Oxford. September 2012.

  28. 10G UDP FPGA 10G PHY Local Link • LL FRAME (64 bit) <-> UDP PKT STREAM • FPGA <-> FPGA - FPGA <-> PC • Tx : MAC, IP ADDR & PORT LUT (LL Frame Header) * • Rx : IP, PORTS, MAC, Filter for PLAYBACK mode • Software Programmable : Packet Length (Jumbo), IFG ...etc *Plan to add Resends on Tx for “Reliable UDP” (TB to PC Farm) Presented by John Coughlan. TWEPP Oxford. September 2012.

  29. PPC DDR2 Memory Controller LL Write DDR2 SODIMM 64 bit 2 GB LL Read DDR2 controller 500 MT/s V5 FX100T 2 x PPC440 Hard IP blocks Each with 4 DMA engines Tx&Rx Local Link Interface 32 bit @ 250 MHz ~ 1 GB/sec Concurrent Read & Write Embedded C code manages DMA engines/buffers Presented by John Coughlan. TWEPP Oxford. September 2012.

  30. TrainBuilder Next Generation • TB • V7/K7 • ≥8 x QSFP+ • >8 x DDR3 SODIMM • XP 144 x 144 @10G • XP -> V7/K7 • QDR Bandwidth? • Bigger Packages • No FMC Presented by John Coughlan. TWEPP Oxford. September 2012.

More Related