SACLA/XFEL におけるイベント駆動型 データ収集システム

1. SACLA/XFELにおけるイベント駆動型データ収集システムSACLA/XFELにおけるイベント駆動型データ収集システム 大端通　JASRI/SPring-8 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

2. Contents • Glance over • SACLA/XFEL • Overview of DAQ system • DAQ Front-end • Future perspective • Summary 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

3. DAQ for Large Image Sensors : >5Gbps CPU board “K Supercomputer” 10 Pflops, 2012.4~ >5 Gbps in total Custom-made VME Camera Link Board CameraLink > 2Gbps/connection Camera Grabber Camera Link カード Camera Link カード Camera Link カード Camera Link カード Camera Link カード Camera Link カード Camera Link カード FPGA Lossless Compression FPGA データ 圧縮 FPGA データ 圧縮 FPGA データ 圧縮 FPGA データ 圧縮 FPGA データ 圧縮 FPGA データ 圧縮 FPGA データ 圧縮 PrPMC Linux CCD array 4M pixels (8 sensor modules) 60Hz 10 Gbps Ethernet PrPMC PrPMC PrPMC PrPMC PrPMC PrPMC PrPMC PC Cluster Preprocess 10Tflops control 10 Gbps Ethernet High Speed Storage Reduced data for on-line monitor Data-handling Servers 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

4. SACLA : XFEL facility at SPring-8(SPring-8 Angstrom Compact free electron LAser) SPring-8 • Began operation in February 2011. • Aiming for < 1Å of wavelength of FEL by 8-GeV linac. • First lasing achieved on June 2011 at 1.2Å . • User experiment will begin on April 2012. ~700m Experimental Hall SACLA e- 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

5. Courtesy of M. Yabashi Anticipating a lot of Type of Experiments Generation of extreme state High peak brilliance x109 XFEL Atomic-level imaging for non-crystalline objects Probing of ultrafast chemical reaction & phase transition Ultrafast pulse < 100 fs Spatial coherence 100% phage flagella, cilia 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

6. Differential Cross Section Structure Factor r (r) : Electron Density Distribution Coherent X-ray Diffraction e.g. Coherent X-ray Diffraction Imaging A lot of images must be taken by a large pixel sensor to reconstruct the original structure. Phase Retrieval Sample Image Reconstruction without the aid of Lenses Fourier Transform Sample Coherent X-rays Unstained Human Chromosome 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28 Courtesy of Y. Nishino

7. Overview of DAQ system

8. Requirements for Data Acquisition(DAQ) • Shot-by-shotdata acquisition in synchronization with the beam operation cycle. • In order to correlate the beam characteristics with the data. • 60Hz at present, higher for future. • Accumulate data of large image sensors into storage without data loss.⇒ >5 Gbpsof data rate with 6M pixels at 60Hz . • Data compression to reduce the bandwidth and storage size. • On-line data-quality monitor for efficient data collection. • Any type of users instrument specific to their experiment can be attached to our DAQ to work with. • Like a Plug-and-play device. 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

9. Schematic of DAQ 京(K) Super Computer Front-end Electronics Front-end Computer On-line Database • Network distributed system. • MADOCAas the control framework. DAQ Control Image Sensors CCD cameras GUI MySQL VME : Solaris PC : Linux Data Buffering High Speed Storage VME : Solaris PC : Linux Beam line Instruments Monitors 1G/10G Ethernet Data Buffering Data Buffering GUI Accelerator Status Database PC farm for Analysis On-line Data Monitor Users Experimental Instruments Detectors Windows LabVIEW Sybase TCP socket client vi 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

10. Schematic of DAQ 京(K) Super Computer Front-end Electronics Front-end Computer On-line Database • Data transfer via TCP/IP over 1- and 10-Gbps Ethernet. • FIFO buffering for non-real-time components. DAQ Control Image Sensors CCD cameras GUI MySQL VME : Solaris PC : Linux Data Buffering High Speed Storage TCP/IP VME : Solaris PC : Linux Beam line Instruments Monitors 1G/10G Ethernet Data Buffering Data Buffering GUI Accelerator Status Database PC farm for Analysis Users Experimental Instruments Detectors On-line Data Monitor Windows LabVIEW Sybase TCP socket client vi 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

11. Event Synchronization: Tag Number • To reconstruct the data set in the same beam shot, a “tag number” is recorded in every data to identify which one is related to which beam shot. • Sequential number of master trigger pulse. • Counted by the trigger-counter module at each station. • Delivered to users instruments as well. • Encoded serial pulse-train, parallel bit-pattern. 1,2,3,.. Master Trigger 60Hz Pulse 1,2,3,.. 1,2,3,.. 1,2,3,.. 1,2,3,.. 1,2,3,.. 1,2,3,.. Trigger Counter Trigger Counter Trigger Counter Trigger Counter Trigger Counter Trigger Counter Trigger Counter Trigger Counter Trigger Counter Trigger Counter Gun Accelerator Components Monitor System Beamline Components Experimental Detectors Users Instruments 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

12. DAQ Front-end

13. DAQ Front-end Trigger tag counter Trigger On-the-fly lossless compression Safety margin < 300 Mbps Gigabit Ethernet << 1Gbps - Protocol overhead - Congestion control Cameralink (base configuration) < 2.04 Gbps DAQ front-end Interface with CCD camera and transfer the data to storage system 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

14. Implementation of DAQ front-end • general purpose FPGA board with custom user logic executable. FPGA: VIRTEX-5 (XC5VFX70T) • CPU board with GbE I/F CPU: PowerQUICC III • Cameralink board. Designed for a RF-beam monitor system and an x-ray diamond monitor system of SPring-8 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

15. Implementation Cameralink CPU board Bandwidth Limit: 760Mbps Repetition limit is ~90Hz 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

16. Implementation IP cores on FPGA 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

17. High Speed Storage : DDN + StorNext • Parallel-writing to a single file system • Achieve over 5 Gbps in total. • Convenient to handle data files in a centralized repository. Data Direct Network S2A9900 Quantum StorNext file system Data file servers Data of CCD sensor #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 1G/10G Ethernet SAN >100Gbps CameraLink ~180TB >GB/sec Single file system additional sensor > 5Gbps in total 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

18. DAQ System Worked as Expected • Intensively used during beam tuning of SACLA to view the beam profile. • Ready for user experiments. June 7, 2011 First lasing at SACLA Beam energy: 7GeV Wavelength: 1.2Å MPCCD image 1 mm 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

19. Future perspective

20. Upgrade plan In near future: Requirement for next gen. DAQ system • Image sensor: 6M ⇒ 76M pixels • Beam repetition: 60Hz ⇒ 120Hz ⇒ 300Hz ⇒ Data rate ~ x26 : ~150Gbps ! ~ x65 : ~380Gbps ! Target specification of DAQ front-end 1st phase Final target 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

21. Overview of DAQ system 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

22. Latest action and future plan - Features of several VISION technologies - Because progress of vision technology is very fast, we can wait until a market is growing up and specification will be settled down. 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

23. Latest action and future plan - Features of several generic serial link technologies - ※ depend on transceiver We started preparing test stand for XAUI and RocketIO, and will finish evaluation within this fiscal year. 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

24. Latest action and future plan - Platform design - FMC (FPGA Mezzanine Card) separate a logic and I/O from a platform. change suitable I/F enough performance. ~20Gbps Study items Actual bandwidth between FMCs . Actual bandwidth of DMA transfer at PCIe Hardware implementation FPGA Logic implementation 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28

25. Summary • We have constructed an event-synchronized data acquisition system for >5 Gbps of data rate for the SACLA experiment. • Shot-by-shot data acquisition up to 60Hz . • Expandable for user instruments. ⇒ Ready for user experiments. • Further upgrades for larger detector size and higher repetition rate will be necessary in future. • Faster data transfer and storage system. • Effective data compression / suppression. ⇒ Start fighting them. 次世代データ収集システム研究会＠大阪大学　核物理研究センター　2012/1/28