Current status of the applications for a live-data-reduction method in MLF, J-PARC

1. J-PARC center, Japan Atomic Energy Agency Y. Inamura Current status of the applications for a live-data-reduction method in MLF, J-PARC

2. Contents • Introduction for MLF Standard software • Structure and examples of the live data reduction • Live data reduction with IROHA2 • Summary

3. Live Data Reduction (LDR) Users Requests : real-time data processing • Confirmation of how measurement progress. • Carry out measurements more efficiently • With increasing intensity of neutron beam, measurement time becomes shorter. • So, users require more speedy analysis to decide next measurement conditions • This is used as a feedback information to automate executions of measurements Live Data Reduction project in MLF Treat and visualize current measuring data with “Safety”, “Speed” and “Flexibility” in real-time

4. MLF Standard Software :DAQ middleware MLF have adopted the event-recording method for data acquisition system Field Controller Controled by HTTP • DAQ middleware consists of many modules • Gatherer modules to obtain neutron events from electronics of detectors • PSD, 1d- and 2d scintillation counters used in MLF • Logger modules to store event data to storages. • Dispatcher, monitor, and some data treatment modules Data Acquisition System Angle Encoder Gather, Logger DAQ middleware Data File Storage DAQ middleware is installed in almost instruments in MLF

5. MLF Standard Software : Instruments and measurements control from the web browser （IROHA2） Chopper Device Server Slit Device Server Data Acquisition Temperature Device Server DAQ Temperature Choppers Beam Slit Controller

6. MLF Standard Software : Instruments and measurements control from the web browser （IROHA2） From outside of J-PARC Access Sequence Server Integrate Server IROHA2 Sequence Server Every Servers are connected using network(HTTP) Management Server Chopper Device Server Slit Device Server Data Acquisition Temperature Device Server IROHA2 Integrate Control Server DAQ Temperature Choppers Beam Slit Controller

7. MLF Standard Software:Data reduction and Visualization (Utsusemi) • Histogram data creation from event-recorded data produced by MLF DAQ system • Data Reduction and visualization functions • GUI ( Commands execution, visualizations ) • Python + C++ “SequenceEditor”, commands sequence execution Commands Sequence 3D Plotter and Slicer( python ) Commands group （Façade Functions） Parameters of selected command 2D plotter ( python ) 1D plotter ( python )

8. More Information about Utsusemi http://mlfuser.cross-tokai.jp/en/compenv/utsusemi.html meet@MLFUtsusemi • Download binary package • Manual ( Only Japanese, sorry ) • Update information

9. MLF Standard Software used in MLF IROHA2 Utsusemi Only Device Ctrl Now Installing Full Install

10. How to treat data during measurement During measurement going, user must access saved data files in storage Field Controller Direct access to files under writing NOT Safety Data Acquisition System Angle Encoder Gather, Logger DAQ middleware We need OTHER data stream instead of direct file access Access on writing data Receive data freely Output data with safety and speedy Data File Storage

11. Required system for Live Data Reduction High data throughput with safety and flexibility • There is an intense progress of technologies for a lot of internet services ! Subscriber Publisher Broker Subscriber ”Publish – Subscribe” model with message-oriented middleware Publisher Data Data Loose coupling between publisher and subscriber through broker Subscriber Publisher Broker Subscriber Publisher Subscribe Publish Data Data • Regardless of Publishers • Access to the broker to get information • Regardless of subscribers • No reply, continue operate Higher Scalability Easy to increase the number of both publishers and subscribers by requests

12. Base Functions in ready Broker： “Redis”, Open-source In-memory DB. Faster. Many Languages has interfaces Publishers： As the new modules in DAQ middleware New Publisher module has already worked with long-term stability Easy installation into BL which uses DAQ middleware Field Controller Data Acquisition System Angle Encoder Broker Subscribe Publish Subscriber Publisher Available for Users Redis Server Subscriber code Publisher module Gather, Logger DAQ middleware RawData RawData Subscribers : • Development codes with Python or C++ • Easy to make codes suitable for each BL Data File Storage

13. Example : On-line monitor for 2D-detector at BL17 Raw data On-line beam monitor for MWPC (Python) • Simple python code can be useful function. • Easy to plot the event data with high throughput Redis Server Subscriber Code MWPC CPS trend graph MWPC CPS Monitor MWPC Detector Map 高速な処理 データは強度をそのまま表示

14. Example : On-line visualization for detectors at BL21 Raw Data Subscriber (C++) • Subscriber is C++ code to read raw data and convert. Visualize raw data by using of ROOT in web browser • Beam profile monitor • Intensity maps for each detector bank Intermediate data Visualize on Web Browser （JSROOT） Redis Server Subscriber Code

15. Example : Live Data Reduction on Utsusemi Raw data Utsusemi-LDR (C++, Python) Only replace a histogram creation function • UtsusemiLive Data Reduction • Subscriber : get differential event data to convert them to histogram format and merge with previous histogram data. • Execution of codes of data reduction on merged data • Data reduction functions are same as off-line • visualization functions • Repeat above steps at regular intervals Redis Server Subscriber code

16. Example : Live Data Reduction on Utsusemi 空蟬LDR (C++, Python) Redis サーバー Subscriber コード GUI to execute for LDR “SequenceEditorLDR”( under testing)

17. Next Step • More easy to use • Problems on use case as a beam monitor • Monitoring script should be started with the begin command on IROAH2 system • The results should be visualized on web interface same as IROHA2 • Monitoring script should be treated and editable on the web interface • Keep simple and flexibility system to install and use

18. Live Data Process Serverworks on IROHA2 • Works as a DEVICE server of IROHA2 • Start execution of a given script and stop it by IROHA2 “Begin” and “End” commands • 1D text or a figure produced by a script can be shown on the web interface Web Interface PC Live Data Process Server IROHA2 servers Devices, Management, Integrate, Sequence Begin/End Status/Info Python Codes Begin/End Status/Info DAQ middleware Figure file 1D text file

19. Live Data Process Serverworks on IROHA2 • Works as one of the DEVICE servers of IROHA2 • Executes a given script by IROHA2 signal as “Begin” and “End” protocol • Flexible for general-purpose data treatments • 1D text or a figure produced by a running script can be shown on the web interface Web Interface PC Live Data Process Server IROHA2 servers Devices, Management, Integrate, Sequence Begin/End Status/Info Python Codes Begin/End Status/Info DAQ middleware Figure file 1D text file Select and edit the Script

20. Live Data Process Serverworks with other servers The results ( figure and 1d plotter ) can be seen on the main web-UI of the instrument control ( IROHA2 integrated server ) • Enable to access from outside of J-PARC with ID and one-time password

21. Data Monitoring with live data reduction (test-bed) Live Data Process server with Utsusemi environment PC Web Interface Live Data Process Server IROHA2 servers Devices, Management, Integrate, Sequence Begin/End Status/Info Utsusemi LDR codes Begin/End Status/Info Subscribe DAQ middleware Redis Publish Figure file Utsusemi Environment 1D text file • “Dockerize” merit • Easy Maintenance • Portability over the beam lines • Set only several IP address

22. Simpledata-driven measurement control with LDR (plan) PC IROHA2 servers Devices, Management, Integrate, Sequence Live Data Process Server Begin/End Status/Info Utsusemi LDR codes Device Control commands Begin/End Status/Info Subscribe DAQ middleware Redis Publish Publish Temperature Device Server Temperature Device Server Begin/End Status/Info

23. Summary • Live data reduction method in MLF • The system is ready to use • Flexible and safe • Already some instruments utilize this system in actual user measurement • Live data process server • Start of script execution is synchronized with “Begin” command of IROHA2 • this server with the live data reduction script • Possibility to achieve the simple data-driven measurement control • Future plan • Confirmation of stability in long-term period • Improvement of the simple data-driven measurement control