1 / 32

Update on the BNL SRS readout and analysis system

Update on the BNL SRS readout and analysis system. Martin Purschke Brookhaven National Laboratory. First presented at the CERN Coll Meeting Is in production at @BNL Presentation about our data on Thursday. Smaller DAQ Systems in PHENIX.

sol
Download Presentation

Update on the BNL SRS readout and analysis system

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Update on the BNL SRS readout and analysis system Martin Purschke Brookhaven National Laboratory • First presented at the CERN Coll Meeting • Is in production at @BNL • Presentation about our data on Thursday

  2. Smaller DAQ Systems in PHENIX • PHENIX has its main, 2 mill channels, 1400MB/s DAQ system that fills a room the size of a squash court • There has always been the need for “smaller”, more lightweight DAQ systems for tests, R&D, calibrations and external beam tests (at FNAL, AGS, CERN, …) - o(100K channels) • I personally also support another project at BNL & UPenn, a Medical Imaging group which develops various detectors for PET / MRI • These small DAQ systems are format-compatible with the main PHENIX DAQ. • Monitoring, debugging, and analysis code developed in the R&D phase can be re-used and built upon later • Existing software can be used in calibration beams, etc • Need to debug the readout of new electronics only once

  3. Our GEM Stack Setup There will be a presentation on Thursday morning (Bob Azmoun) about the data from this setup. x-y step motor Collimated CO60 Source Hybrids GEM Stack

  4. RCDAQ Client RCDAQ Client Command line RCDAQ Client Script RCDAQ – a lightweight and powerful DAQ RCDAQ server Writes to stdout (logfile) but doesn't have a UI component itself Client-server based DAQ based on “Remote Procedure Call” network protocol. Arbitrary number of clients, clients can be remote Network USB PCIe Hardware Hardware Hardware

  5. Devices implemented in rcdaq RCDAQ This is our largest client currently, data rate saturates 4 gigabit lines “TSPM” Prototype DRS4 Eval board “USB Oscilloscope” Nice for triggering, say, for cosmics PET Scanner for Mammography / Rodents (3072 LYSO crystals) The SRS System (that's why I'm here of course) Read out with 4 “Timing and Signal Processing Modules” (TSPM) Not shown: PHENIX's “Data Collection Module 2”

  6. RDAQ Features • Very modest investment in setup, maintenance, manpower • You can be up in one hour starting from scratch: • Download, configure, make && make install • Comes with working example scripts and pseudo devices (random numbers) to read out to start playing w/o hardware • Runs on virtually any modern Linux distro (rule of thumb: if root is available on your OS, it will work) – looking into MacOS • Have ran it on the “Redhat family”: { RH5/6, Scientifix Linux, Fedora}, the Ubuntu family, Gentoo (Gentoo is the prime development platform) • All it takes for you is to “buy into” the raw data format and a root interface library

  7. (Almost) Everything is a Plugin • The devices in rcdaq have very diverse and sometimes proprietary drivers / libraries (e.g. the DCM2 PCIe board needs the Jungo / windrvr6 kernel module + libs / utils) • An initially monolithic rcdaq soon became untenable • Switched to a plugin concept, where support for a device gets loaded on demand • Locality of support libraries, ease of distribution, etc • rcdaq by itself knows only very few generic and pseudo-devices, everything else is loaded as plugins

  8. No configuration files • … only config scripts. Much more powerful. • Scripts in both in the rcdaq setup as well as for the SRS • Each interaction with the system is a shell command • You can have conditions, loops, error checking, you name it # Setting up the Hybrids for sa in 0x801 0x401 0x201 0x101 0x8001 0x4001 0x2001 0x1001 0x802 0x402 0x202 0x102 0x8002 0x4002 0x2002 0x1002 ; do srs_control SETVALUE 6263 0x01 $sa 0x19 srs_control SETVALUE 6263 0x02 $sa 0x80 srs_control SETVALUE 6263 0x03 $sa 0x4 # ...stuff deleted... done You may recognize this as the configuration sequence for all master and slave hybrids – port, address, subaddress, value

  9. Network Transparency All client-server interaction based on RPC - Remote Procedure Call • Widely established standard (RFC 1831) for remote execution of code controlled by a client • Makes it look like a local function call, but the function executes on a server • The ubiquitous NFS is based on RPC, it is available virtually everywhere • It is a network protocol, so client and server don't have to be on the same machine • Can have DAQ and control machine in different rooms (or as far apart as you like as long as the connection traverses the firewalls) • Very robust and easy, and an open standard built into virtually any OS Your rcdaq server machine can be in a hot area but all your controls, including yourself, can be outside

  10. A minimalistic session ( could be typed in) rcdaq_client load /home/rcdaq/install/lib/librcdaqplugin_srs.so rcdaq_client create_device device_srs 1 1010 10.0.0.2 rcdaq_client daq_open rcdaq_client daq_begin 201 ..... run 201 is in progress .... rcdaq_client daq_end rcdaq_client daq_begin ...... run 202 is in progress ..... rcdaq_client daq_end rcdaq_client daq_close Define readout Start logging Start a run (201) End the run Stop logging

  11. … and you get The raw FEC sampled waveform $ printf "rcdaq-%08d-%04d.evt\n" 221 0 rcdaq-00000221-0000.evt The decoded time sample vs channel plot

  12. RCDAQ configuration As I said, everything is a script: #! /bin/sh # load the srs plugin rcdaq_client load /home/rcdaq/install/lib/librcdaqplugin_srs.so # create the read list rcdaq_client create_device device_srs 1 1010 10.0.0.2 % rcdaq_client daq_status -ll Stopped Filerule: rcdaq-%08d-%04d.evt Buffer Sizes: 65536 KB List of loaded Plugins: - SRS Plugin, provides - - device_srs (evttype, subid, IP addr) - readout an SRS crate - device_srspsrams (evttype, subid, IP addr) – read back the SRS configuration % rcdaq_client daq_list_readlist SRS Device Event Type: 1 Subevent id: 1010 IP: 10.0.0.2 % The “rcdaq_client <command>” construct can be made more convenient via aliases for interactive use

  13. The Quest for Speed • You may have seen many DAQs which format the raw data into a root tree, or some other generic format • We don't do that • Most modern devices give their data in a nicely compact format already, the SRS is no exception • We encapsulate the device data unaltered into a “Packet” with envelope information what is inside – no wasting cycles on reformatting • The envelope information identifies a decoding algorithm how to deal with the data • A typical event from PHENIX has about 1200 packets or so, one for each “device” which we read • Here we tend to have just one, same deal Event type packetid rcdaq_client create_device device_srs 1 1010 10.0.0.2 SRS coodinates

  14. Raw Data and Analysis • Flat files with a structure to get at events and the packets therein • Needs a root library (or standalone one for root-less ops) to get at the data • Comes with a framework called “pmonitor” for no-frills monitoring or analysis • Pmonitor supports true monitoring – looking at histograms while they are being filled and updated • Pmonitor is PHENIX's both online monitoring and offline framework • Access to files, rcdaq online streams, list of files • Another cool features: if file1 and file2 are valid raw data files, the concatenation is again a valid file – cat file1.evt file2.evt > file_combined.evt • Wealth of tools to quickly inspect data without writing any software

  15. Example of such a tool: ddump $ ddump -O -i srs-00000083-0000.evt -- Event 2 Run: 83 length: 4850 type: 1 (Data Event) 1329511626 Packet 1010 4842 -1 (ONCS Packet) 70 (IDSRSV01) - -- ----------------------4 Number of Hybrids: 4 Framecounter: 0 HDMI Channel: 0 Description: ADC Words: 2410 Rows : 16 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 123 122 126 127 125 124 116 117 119 118 114 115 113 112 80 81 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ---------------------------------------------------------------------------------------- 0 | 2643 2634 2662 2714 2719 2680 2647 2650 2659 2614 2626 2710 2682 2645 2671 2640 1 | 2634 2605 2614 2637 2630 2644 2623 2623 2641 2645 2661 2636 2594 2601 2622 2590 2 | 2622 2604 2606 2627 2634 2639 2636 2625 2614 2616 2635 2663 2620 2604 2625 2611 3 | 2612 2596 2605 2619 2599 2601 2587 2595 2609 2609 2614 2633 2598 2583 2609 2583 4 | 2618 2588 2594 2626 2635 2620 2602 2567 2564 2588 2625 2650 2611 2615 2633 2584 5 | 2644 2622 2638 2641 2648 2655 2650 2634 2623 2624 2640 2666 2643 2629 2626 2608 … 126 | 2497 2505 2518 2499 2477 2477 2487 2514 2500 2505 2507 2486 2473 2435 2456 2479 127 | 2481 2463 2480 2481 2470 2467 2466 2473 2493 2479 2480 2452 2462 2474 2460 2466 Framecounter: 1 HDMI Channel: 2 Description: ADC Words: 2410 Rows : 16 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 173 175 174 170 171 169 168 184 185 187 186 190 191 189 188 180 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ---------------------------------------------------------------------------------------- 0 | 2649 2654 2581 2604 2740 2785 2703 2686 2696 2705 2659 2644 2758 2704 2615 2632 1 | 2539 2556 2533 2518 2571 2588 2621 2620 2604 2596 2589 2575 2606 2581 2575 2579 2 | 2522 2551 2558 2575 2604 2607 2594 2594 2600 2582 2583 2578 2617 2612 2595 2562 3 | 2569 2563 2559 2557 2582 2574 2593 2634 2620 2616 2612 2596 2618 2589 2611 2616 4 | 2510 2505 2507 2535 2588 2597 2604 2627 2590 2585 2585 2568 2593 2562 2565 2555 Quick data inspection

  16. Data APIs • SRS Crate data are conceptually 3-dimensional • {hybrid, channel, time sample} ---- NO need to decode the hybrid data on your own • You can query the size of any dimension • You can get a hybrid's raw samples if you need that • Expert access to anything you can think of • Yet simple non-expert access for rapid results • Any future raw data format changes are carried by the access libraries and hidden from your code

  17. SRS interfaces • Getting the actual ADC samples How many actual time samples / slots did hybrid 0 have ? (28 here) for ( j = 0; j< 128; j++) { for (i = 0; i< p->iValue(0, “NSAMPLES”); i++) { int c = p->iValue ( j, i, 0); if (c > 10) h2->Fill ( i, j, pedestal[j] - c ); } Give me the ADC value for channel “j”, time slot “i”, hybrid 0

  18. Special Events • Your DAQ is only half as good if it doesn't allow you to capture “external” parameters about your data • A description, numeric parameters such as HV, device configuration parameters, position of a collimator, temperatures, and so on - “Slow controls” in general rcdaq_client create_device device_srsparams 9 905 10.0.0.2 rcdaq_client create_device device_srs 1 1010 10.0.0.2 • The “device_srsparams” doesn't read any SRS data, but reads back all configuration parameters from the system and saves them in a packet. They are in the data stream, and cannot get separated from the data. The “9” refers to a begin-run event, which is generated when we start data taking • “devices” to read (small) files, such as the config script itself (text) • Another one to extract numbers from a file and store them as actual numbers • You can capture virtually all setup parameters automatically

  19. ELOG Support Stefan Ritt's elog is the workhorse electronic logbook RCDAQ can make automated entries in the log We just need to tell it the whereabouts of the log (host, port, name) rcdaq_client daq_clear_readlist rcdaq_client load librcdaqplugin_srs.so rcdaq_client create_device device_srs 1 1010 10.0.0.2 rcdaq_client elog localhost 666 RCDAQLog

  20. GUIs • MLP's rule: GUIs must not be stateful • Statelessness allows to have multiple copies of the same GUIs at the same time • Allows to click start on one instance, click stop on another • And allows to mix GUIs with commands and scripts • (also makes GUI design so much easier)

  21. Our setup GEM Stack Hybrid HDMI Cable DAQ machine (we have something more powerful, yes) SRS Crate

  22. X-Ray scan over 10x10 positions The x-ray gun has an aperture of 0.14x4mm The whole data set was acquired and analyzed “on autopilot” This was mostly a showcase for my co-workers how this works All information (x-ray gun position, everything) is included in the data

  23. $ ddump -O -p 905 -t 9 /data/purschke/srsdata/xrayscan-00002000-0000.evt X-ray gun position 15748 7874 in run 2000 157480000 78740000 2000 $ $ ddump -O -p 906 -t 9 /data/purschke/srsdata/xrayscan-00002000-0000.evt Packet 906 8 -1 (ONCS Packet) 6 (ID4EVT) 0 | 157480000 78740000 2000 This saves the file as text This extracts the (3) numbers How did we do the “autopilot” aspect? We set up two “devices” reading a file “position.txt” • rcdaq_client create_device device_file 9 900 $0 • rcdaq_client create_device device_file 9 905 position.txt • rcdaq_client create_device device_filenumbers 9 906 position.txt rcdaq_client create_device device_srs 1 1010 10.0.0.2 For a long list of x-y positions of the x-ray gun, the main script “drives” the step motor there, and writes the current x-y positions to “positions.txt” Start taking data, wait for a given number of events, stop Move to next position

  24. Question: At the last meeting at CERN we had a live demonstration of the system I could bring in the BNL system on Thursday, sit with interested folks somewhere in a quiet place Show of hands here, or if you are shy, email me later, and I'll try to set something up in a quiet corner

  25. Summary The RCDAQ + assorted analysis software is a viable alternative to the ALICE system “full” DAQ system, device data, auxiliary data, elog support Lightweight and easy In production at BNL System is available to RD51 members (for now, email me) Fast readout (little overhead, reached Gbit line speed) Largely Linux-flavor agnostic, you don't have to devote your system to it Mac implementation might come – should work on any Unix flavor Has all analysis support, setup examples, GUIs, you name it.

  26. Backups

  27. Interface to our SRS data... • Getting the raw samples for hybrid 0: How many raw samples did the FEC card give for hybrid 0? for (i = 0; i< p->iValue(0, “NRAWSAMPLES”); i++) { h1->Fill( i, p->iValue(i, 0, "RAWSAMPLES")); } Give me FEC raw sample “i” for hybrid 0

  28. $ ddump -O -t 9 -p 910 rcdaq-00001001-0000.evt This is Run 1001 with the High Voltage set to 1510 This uses sample number 5 of the calorimeter. $ ddump -O -t 9 -p 911 rcdaq-00001001-0000.evt Packet 911 6 -1 (ONCS Packet) 6 (ID4EVT) 0 | 5e6 5 $ ddump -O -t 9 -g -d -p 911 rcdaq-00001001-0000.evt Packet 911 6 -1 (ONCS Packet) 6 (ID4EVT) 0 | 1510 5 $ 0x5e6 = 1510 “HV Scan” #! /bin/sh INITIAL_RUN=$1 [ -n "$INITIAL_RUN" ] || INITIAL_RUN=1000 RUN=$INITIAL_RUN for highvoltage in $( seq 1500 10 1600 ) ; do # here you would do something to actually set the HV cat >> description.txt <<EOF This is Run $RUN with the High Voltage set to $highvoltage This uses sample number 5 of the calorimeter. EOF echo "starting run $RUN with High Voltage set to $highvoltage" rcdaq_client daq_begin $RUN # engineer some way for the run to end somehow, automatic end after # a given number of events, or some such. Here we do sleep 30 rcdaq_client daq_end done

  29. This will cause this script itself to to be captured so you always know how rcdaq was configured in this run A neat trick • Say you have a rcdaq setup script such as #! /bin/sh rcdaq_client daq_clear_readlist rcdaq_client load librcdaqplugin_srs.so rcdaq_client create_device device_file 9 910 description.txt rcdaq_client create_device device_filenumbers 9 911 description.txt rcdaq_client create_device device_srs 1 1010 10.0.0.2 ELOG=`which elog` [ -n "$ELOG" ] && rcdaq_client elog localhost 666 RCDAQLog rcdaq_client create_device device_file 9 900 $0

  30. pmonitor Evolved as a PHENIX standard monitoring framework Was soon recognized as a valuable tool for offline processing (just leave the online capabilities alone) Supports online/interactive and batch-style operation Still has all the online power under the hood Relieves the user of the mundane and repetitive tasks (opening a stream, event loop, inspection points, flow control) Shields the user from the complexities of root linking Much, much easier yet much more powerful

  31. Monitoring What's the big deal??? Normally when you process data, you run through a dataset. When done, you save histograms, then look at distributions, fit, whatever In a monitoring process, you want to look at histograms etc while the processing of data is in progress. You want instantaneous feedback how you detector is doing, etc Manipulate histograms/objects on the fly, plot, fit, reset, rescale, draw,... Control the processing (update gains, scale factors, what have you)

  32. Repetitive tasks Initialize (book histograms, set up constants, read database, whatever) Choose what data stream to open Open it Loop through the events in the stream Call analysis routine for each event Close stream Save output Only those are really project-specific How does a standard analysis process look like? Initialize (book histograms, set up constants, read database, whatever) Choose what data stream to open Open it Loop through the events in the stream Call analysis routine for each event Close stream Save output Pmonitor relieves you of all repetitive (white) tasks in the above list (and then some)

More Related