220 likes | 346 Views
CWG9 and Event Display. B. von Haller. 17.07.2014. CERN. CWG 9 in a nutshell. Started in May 2013 along with O 2 Group working on the Data Quality Monitoring the Quality Assurance the Visualization For Run 2 and Run 3 . CWG9 Members. Renu Bala Francesca Bellini Mihaela Gheata
E N D
CWG9 and Event Display B. von Haller 17.07.2014 CERN
CWG 9 in a nutshell • Started in May 2013 along with O2 • Group working on • the Data Quality Monitoring • the Quality Assurance • the Visualization • For Run 2 and Run 3 B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
CWG9 Members • RenuBala • Francesca Bellini • MihaelaGheata • Lukasz Graczykowski • MalgorzataJanik • Andreas Morsch • MihaiNiculescu • JeremiNiedziela • Ankita Sharma • Maciej Szymanski • Barthélémy Von Haller B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Definitions of DQM and QA Data Quality Monitoring & Quality Assurance • Feedback on the quality of data • Online (DQM) • Make sure to recordhigh quality data • Identify and solve problem(s) early • Offline (QA) • Make sure to analyze high quality data • Identify high quality runs • Involves • [Online gathering of data] • Analysis by user-defined algorithm • Production of monitoring objects such as histograms • Assessment of the quality of the data based on the objects • Storage of monitoring data • Visualization (+ human assessment of quality) B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
CWG9 Tasks • Group focusing on the data Quality Assurance, online and offline, and the visualization of its results and the data itself • Run 3 • Study how to monitor data efficiently and in plenty without interfering with the data taking • Discuss QA output and results, incremental QA and procedures to formalize if the results are acceptable or not • Determine the needs, and design the software, to access, visualize and interpret the results • Define and develop the software to visualize data, raw and reconstructed • Participate to the writing of the Technical Design Report and to the possible prototyping • Run 2 • Production data taking period -> coordinate maintenance and improvements of software • Opportunity to test concepts and software for Run 3 Storage Dataflow Monitoring Object Generation Automatic QualityAssessment Visualization B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Event Display People currentlyworkingdirectly on ED Maintenance and developmenttaken over by Warsaw group • JeremiNiedziela – PhDstudentat CERN • MaciejSzymański – Service Task on ED • Jakub Sala, Jakub Abelski, Adam Felis – Summerstudents • Warsaw group @ WUT for support • Mihai – Offline, former ED developer B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Event Display Offline recoEvent Display Off.Reco Off. Reco ED ED Run 2 tasks • Run 2 requirements • Split reconstruction and online ED • Allow multiple sources and unifyexistingEDs • Bookmarks • Under development by J. Niedziela and M. Szymanski • Workwill continue over the summer • Readyfor the commissioning and cosmics HLT HLT HLT HLTED HLTED B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Future work • Event Display • Alternativeplatforms (mostly web and/or mobile) • SummerstudentsfromFaculty of Mathematics and Computer Science startedworking on it • Longer term collaboration on thistopicis possible in order to implement a full fledgeevent display • Possible use of earlyalfa components • Prototypingneededhere • DQM/QA • Prototype of merging QA objectsusingmap-reducewithZeroMQ • Web based ROOT objects browser B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Future work – CWG10 Control & Configuration • Control and Configuration of hierarchical distributed systems • Development of a zeroMQ-based prototype for configuration distribution and process control • Evaluation of performance based on number of nodes, hierarchical levels, processes and configuration size • Process Management • Evaluation of technologies for process execution in sandbox/container (Docker, etc.) • Evaluation of performance based on number of nodes, hierarchical levels, processes • Evaluation of ZooKeeper for possible usage in O2 • Configuration management • Synchronization • Process control B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Conclusion • CWG9 is an important and dynamic working group in the O2 project • WUT is a key player in CWG9 • In terms of people and responsibility • Crucial for the future of Visualization in ALICE • The ED is already profiting greatly from its involvement • There are many more tasks and opportunities for collaboration in CWGs B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Backups B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
TDR and feasibility tests • April 2014: First draft • August 2014: Submission • CWG9 participates to • Chapter 4 System architecture : Quality control and assessment • Table 1: detectors needs • Explain DQM/QA architecture and the choices made • Figure 1: DQM/QA architecture • Explain Event Display architecture and the choices made • Figure 2: Event Display architecture • Chapter 5 Technology survey, evaluations and prototypes • Mergers architecture and feasibility tests with 0MQ • Results of storage tests (e.g. DB technologies) • [Web gui architecture (ROOT JS lib + DABC)] • Event display design as tested (cf Run 2) • Chapter 6 System Design B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Past and Current tasks • Bring everyone aboard [done] • Make people aware of others’ work in the field • Give a picture of the current situation to everyone • Run 2 • Event Display review and meetings, Warsaw involvement • Core refactoring • New features • Knowledge transfer Gain stability and support for Run 2 • DQM/QA review and preparation • Proposal for the online reconstruction and calibration • Run 3 • System requirements and system functionalities document [done] • Detectors needs survey • Definition of the future architecture and design • Prototypes and feasibility tests • Technical Design Report redaction B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
O2 Technical Design Report Schedule • October ‘13: • Define table of content • Establish editorial board • December ‘13: • System Requirement Document • High-level dataflow model • Computing platforms benchmarks • Networking benchmark • June ‘14 • Software framework architecture • Sep ‘14 • TDR B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
CWG9 TDR Timeline Proposal • January 14 • Define list of tables and figures • Draft of the architecture of the system • Launch subsystems exhaustive survey Submit list of tables and figures to TDR EC • February 14 • Draft tables and figures • Skeleton of 4.2.4 and 5.6 Submit skeleton to TDR EC • March 14 • Finalize tables and figures, including subsystems input • Iterate on text using input of TDR EC Submit text and final tables and figures to TDR EC • April 14 • Finalize text • Submit final text to TDR EC • May 14 • Iterate over our sections using CWGs input • Review work of other CWGs (especially what concerns us!) B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
TDR – skeleton, tables & figures Chapter 4 System architecture • 4.2 Data processing and computing model • 4.2.4 DQM and QA • « Quality control and assessment » • Table 1: detectors needs • Explain architecture • Figure 1: architecture • Explain the choices • Figure 2: Event display arch. B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
TDR – skeleton, tables & figures Chapter 5 Technology survey, evaluations and prototypes • 5.6 DQM and QA • « Quality control and assessment » • Technologies and design choices available concerning key points of our system • Storage • Access to results worldwide • Event display • Feasability tests & prototypes • Table 1: results of storage tests (e.g. DB technologies) • Figure 1: Web gui architecture (ROOT JS lib + DABC) • Figure 2: Event display design as tested (cf Run 2) B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Subsystems survey • What “tasks” (name it agents or algorithms if you prefer) will your subsystem need in Run 3 ? • For each of these task or group of tasks, tell us • Whether it already exist today and if so what is its performance. • What is the expected performance of such a task in Run 3. • How many plots are expected to be produced (for the shifter and for the experts). • Percentage of events needed to carry out the task online (minimum, optimal). • What is the input ? i.e. at which stage will it run ? • How fast the response has to be taken into account in the data flow ? • Whether the DQM/QA results have to become persistent and for how long ? • What does “Calibration QA” mean to you ? B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Survey status B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
Current and short term work • Run 2 : Prepare a proposal for the QA in relation with the online reco, calibration and monitoring (QA tasks, validation, bridge to DQM, …) • Online Calibration • Mainly for TPC • Many open questions on • Requirements • General architecture • Implementation • CDB • Working on a proposal to meet requirements while minimizing work • Use analysis QA train within HLT for reco monitoring • Use analysis QA train within HLT for calib monitoring • Use AMORE for raw data monitoring • Use AMORE infrastructure for storage and visualization • Run 2 – Review and preparation • Detectors « interviews » • (DATE Monitoring update) • Run 3 : Prepare requirements of the future system following CWG1 input • For the TDR (2014) • Define requirements and general architecture and features of the QA-DQM-Viz for Run 3 • Write it • Event Display • Decentralized model under implementation • Better stability • Split GUI and reconstruction • Possibility to switch between offline and HLT reco • Bookmarks (for users and for PR) • Involvement of the Warsaw group • Implemented by the end of 2013 B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014
HCDB OCDB DCS, GRP QC (sort of) Reco Selection/filters Calib Calibparam Raw ESDs QC QC Data on Castor Reading Writing Producing DBsdata procedures PHYSICS Run B. von Haller | CWG9 DQM-QA-Visualization | 17.07.2014