220 likes | 305 Views
Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System. The Case of the ANTARES Data Acquisition and Detector Control. S. Anvar, CEA Saclay Dapnia On behalf of the Antares collaboration. A Neutrino Telescope. http://antares.in2p3.fr. The Antares Collaboration.
E N D
Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System The Case of the ANTARES Data Acquisition and Detector Control S. Anvar, CEA Saclay Dapnia On behalf of the Antares collaboration
A Neutrino Telescope http://antares.in2p3.fr RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
The Antares Collaboration • Institutes • Particle physics • Astronomy • Sea science RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Constraints • Environmental constraints • Baseline data rate of ~100 kHz / PMT • Data rate fluctuations ~2MHz / PMT due to bioluminescence • Strong uncertainties on environment parameters • Reduced space and thermal conductivity • … • Reliability (MTBF) • Maximize simplicity • Minimize consumption • Minimize number of components • … • Deployment constraints • High cost of submarine cables • Limits on string cable rigidity or sea operations • … RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Slow control Data Clock Energy Design principle Offshore Onshore Electro-mechanicalcable • 12 strings • 25 detecting nodes per string (every 12.5m) • 3 optical modules (PMTs) per acquisition node • 2 "ARS" digitizing chips per optical module Processingnodes Acquisitionnodes • Real time processing: • Offshore: readout 1800 ARS chips spread out over 30 000 000 m3 @ 2500m underwater • Onshore: perform real time trigger computations on incoming data. String control Detectors Junctionbox Electro-opticalcable (50 km) RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Ethernet 100 Serial links (RS485, RS232) Each offshore acquisition node: one DAQ/SC Board • Achieved power consumption of ~5W including optical Ethernet tranceiver laser DAQ/SC Board ARS readout/SCLocal trigger106 gatesXilinx FPGA SDRAMMemory 64 MB Real-timeembedded CPUPPC MPC860VxWorks RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
The ARS chip and the DAQ/SC board DAQ/SC Board ARS chip RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
100 Mb/s M Sectors 1 Gb/s M To Shore (DWDM) SCM SCM Network Topology (1) Cable rigidity Cable diameter Connection complexity Each LCM has its own I/O data flow(Slow Control & Physics Data) RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Network Topology (2) RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
The offshore switch board Offshore switch Board Allayer chip RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
The electronic crate RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
N N N N N Network Topology (3) • Solution • barrel shifting • synchronized by global clock N N N N N ~15 Mb/s SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000) Used as data concentratormany input ports one output port risks of congestion intelligent flow control in node ON-SHORE SWITCH: (in: 60x1000, out: 20x1000) Instant ~6 Gb/s PC PC PC PC PC PC PC RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Massively Distributed and Heterogeneous System • Final System: A Network of ~320 Nodes • ~300 Offshore Nodes (MPC860 + VxWorks) • ~20 Onshore Nodes (PC + Linux) • A number of distributed applications running in parallel and communicating • DAQ • L2 Trigger • Slow Control / Acoustic positioning acquisition • L1 trigger ? • Run control RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Software Technologies • Programming Languages • C/C++ for offshore software components • C++ / Java for onshore processing • Java / C++ for onshore display • Distribution • Multiple messaging and data servers (ControlHost) TCP/IP based • VxWorks Zero-copy TCP/IP (~25Mb/s ~38 Mb/s per node) • Offshore network traffic control (barrel shifting) • Data and configuration storage • Oracle database • Root files • Design & Development • Object Paradigm • UML (Unified Modeling Language) + Extensions • Model-driven engineering RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Interrupt – shared memory communication Data Memory (SDRAM) ARS readout (FPGA) write intrHandler (RTOS) frameSendertask (RTOS) Semaphore(RTOS) read and write Actual frame formatting and sending through TCP/IP socket RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
asynchronous method call :ARS_Readout :FrameSender reference to design pattern acq(in frameID:int) Interrupt - SharedMem 6 1 <<RTOS>> FS_Node <<fpga>> AR_Node FrameSender ARS_Read readout framer +acq(in frameID:int) +fillBuffers(id:int) 6 1 frameSender <<reside>> <<reside>> ARS_Readout FrameSender Interrupt – shared memory communicationseen as asynchronous method call RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Implemented in VHDL RealTime::Thread #run() PrmRefType IntrSharedMemSkeleton #run() #installInterrupt() +accept(in pr:PrmRefType) ARS_Read FrameSender FrameSenderSkeleton +fillBuffers(id:int) +acq(in x:Frame) {bind(Frame)} +accept(in x:Frame) IntrSharedMemProxy +doIntr(in x:PrmRefType) Applying the design pattern: model transformation 6 reusablesoftware 1 1 frameSender RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Widespread belief Performance goes against abstraction & flexibility Wrong if the issue is explicitly addressed ActuallyAbstraction modularity specific optimization perf. RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
DAQ/SC Board ARS readout/SCLocal trigger106 gatesXilinx FPGA SDRAMMemory 64 MB DAQ/SC Board ARS readout/SCLocal trigger Real-timeembedded CPU System On Chip Xilinx Virtex II Pro?Xilinx Virtex IV Real-timeembedded CPUPPC MPC860VxWorks SDRAMMemory 64 MB Ethernet 100 Serial links (RS485, RS232) Ethernet 100 Serial links (RS485, RS232) Evolutions (1) • Lower consumption • Reduced number of components increased reliability RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Evolutions (2) • Present scheme: « all data to shore » • Unexpected variable baseline rates: ~100 400 kHz instead of ~70 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Present scheme: « all data to shore » Unexpected variable baseline rates: ~100 400 kHz instead of ~100 Evolutions (2) From 5 to 30 cm/s (!) • Software L1 trigger • Reduced L1 partial data sent from offshore to central L1 workstation • Central L1 computes trigger with global vision of detector • Trigger decision broadcasted through software • Expected gain in data rate: x1000 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Slow control Data Clock Energy Conclusions • HEP DAQ and trigger techniques • High data rates • Very weak signal • Highly distributed system • Soft real time constraints • … • Challenges unusual for HEP • No known beam (calibration) • Very embedded system • Very hostile environment • Uncontrolled factors (biolum) • … • Both Robust and Flexible Design • Allows adaptation • Allows for remote evolution • … RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control