1 / 22

Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System

Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System. The Case of the ANTARES Data Acquisition and Detector Control. S. Anvar, CEA Saclay Dapnia On behalf of the Antares collaboration. A Neutrino Telescope. http://antares.in2p3.fr. The Antares Collaboration.

elpida
Download Presentation

Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Addressing the Software and Hardware Trade-Offs of an Embedded Distributed System The Case of the ANTARES Data Acquisition and Detector Control S. Anvar, CEA Saclay Dapnia On behalf of the Antares collaboration

  2. A Neutrino Telescope http://antares.in2p3.fr RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  3. The Antares Collaboration • Institutes • Particle physics • Astronomy • Sea science RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  4. Constraints • Environmental constraints • Baseline data rate of ~100 kHz / PMT • Data rate fluctuations ~2MHz / PMT due to bioluminescence • Strong uncertainties on environment parameters • Reduced space and thermal conductivity • … • Reliability (MTBF) • Maximize simplicity • Minimize consumption • Minimize number of components • … • Deployment constraints • High cost of submarine cables • Limits on string cable rigidity or sea operations • … RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  5. Slow control Data Clock Energy Design principle Offshore Onshore Electro-mechanicalcable • 12 strings • 25 detecting nodes per string (every 12.5m) • 3 optical modules (PMTs) per acquisition node • 2 "ARS" digitizing chips per optical module Processingnodes Acquisitionnodes • Real time processing: • Offshore: readout 1800 ARS chips spread out over 30 000 000 m3 @ 2500m underwater • Onshore: perform real time trigger computations on incoming data. String control Detectors Junctionbox Electro-opticalcable (50 km) RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  6. Ethernet 100 Serial links (RS485, RS232) Each offshore acquisition node: one DAQ/SC Board • Achieved power consumption of ~5W including optical Ethernet tranceiver laser DAQ/SC Board ARS readout/SCLocal trigger106 gatesXilinx FPGA SDRAMMemory 64 MB Real-timeembedded CPUPPC MPC860VxWorks RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  7. The ARS chip and the DAQ/SC board DAQ/SC Board ARS chip RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  8. 100 Mb/s M Sectors 1 Gb/s M To Shore (DWDM) SCM SCM Network Topology (1) Cable rigidity Cable diameter Connection complexity Each LCM has its own I/O data flow(Slow Control & Physics Data)  RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  9. Network Topology (2) RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  10. The offshore switch board Offshore switch Board Allayer chip RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  11. The electronic crate RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  12. N N N N N Network Topology (3) • Solution • barrel shifting • synchronized by global clock N N N N N ~15 Mb/s SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000) Used as data concentratormany input ports  one output port  risks of congestion  intelligent flow control in node ON-SHORE SWITCH: (in: 60x1000, out: 20x1000) Instant ~6 Gb/s PC PC PC PC PC PC PC RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  13. Massively Distributed and Heterogeneous System • Final System: A Network of ~320 Nodes • ~300 Offshore Nodes (MPC860 + VxWorks) • ~20 Onshore Nodes (PC + Linux) • A number of distributed applications running in parallel and communicating • DAQ • L2 Trigger • Slow Control / Acoustic positioning acquisition • L1 trigger ? • Run control RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  14. Software Technologies • Programming Languages • C/C++ for offshore software components • C++ / Java for onshore processing • Java / C++ for onshore display • Distribution • Multiple messaging and data servers (ControlHost) TCP/IP based • VxWorks Zero-copy TCP/IP (~25Mb/s  ~38 Mb/s per node) • Offshore network traffic control (barrel shifting) • Data and configuration storage • Oracle database • Root files • Design & Development • Object Paradigm • UML (Unified Modeling Language) + Extensions • Model-driven engineering RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  15. Interrupt – shared memory communication Data Memory (SDRAM) ARS readout (FPGA) write intrHandler (RTOS) frameSendertask (RTOS) Semaphore(RTOS) read and write Actual frame formatting and sending through TCP/IP socket RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  16. asynchronous method call :ARS_Readout :FrameSender reference to design pattern acq(in frameID:int) Interrupt - SharedMem 6 1 <<RTOS>> FS_Node <<fpga>> AR_Node FrameSender ARS_Read readout framer +acq(in frameID:int) +fillBuffers(id:int) 6 1 frameSender <<reside>> <<reside>> ARS_Readout FrameSender Interrupt – shared memory communicationseen as asynchronous method call RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  17. Implemented in VHDL RealTime::Thread #run() PrmRefType IntrSharedMemSkeleton #run() #installInterrupt() +accept(in pr:PrmRefType) ARS_Read FrameSender FrameSenderSkeleton +fillBuffers(id:int) +acq(in x:Frame) {bind(Frame)} +accept(in x:Frame) IntrSharedMemProxy +doIntr(in x:PrmRefType) Applying the design pattern: model transformation 6 reusablesoftware 1 1 frameSender RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  18. Widespread belief Performance goes against abstraction & flexibility Wrong if the issue is explicitly addressed ActuallyAbstraction  modularity  specific optimization  perf. RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  19. DAQ/SC Board ARS readout/SCLocal trigger106 gatesXilinx FPGA SDRAMMemory 64 MB DAQ/SC Board ARS readout/SCLocal trigger Real-timeembedded CPU System On Chip Xilinx Virtex II Pro?Xilinx Virtex IV Real-timeembedded CPUPPC MPC860VxWorks SDRAMMemory 64 MB Ethernet 100 Serial links (RS485, RS232) Ethernet 100 Serial links (RS485, RS232) Evolutions (1) • Lower consumption • Reduced number of components  increased reliability RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  20. Evolutions (2) • Present scheme: « all data to shore » • Unexpected variable baseline rates: ~100  400 kHz instead of ~70 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  21. Present scheme: « all data to shore » Unexpected variable baseline rates: ~100  400 kHz instead of ~100 Evolutions (2) From 5 to 30 cm/s (!) • Software L1 trigger • Reduced L1 partial data sent from offshore to central L1 workstation • Central L1 computes trigger with global vision of detector • Trigger decision broadcasted through software • Expected gain in data rate: x1000 RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

  22. Slow control Data Clock Energy Conclusions • HEP DAQ and trigger techniques • High data rates • Very weak signal • Highly distributed system • Soft real time constraints • … • Challenges unusual for HEP • No known beam (calibration) • Very embedded system • Very hostile environment • Uncontrolled factors (biolum) • … • Both Robust and Flexible Design • Allows adaptation • Allows for remote evolution • … RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control

More Related