1 / 16

Modular Data Acquisition Introduction and applicability to LCLS DAQ

Modular Data Acquisition Introduction and applicability to LCLS DAQ Michael Huffer , mehsys@slac.stanford.edu Stanford Linear Accelerator Center December 14, 2006 Representing : Ryan Herbst Chris O’Grady Amedeo Perazzo Leonid Sapozhnikov Eric Siskind Dave Tarkington Matt Weaver.

forest
Download Presentation

Modular Data Acquisition Introduction and applicability to LCLS DAQ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Modular Data Acquisition Introduction and applicability to LCLS DAQ Michael Huffer, mehsys@slac.stanford.edu Stanford Linear Accelerator Center December 14, 2006 Representing: Ryan Herbst Chris O’Grady Amedeo Perazzo Leonid Sapozhnikov Eric Siskind Dave Tarkington Matt Weaver

  2. Outline • Introduction • Concepts • Architecture • Implementation • Examples… • Petabyte scale, low access latency storage for SLAC Computer Center • LSST camera data acquisition system • Application design • Discuss applicability for LCLS Data Acquisition?

  3. The Module • Is the basic building block of the architecture • Specified as: • A hardware design (schematics, BOM & layout guidelines) • A series of base services implemented as: • VHDL (interfaced through core IP libraries) • Software (OO interface - provided through header files and shared libraries) • documentation • Module neither specifies or constrains application’s physical partitioning model • Architecture specifies three different types of modules • CEM (Cluster Element Module) • Provides a processor + RTOS (the Cluster Element) • Provides many channels of generic, high speed, serial I/O • Provides commodity network interface (10 GE & 100-Base-T Ethernet) • fCIM (Fast Cluster Interconnect Module) • Provides 10 GE connectivity for up to 64 Cluster Elements • sCIM (Slow Cluster Interconnect Module) • Provides 100 Base-T & 1 GE connectivity for up to 64 Cluster Elements

  4. Each lane operates up to 10 Gb/sec May mix and match lanes to each CE Common to both elements To sCIM To fCIM Cluster Element Module (CEM) PHYs (4-20) PHYs (0-16) • Two variants… • One channel CE • Two channel C2 A Cluster Element (CE) is a processor • Two variants… • footprint: • ~ 50 cm2 • power: • ~ 7 watts total + • ~ 3/4 Watt/port CE CE CE reset options 10 GE 100B-T JTAG reset reset options 10 GE 10 GE 100B-T 100B-T JTAG reset CEM (1 channel) CEM (2 channel)

  5. To CE Supports a variety of electromechanical standards X2 & XENPACK MSA CX4 Long haul & short haul fibers To management network Fast Cluster Interconnect Module (fCIM) Is a collection of managed switches 10 GE (0 – 8) • footprint: • ~ 144 cm2 • Power: • ~ 1 ½ Watt/port • 64 elements ~ 110 watts fCIM 1 GE 10 GE (0 – 8)

  6. To CE To management or control network 1 GE Slow Cluster Interconnect Module (sCIM) 100B-T (2 – 64) • footprint: • TBD (less then fCIM) • Power: • TBD (much less then fCIM) Is a collection of unmanaged switches sCIM Supports a variety of electromechanical standards

  7. fCIM is managed control network CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE 32 Element Cluster 1 GE To management or control network sCIM fCIM To data network 10 GE

  8. Fabric clock Right side MGT clock Right side Memory (512 Mbytes) Micron RLDRAM II Right side Configuration memory 128 Mbytes) Samsung K9F5608 Right side Multi-Gigabit Transceivers (MGT) 8 lanes Left side MFD Reset options JTAG Reset Left side 100-baseT CEM block diagram Right side PPC-405 (450 MHZ) FPGA (SOC) 200 DSPs Lots of gates Xilinx XC4VFX60

  9. Base services provided by CEM • “Fat” Memory Subsystem • 512 Mbytes of RAM • Sustains 8 Gbytes/sec • “Plug-In” DMA interface (PIC) • Designed as a set of IP cores • Designed to work in conjunction with MGT and protocol cores • Bootstrap loader (with up to 16 boot options and images) • Interface to configuration memory • Open Source R/T kernel (RTEMS) • 10 GE Ethernet interface • 100 base-T Ethernet interface • Full network stack • Utility software to manage I/O

  10. Extended services provided by CEM • Pretty Good Protocol (PGP) • Physical interface is serial with 2 LVDS pairs/lane) • Point-to-Point connectivity • Allows clock recovery • Full duplex • Symmetric capabilities in either direction from either end • Provides reliable frame (packet) transmission and reception • Deterministic (and small) latency • Lightweight “on the wire” overhead • Specifies 4 VCs in order to provide QOS • Implemented as an IP core • Small footprint • Interface hides user from protocol details and implementation • Implemented on CE (through the conical model described above) • Asynchronous • Extensible in both bit-rate and # of lanes • Flash Memory Module (FSM) • Provides as much as 256 Bytes/CE of persistent storage • Low latency/high bandwidth access(1 Gbyte/sec) • Interfaced using PGP

  11. PGP 1 lane @ 250 Mbytes/sec PGP core & interface FSM FSM FSM FSM CE Cluster Element as used in petacache 65 Gbyte flash memory (Flash Storage Module) Application specific Called a SAM (Storage Access Module) 10 GE 100B-T From management network To/From fCIM To/From sCIM To client nodes on client network

  12. PGP (fiber-300 M) 1 lane @ 300 Mbytes/sec PGP core & interface Raft Readout System Cluster Element as used in LSST DAQ In cryostat Services 9 CCD mosaic 288 Mbytes/sec (Replicated 25 times) Application specific CE Called a RNA (Raft Network Adapter) 10 GE 100B-T To/From fCIM To/From sCIM To client nodes on DAQ network From Camera Control System on CCS network

  13. 1U Air-Outlet 1U Fan-Tray 8 U 1U Air-Inlet The “Chassis” High-Speed Network Card (8U) Passive Backplane X2 (XENPACK MSA) Accepts DC power Daughter board Card (4U)

  14. Guider Array (2) WFS Array (2) Chassis Physical Interfaces (19”) client network card 10 GE CCS network card 1 GE to odd raft to even raft Science Array (12) bank 8U Number is TBD to CCS Daughter cards replicated twice for: Redundancy & simulation to DAQ network

  15. A prescription for application design • Partition problem into three domains: • Device/sensor specific Read-Out (RO) • Device/sensor monitoring and configuration • Data transport and processing • Define a consistent and regular interface between RO & CE systems • independent of device/sensor • Define CE customization • How many lanes of I/O necessary between RO and CE? • What are the protocols on these lanes? • Specify data processing • How should this processing be partitioned between software and hardware? • CE number • What is the underlying, inherent, parallelism of the data (if any)? • How many CPU cycles and gates should be dedicated per data byte? • processing effort/byte • Define physical partitioning of design • How many boards? • What type and number of modules on a board? • Incorporate with custom logic? The later two are within the realm of the CE

  16. Many different types of devices • Physically separated • Processing/byte/device is low RO RO RO RO CE RO RO RO RO • Many different types of devices • Physically separated • Processing/byte/device is high CE CE CE CE CE CE CE CE RO • Homogeneous devices • Perhaps physically separated • Processing/byte is high Typical usage patterns

More Related