1 / 27

Atlas SCT and Pixel ROD

Atlas SCT and Pixel ROD. Atlas Wisconsin Group Khang Dao, Damon Fasching, Douglas Ferguson, Owen Hayes, Richard Jared, John Joseph, Krista Marks, Mark Nagel Sriram Sivasubramaniyan (Oklahoma), Alden Stradling and Lukas Tomasek (Institute of Physics AV CR, Prague). October 2, 2002.

jerrod
Download Presentation

Atlas SCT and Pixel ROD

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Atlas SCT and Pixel ROD Atlas Wisconsin Group Khang Dao, Damon Fasching, Douglas Ferguson, Owen Hayes, Richard Jared, John Joseph, Krista Marks, Mark Nagel Sriram Sivasubramaniyan (Oklahoma), Alden Stradling and Lukas Tomasek (Institute of Physics AV CR, Prague) October 2, 2002

  2. ROD Overview Notes • FPGA real time data path with DSPs for back-end processing and operation control. • Different firmware and part loaded for SCT or Pixel RODs • 9U VME card, custom back plane. • Back-of-Crate Card for optical interface to modules. • Clock and Trigger provided by Timing Interface Module . • Supports 100 kHz trigger rate without an upgrade. • SCT: 96 input links at 40 MHz • Pixel supports modules with 2 80MHz , 1 80MHz or 1 40 MHz links. • Diagnostic FIFO Memories on board for testing and data monitoring • Register based interface with DSP list processor. • Fiber Optic S-Link to Readout Buffers. • 12 FPGAs, 4.7 Million gates of programmable logic. • 5 DSPs, One for control and 4 (floating point) for calculations. • Cypress Semiconductor VME Slave Interface chipset (A32/D32 transfers).

  3. Formatter and EFB FPGA Router FPGA S-Link & DSP DMA Data RCVR ROD Controller: Operation, Command, Trigger ROD Implementation Model: FPGA Front End / DSP Back End Xon/Xoff Router Halt Output S-link Data Slink on BOC 96 96 Data Valid S-link WR S-link Data 43 S-link Clk Serial Input Link Cnfg, Reset S-link Test R/W Bus Control & Status for Block Xfer/ DMA / S-Link ... DMA Cont. DMA 1 DMA 4 R/W Bus Input Memory & XCVR Control R/W Bus Cnfg, Reset Token and Dynamic Mask Busy DSP Event Trapping, Histograms (Four DSP Chips) Host Port Interface Back of Crate Card / Front End Electronics Control, Status, Mask Event Data/Tirg Type Dec. Trig Count R/W Bus Program Reset Manager Data Bus BOC Setup Bus VME Bus 32 VME Slave Interface Cont / Addr Serial Output Link ROD Busy Timing Interface Module Trig Data (TTC Bus) 48

  4. Data Receiver Implementation FIFO Data Register FIFO In Register FIFO Out Register FIFO Data Register 48 48 Data Input FIFO 4k words Link Input from BOC Link or Diagnostic Data to Formatter Back plane Connector ADV OE RT 48 48 Data Input FIFO 4k words ADV OE Control FIFO Operation Advance FIFO and latch new input Input Memory & XCVR Control Enable FIFO output data R/W Bus Data Retransmit FIFO Read or write data 48 bits are read as 32 bit word bits 0:31 16 bit word bits 32:47 Select 16 or 32 bit part of word

  5. Data Receiver Notes • Input Diagnostic Memories. • Play or record data to perform diagnostics and tests. • Start of record operation is triggered by the ROD Controller. • Trapping (record) of Link Data can be synchronized to triggers or configuration commands. • The controller can retransmit the list in a loop. The retransmit can be issued using the serial inputs (SP0 or SP1) from the DSP, or by a preset counter used in one of the Test Bench modes. Trigger information (from the ROD TIM FIFO) can also be retransmitted so that the ROD can have trigger rates up to 100kHz. • The SCT ROD implementation of the Input Diagnostic Memories is two groups of 4k x 48 bit FIFOs. The Pixel ROD will use 64k deep FIFOs.

  6. FPGA Formatter and EFB Event ID Checking, Dual EFB Design Event Formatting Error Counters Debug FIFO 4k x 39B Formatter 0 Serial to Parallel Conversion Flag Header and Trailer Errors Data Modes: Raw, Condensed, Expanded Header Trailer Limit Busy Limit Derandomizing FIFO 4k Deep Debug Memories Event Fragment Builder FPGA TokenCmd Token30 Gatherer Halt Output Data Input Links 12 Data Valid 32 Token01 46 Halt Output Input Links 12 Formatter 1 16Kx46 FIFO Control Input Links 12 Almost Full Data Valid Formatter 2 46 To/From Router FPGA Input Links 12 Format # Formatter 3 46 96 Input Links 16Kx46 FIFO Control Data Valid TokenCmd Token74 Almost Full Input Links 12 Formatter 4 Halt Output EV Fragment Header/Trailer 32 Token45 Input Links 12 Data Formatter 5 32 Cont header,trailer or data Input Links 12 Format # Formatter 6 Input Links 12 Formatter 7 Event ID Data / Trig Type Dynamic Mask Control, Status, Mask R/W Bus Debug FIFO 4k x 39B Dec. Trig Count Busy R/W Bus Mode Bits To/From Controller FPGA To/From Controller FPGA *Proposed for Pixel ROD: Formatters 0,1,4,5 only

  7. Formatter 0: 12 Links Shown Token from Controller EFB Halt Output Serial to Parallel Conversion and correction of 1 bit errors in header and trailer Record error in data words 32 b 39 b FIFO 256 W by 32 bits 32 b Data to EFB Link 0 7 b Valid Data Valid to EFB Cont Trailer Readout Control Trailer & Word in Busy Trailers to Controller Decrement Trigger Counters Token to Next Link Link 1 12 ea. converters 12 ea. FIFOs and Control Readout controller provides: Readout of data on token Link time out error detection, header generation Data over flow detection Header trailer limit Busy function and limit Dynamic mode bits Value Meaning 00 Normal 01 Skip Readout 10 Mask 11 Dump 1st read 2nd Trailers to Controller ... ... Link 10 Mux Serial to Parallel Conversion and correction of 1 bit errors in header and trailer Record error in data words Link 11 Busy to Controller 2 b Header Trailer Limit to Controller Token from Previous Link Config, Raw ABCD format per link bases Condensed 2 hits per 16 bits & not Config, Raw Not condensed and not Config, Raw Expanded 32 bits for 1 or 2 hits 39 b FIFO 256 W by 32 bits 32 b 7 b Valid Edge Mode Timeout data_to_Large 2 b Cont Trailer Readout Control Link on/off Trailer & Word in Fatal Error to Controller Busy Condensed Mode and Raw Mode Selects Token to Next Formatter Dynamic Mode Bits 2 bits per link R/W Bus Static Link Mask 1 bit per link R/W Bus

  8. Formatter • Formatter FPGAs have separate versions of logic for SCT and Pixel Modules. • Link output FIFOs are implemented with FPGA internal memory. The maximum size of the Link FIFO with 12 links of input data is 256 words deep. • The next version of the Formatter will provide dynamic memory allocation to provide a 512 words deep FIFO if one link from a module is masked off. This modification will combine 256x32 FIFOs on adjacent links in the case that the Static Mask Bit is set. • Individual links can be put in raw, expanded or condensed hit data mode (SCT) by writing the appropriate value to the Formatter Configuration Register. • Correct for loss of synchronization (module data is ahead or behind other modules) in a module (not required but is in VHDL). The dynamic mode bits correct for a link by advancing or withholding the link data with respect to the other links. The values are shown below:Dynamic mode bits: Value Meaning • 00 Normal Readout Mode • 01 Skip Readout of FIFO for this Trigger • 10 Mask Link for this Trigger (full contents of the FIFO will be flushed) • 11 Dump current event, read second for this trigger • The L1 ID number is corrected in the EFB. BCID numbers are ignored.

  9. Debug FIFO Implementation FIFO In Register FIFO Data Register FIFO Data Register FIFO Out Register 39 39 Data OE Input FIFO 4k words Data Input from Formatter Link or Diagnostic Data to EFB ADV OE RT 39 39 Data Input FIFO 4k words OE ADV OE Control FIFO Operation Advance FIFO and latch new input Input Memory & XCVR Control Enable FIFO output data R/W Bus Data Retransmit FIFO Read or write data 48 bits are read as 32 bit word bits 0:31 16 bit word bits 32:47 Select 16 or 32 bit part of word

  10. Debug FIFO Notes • Debug Diagnostic Memories. • Play or record data to perform diagnostics and tests. • Start of record operation is triggered by the ROD Controller. • Trapping (record) of Formatter Output Data can be synchronized to triggers. • The SCT and Pixel ROD implementation of the Debug Diagnostic Memories is two groups of 4k x 39 bit FIFOs.

  11. Event Fragment Builder (EFB) Data 39 Router Halt Output EFB Data Engine 1 Event ID checking Error Counters Format event Event Data 43 Halt Output 16Kx46 Output FIFO Output Control Data Valid Almost Full Format # Data 39 43 EFB Data Engine 2 Event ID checking Error Counters Format event 43 Event Data To/From Router FPGA Halt Output 16Kx46 Output FIFO Output Control Data Valid Almost Full EV Fragment Header/Trailer Format # Word counts Error Summary Dyn Mask & L1-BCO 32 Control, Status, Data Type Header/trailer Event Fragment Generation Data Output Control Engine FIFO 32w x 107b 2 ea. R/W Bus Event Data Valid To/From Controller FPGA Derandomizing FIFO 4k x 16b Read Event Event Data / Trig Type Header FIFO 64w x 32b

  12. EFB: Overview Provides two parallel streams with data processing engine from half of the input links to mask the Formatters link-to-link latency. • Each engine has a 16k word output FIFO for derandomizing of data and to hold large events. • The two data processing engines helps achieve 40 MWords/sec bandwidth to the S-Link. The Event ID,Trigger Type, ECR, and Dynamic Mask are sent to the EFB from the Controller. This data arrives after TIM has transmitted an L1 trigger, Serial ID and Trigger Type data to the Controller and before a FE Command Pulse can be sent to the Formatters.

  13. EFB: Data Engine - 1 Engine Shown Format Data & Count Output FIFO Almost Full Halt Output Data Valid 43 Output FIFO Data Add or subtract from L1 number 39 BCID / L1ID Check Error Detect, Count WR Output Data Data Link 1-48 Format # Mode bits add or sub 1 from L1ID Mask BCID Check L1ID and BCO Error Summary Data lnput bits to error summary Header bit error (0) Trailer bit error (1) Flagged error (2) Sync error (3) Hit pattern error (4) L1ID error (5) BCID error (6) Timeout error (7) Almost full (8) Data overflow (9) If summary AND Mask =True Count = Count +1 Summary Error = Summary Error | current Error Dynamic Mask & L1-BCO Word count to WC FIFO R/W Bus Mask (96 ea.) Summary and Count to Error FIFO

  14. EFB: Data Engine • Correct L1 ID Number • Based on the Dynamic Mask bits, the L1 ID increments or decrements. This is used when a link is resynchronized on the ROD without stopping ATLAS triggers. • BC ID / L1 ID Check: • For Link Header, compare expected BCID and L1ID with the decoded data and flag any errors. All other data types are ignored. • Separate Event FIFOs provide L1/BCID data so each EFB data path can work independently on different events. • BCID check can be globally disabled if required. • Error Detect: • See EFB Data Engine above for error bit definition. • Looks at data bit fields to identify errors. • Generates the error summary block for the event. • One maskable error counter per link. • One global error counter • BCID errors can be ignored for a link that has been resynchronized

  15. Router: Data Format and Trap S-link Data 32 Slink on BOC Event Header Data Read event data and format for S-Link and DSP error data. Detect Errors in the Link Headers, and create error code words for the Error Format. Detect Event Headers to enable trapping algorithms 32 Event Trailer Data 43 DSP Error Data Output FIFO Data Event Data Valid Header Detect Data Type 1 event trapping filters Filters can trap on event types ATLAS, TIM or ROD or error data Each trap has a divided by factor that will reduce data to DSPs Control & Status for Block Xfer/ DMA / S-Link 4 DMA transfer engines. Each engine has a 1k 32b word derandomizing buffer. Each buffer is broken into 256 word blocks. DMA moves full blocks of data. The blocks are filled as shown below: small event word count & continue bit in last word large event word count & continue bit in last word exactly one block word count & continue bit in block 2 last word R/W Bus ... DMA Cont. DMA 1 DMA 4

  16. Router: Overview • Write link and error information into the Link Headers for the S-Link Data Format. • When a link header is detected in the data, the Router moves the link number and error bits information from bits 45 down to 32 and overwrites the L1ID and BCID fields. • Write to S-Link if Data Valid is true, and the User has not set the S-Link Masking bit. • Detect errors in the Link Headers, and generate an error code for each link in Error Data Format (for DSPs). • The error codes are stored in a block of registers for the duration of the Event. When the data from EFB Engine 2 is done in the Router, the Error codes are written into the Event Trailer • Two bits error codes for each link • 00 = no data • 01 = good data • 10 = BCID or L1ID error • 11 = time out error • Router FPGA routes data to the S-Link and the Slave DSPs. • Internal FPGA dual port memory are used to asynchronously connect to DSPs. • During runtime monitoring, data is DMA’d into the internal SRAM of each DSP at 80 Mwords/sec.

  17. Back End DSPs • The Back End DSPs are TI TMS320C6701, floating point processors. The CPU clock is 160MHz, and each DSP has 32MBytes of SDRAM. • The DSPs are removed from the data path allowing them to work independently without affecting the ROD data taking • The VME host can access the Slave DSPs through a 16 bit Host Port Interface. • SDSPs are hardwired for HPI Boot and Little Endian byte order. • The interface to the Slave DSP is through the Master DSP to the ROD bus. • The Slave DSPs respond to D32 block transfers from the VME master. (2.2Mbytes/S) • The back end DSP performs histogramming, spying, and event trapping tasks. • Calibrations: raw data, histograms, summaries to reduce VME traffic and load on crate processor, or comparison to reference histograms. • Can trap specific events, i.e. embedded calibrations. • Uses Trigger Type Info.

  18. ROD Controller: Implementation ROD Controller FPGA FE Command Streams 48 BOC Busy to TIM TIM TIM Data EMIF Bus ASYNC Master DSP TMS320C6201 Event ID Error DSP Serial Ports(1:0) HPI Bus (VME) EFB Event ID/ Dynamic Mask Dec. Trig Count Interrupts Formatter Token and Mode Bits FIFO Occupancy Status Input Memory & XCVR Control PRM FPGA GPI/O R/W Bus to FPGAs, BOC, FIFOs, SDSPs

  19. ROD Controller: Master DSP • The Master DSP is a TI TMS320C6201B, fixed point processor. The CPU clock is 160MHz, and the DSP has 16MBytes of SDRAM (upgradeable to 32MBytes). • The VME host can access the Master DSP through a 16 bit Host Port Interface. • The Master DSP provides the main path of communication to the ROD from the VME through the Host Port Interface. • The Master DSP has 2 serial communication ports (SP0 and SP1) that are used in the ROD to send serial commands to the FE Modules

  20. ROD Controller FPGA To data RCVR From Formatter From Formatter To Formatter To EFB Event ID Data Trig Type Dynamic Mask FIFO Dec. Trig Count Token and Mode Bits Input Memory & XCVR Control Decr Generate Dynamic Mask 96 4 bit Counters Trigger Operation Control r/w Incr All Zero Busy to TIM 48 bit Mask & Input bit by bit Back-of-Crate Card 48 r/w TIM Data Serial Expansion Event Data Control links 48 48 2 input or Circuits 48 bit Mask & Input bit by bit 48 TIM FIFO 4k W 8b ROD BCID, L1ID and Trigger Type in ROD based triggers r/w r/w r/w R/W bus to FPGAs and BOC Back End DSPs R/W Control Registers Host Port Interface (HPI) Bus DSP Serial Port 1 DSP Serial Port 0 Master DSP r/w

  21. ROD Controller: FPGA • The main function of the ROD Controller FPGA is to control the operation of the ROD and coordinate the flow of data on the board in real time. • Listing of the real time functions: • Process commands from the TIM: Trigger, fast and slow commands and calibration signals • Process serial data from the TIM: Event ID and Trigger Type. • Count and store ECR commands, and include the count in the Event Header in the L1ID field. • Send FE Commands to the Modules. • Supply the Formatter Mode Bits and the EFB Event ID and Dynamic Mask Bits to the Data Path when an L1 trigger has been received and the appropriate data has arrived from the TIM. • Count and store triggers, and issue them to the Data Path when ROD is ready to process an Event. • Listing of quasi real time functions: • Issue and count corrective triggers sent to the data path. • Issue Corrective Mode Bits for an Event. • Issue a corrective Dynamic Mask and transfer it to the Default Mask. • Change the masking of FE Command Streams. • Interrupt the Master DSP to indicate the type (periodic reset) of trigger sent to the data path. • Trap Configuration or Event data in the Input Diagnostic FIFOs. • Listing of non real time functions: • Read/Write to the internal memory mapped register set. • Read/Write to the BOC, diagnostic FIFOs, and the Slave DSPs.

  22. ROD Status • The main functions of the SCT ROD are functional except for those stated below. • Functions with errors: • VME block transfer write. • Router to slave DSP DMA (error in one in a million events) • Slave DSP 1 and 2 sometimes stop working. • Hardware/firmware updates needed: • SCT module with only 1 link active need to automatically change the formatter derandomizing memories from 256 words (normal for 2 links) to 512 words automatically. • Update and testing of firmware for Pixel ROD. • Add modes (control of what is allowed in a given state) to the controller. • Software completed that needs to be tested or updated and documented: • Primitive Send Configuration Module configuration from a ROD based structure. (updated, doc and add pixel) • Histogramming of scan data (add pixel and machine language incrementing). • Histogramming control software (add pixel, update and document). • Primitive Histogram Setup sets up variables (changes requested) • Event Manager. Reads data from router and distributes the data to tasks (update,document). • Primitive Task Operation control, basis operation coded (test data return)_ • Primitive Send Data from slaves to host (update reading and documentation) • Primitive Build Stream builds a command stream (add pixels, update and Doc) • Primitive Read & Write Module configuration to ROD bases structure (add pixel, update and doc). • Software not coded: • Occupancy map of events, Error Counting and Link Re-synchronization

  23. ROD Status Availability of RODs 1. Four existing RODs are allocated to Cambridge 1, Oxford 1, Iowa State 1 and Wisconsin 1. 2. Two new RODs will be fabricated by November 15 and allocated initially to pixels 1 and SCT 1. 3. Seven RODs will be fabricated in January and allocated to pixels 3(?) and SCT 4(?) (exact split will be determined at a later time). 4. Production will start after a user evaluation and PRR

  24. ROD Schedule Item Start Date End date mm/dd/yy Fabricated and debug 2 RODs 10/10/02 11/15/02 Test/Evaluate ROD 11/15/02 12/15/02 Fabricate and debug 7 RODs 12/15/02 2/1/03 User Evaluation of ROD, TIM and BOC 12/1/02 3/1/03 FDR/PRR Off-Detector Electronics 3/1/03 Parts Order SCT RODs 3/1/03 6/1/03 ROD Fabrication 95 ea. 6/1/03 9/1/03 ROD Debugging 8/1/03 10/1/03

  25. ROD US ATLAS FDR Summary Introduction: A Final Design Review of the Read Out Driver (ROD) intended for use by the ATLAS SCT and Pixel subsystems was held at LBNL on 20-Aug-2002. The primary intent of this review was to evaluate the present status of the ROD design for use by the SCT and Pixels with a goal of approving the build of 9 pre-production RODs so that further user evaluation can be obtained prior to a Production Readiness Review (PRR) as early as possible. While the goal is for the ROD design to be compatible for both SCT and Pixels, it is recognized that there has been as yet no user evaluation of the design with Pixel readout. Given the schedule constraints of the SCT and the overall cost constraints of the project, the plan is to evaluate the appropriateness of the present ROD design for both SCT and Pixels and decide on the pre-production build knowing that further design modifications may be needed for Pixels after appropriate user evaluation is completed with that readout. Recommendations: The committee was impressed with the amount of work accomplished since the last review. Basic functionality of the SCT ROD has been shown along with the first examples of on-board histogramming.----- etc.

  26. ROD US ATLAS FDR Summary • Recommendations Continued: • ---Therefore, the committee strongly recommends that the design group proceed as quickly as possible to complete layout of the next version of the ROD PCB and fabricate sufficient boards to eventually load 9 boards. Initial build and debug of two boards. Once basic functionality is demonstrated, the remaining 7 boards can be loaded. • The alternate footprint (BGA456 in addition to BGA676) for the Formatter on pixel boards should not be executed. • The allowable power supply margins of the FPGAs and DSPs should be analyzed vs. the regulation spec of the DC-DC converter over the temperature and supply range expected for the board. • Attempt to optimize the FPGA VHDL code as planned to reduce the utilization. • A discussion should be held with the person now re-designing the S-link board for ATLAS reviewing the ROD’s S-link interface vs. those of the S-link to confirm that there are no compatibility issues. • There seem to be legitimate concerns about how the detailed phase of the data streams from the on-detector electronics will be monitored and timed relative to the TIM-generated crossing clock. Since the BOC is the module responsible for this synchronization, it is suggested that this is the correct place to implement monitoring of the timing of the “raw” data streams. The SCT/Pixel off-detector design group should consider how this may best be implemented in the final system.

  27. ROD US ATLAS FDR Summary Recommendations Continued: 7. Since the current ROD design does not meet the requirement to readout events via VME at 1kHz with a full 16 RODs/crate and it would require a major redesign to try to accomplish this, but it appears that one ROD/crate would be sufficient for any planned ROD use, the committee recommends that this requirement be changed to “VME readout at 1kHz for 1 ROD/crate.” 8. The SCT community is strongly urged to devote more resources to write the necessary DAQ software so that more detailed user evaluation can be performed. To expedite this work, it is recommended that one of the existing RODs be sent immediately to Oxford to allow that group to begin learning to use the ROD. The committee wishes to thank the ROD design group for their time preparing for this review and presenting a clear status of the project. Those members of committee who are members of the ATLAS collaboration would also like to thank Christopher Bebek and Henrik von der Lippe for their contribution to ATLAS by spending the time to review this complex development.

More Related