Performance of the CMS Silicon Tracker Front-End Driver

1. Performance of the CMS Silicon Tracker Front-End Driver R. Bainbridge, E. Corrin, C.Foudas, J. Fulcher, G. Hall, G. Iles, J. Leaver, M. Noy, M. Raymond, O. Zorba Imperial College D. Ballard, I. Church, J.A.Coughlan, C.P.Day, E.J.Freeman, W.J.F.Gannon,R.N.J. Halsall, M. Pearson, G. Rogers, J. Salisbury, S. Taghavi, I.R.TomalinCCLRC Rutherford Appleton LaboratoryI. ReidBrunel University 10th Workshop on Electronics for LHC Experiments and Future Experiments Presented by Greg Iles: gm.iles@imperial.ac.uk LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

2. Off Detector (counting room) Optical links transmit equivalent to 1.3 TB/s @ 100kHz trigger rate 440 Front End Drivers (FEDs) On Detector: 9M silicon strips 73k APV25 readout chips Analogue readout via 43k optical readout links ADC ADC FPGA FPGA x12 x12 ADC ADC APV readout chip APV readout chip 128:1 128:1 Microstrip Tracker readout chain DCU AOH APV MUX 2:1 x96 9U VME back plane PLL FED Front End Module Detector x8 FPGA S-link card DAQ Throttle signals FMM RAM Transition card LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

3. Front End Driver (FED) 96 optical fibres inputs, each a multiplexed pair of APVs 8 front end blocks each driven by a 12 way optical ribbon cable Raw input data rate (all 96 fibres) = 3.4GB/s. Output rate down slink = 50MB/s /% occupancy VME FPGA Front-End data processing FPGA S-Link Back End “System” FPGA FE Unit Power LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

4. Front End (FE) Unit on FED Opto-to-electrical conversion Digitise & sync data Find hit clusters Optical ribbon cable input Opto-RX, 12 way 3 x Delay FPGA (ADC clk timing) Virtex II, 2M gate FPGA performs signal processing Analogue circuitry duplicated on secondary side 6 x Dual 40MHz, 10bit ADCs 12 x Buffers De-multiplexed fibre channel = APV Data Frame • To extract hit need to perform: • Common mode subtraction • - Pedestal subtraction • - Cluster finding • - Sync checking Digital header 128 analogue values (one for each microstrip) Signal magnitude MIP Time LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

5. At LECC ’03 in Amsterdam Start ‘03: The first two FEDv1 boards were manufactured June ‘03: After testing showed there were no major faults a further 3 FEDv1 boards were produced Progress in the last year Sept. ‘03: Further 6 boards had serious problems Start ‘04: A further batch of 6 boards manufactured and assembled at different company Spring ‘04: FEDs distributed to CERN, Pisa, Lyon June ‘04: Beam test at CERN Sept. ‘04: FEDv2 should return from manufacture. End ‘04: Manufacture a further 20 FEDv2 assuming no surprises. FED status The Manufacture of the CMS Tracker Front-End Driver Poster by John Coughlan Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon Fulcher LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

6. CERN X5 Area 4 FEDs 252 fibres 65,000 strips 2 FEDs slink Provided excellent opportunity for system integration Large complex system useful for finding system weak points. Beam Test June ‘04 Beam LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

7. Testing in the lab • FED • Tested with FED Tester Ensemble • FED Tester Ensemble • Drives all 96 FED channels with data similar to that expected in CMS. • 100kHz Poisson L1A. • Also provides clock, L1A and throttling from built in Trigger Control System (TCS) • An ensemble is made up of 4 FED Testers. FED under test sandwiched between 2 additional FEDs and crate closed to simulate airflow & temperatures in fully populated crate. VME access via SBS 620 PCI-VME bridge Master FED Tester: Provides clock and control to additional 3 FED Testers LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

8. The FED Tester Provides 24 optical channels & Trigger Control System (TCS) Data for 3 channels loaded into FPGA. Converted to analogue form by 3 DACs. Cross-point switch controls distribution of the 3 unique channels to the 24 channels. 8 three channel TOB type AOHs convert the electrical signal to optical signals. Temp of AOHs controlled to +/-1ºC Optical outputs Fibre spools AOHs VME FPGA x-point switches and buffers on back side System FPGA DACs Master & slave I/O Power LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

9. Laboratory test set-up Access VME with SBS620 PCI-VME link VME crate PC FT (slave) FT (slave) VME crate FT (slave) J1 Connector FT (master) J2 Connector Simulate Local Trigger Control System Merge 96 fibres into 8 ribbons of 12 fibres Throttle signals from FED to FT Clock & L1As from FT to FED Slink PC (PCI-X slots) J0 Connector Generic PCI Card Slink-Tx Slink Rx LVDS cable FED LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

10. Carrier for slink transmitter Buffers control and data signals Buffers throttle signals FED v1 and V2 compatible Status Returned from manufacture in August. S-link verified Throttle signals still under test Slink transition card FED VME Backplane DAQ – Slink Transmitter Slink data and control signals Throttle signals to Fast Merge Module (FMM) LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

11. Check Verify data transmitted by the FED via slink is not corrupted. Set-Up FED sending test patterns (scope mode, sample size 6) 100 kHz repetitive trigger CPU can only verify data transmitted at ~28 Mbytes/sec Hence FED asserts “BUSY” and trigger rate falls to ~17.1 kHz Results Verified 1 TB of data in 10.8 hrs No errors No errors observed so far, however it would take 146 days to guarantee no more than 1 error per week in CMS. However.... In rate test with transition card borrowed from another sub detector (no signal buffering) we did observe occasional errors. More tests needed. Add CRC check. Slink verification LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

12. FED configuration Scope mode, repetitive triggers at 100kHz. Varied scope sample length to vary event size. Results & Conclusions Maximum transfer rate = 469MB/s Observed S-link receiver exerting back pressure for events > 4.88kB PC rather than FED setting upper limit. Switched to random triggers at 100kHz OK in scope mode & zero suppressed mode Requirements for CMS 200MB/s Average 400MB/s Peak Slink max data throughput LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

13. Temperature Monitoring Max temp of OptoRx = 70°C Simple air deflector in centre of card lowers temperature in hottest region by ~10°C OptoRx #0 Air deflector OptoRx #7 Measurements from on-board sensors and thermocouples Fan direction LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

14. Where are Tracker buffers ? Laser Driver APV Front-End FPGA 1 Mux FE Buffers, 2kB, ~125 ZS events Chan 1 Buf 1 BE Buffers, 2MB, ~1000 ZS events Quad Data Rate SRAM. Handles 640MB/s in/out simultaneously. 40 MS/s @ 10bit Bus 1 Back-End FPGA Buf 12 Chan 12 BE Buf 80MB/s APV Buffers, ~10 in decon mode controlled by APVE Front-End FPGA 8 Chan 1 200 MB/s Avg 400 MB/s Peak 640 MB/s Unlimited Buf 1 Bus 8 Head Buf 40 MS/s @ 10bit Buf 12 Header Buffer Chan 12 LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

15. Buffer tests • Check FED buffers can handle high rates & occupancies • Want to test “unconstrained” FED • Don’t want slink back pressure limiting results • Slink throttle disabled. • Data sent to slink oblivion • Ideally would like all FED buffers to assert “Busy” or “Warn” when becoming full. • If buffer overflow inevitable -> Detect the event, but ignore data • Set flag to indicate data loss and record the number of these events • Not possible yet, although BE buffer can assert throttle signals • Use different approach. Not perfect, but still informative. • No throttling used • Count number of triggers before FED hangs LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

16. Buffer test results Conditions: Data taken in Zero Suppression mode Single strip clusters used to create largest event size possible and thus worst case. APV frame occupancy kept constant. - e.g. 6 strips = 4.7% occupancy - Easier to understand results. - Large buffers -> Valid approx Performed 5000 "tests“. Each comprised 100k triggers, 100kHz Poisson distributed Results: FED handles single strip occupancy up to 6.25% (8 single strip clusters) LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

17. FED Tester Ensemble Sending isolated hits (i.e single strip clusters) 2x106 Poisson L1As @ 100kHz FED Currently transmits 48MB/s of debug information in zero suppressed mode in addition to 32MB/s of status information. Slink Limited to average rate of 248 MB/s CMS Maximum = 3% occupancy, but large cluster width > 1 Deadtime Versus Occupancy LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

18. Cluster algorithm Following is simplification, however: 1 byte pos 1 byte width 1 byte for every hit strip Calculate average event size Make some simplifications Ignore data padding for byte aligned data Ignore 2 dead channels Beam test June 2004 Average cluster width = 1.29 strips Average cluster event size = 3.29 bytes Cluster width extrapolation LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

19. FED v1 Commissioning tests at RAL, Imperial & CERN continuing well. Baseline firmware and software operational. FED v2 and S-link Transition card Both back from manufacture in August ’04 Only preliminary results so far, but everything seems OK Future System integration Calibration of Tracker, Databases, Large system issues (e.g. error handling) Production issues Tender, Industrial testing Continued testing Beam test in October, Continued testing with the Fed Tester. Commissioning Start to plan Summary LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles

20. More information... The Manufacture of the CMS Tracker Front-End Driver Poster by John Coughlan Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon Fulcher LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles