Triggering for B Physics at Hadron Colliders

1. Triggering for B Physicsat Hadron Colliders Erik Gottschalk Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

2. BTeV Trigger Work Breakdown Structure (WBS 1.8) Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

3. 9,999 9,999 9,999 2d 120d 12-1-05 BTeV Trigger Work Breakdown Structure (WBS 1.8) Sheldon’s Symposium (60th BDay) 12-2-05 Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

4. Overview • Introduction and overview of the BTeV trigger • History of BTeV trigger R&D • R&D highlights • Trigger Movie • Outlook Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

5. Introduction to BTeV Trigger (circa 2005) • The challenge for the BTeV trigger and data acquisition system is to reconstruct particle tracks and interaction vertices for EVERY interaction that occurs in the BTeV detector, and to select interactions with B decays. • The trigger performs this task using 3 levels, referred to as Levels 1, 2, and 3:“L1” – looks at every interaction and rejects at least 98% of min. bias background“L2” – uses L1 computed results & performs more refined analyses for data selection“L3” – rejects additional background and performs data-quality monitoringReject > 99.9% of background. Keep > 50% of B events. • The data acquisition system (DAQ) saves all of the data in memory for as long as necessary to analyze each interaction, and moves data to L2/3 processing units and archival data storage for selected interactions. • The key ingredients that make it possible to meet this challenge: • BTeV pixel detector with its exceptional pattern recognition capabilities • Rapid development in technology – FPGAs, processors, networking Note: see glossary at the end of this talk Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

6. 2.5 MHz 500 GB/s (200KB/event) 50 KHz 12.5 GB/s (250KB/event) 2.5 KHz 200 MB/s (250KB / 3.125 = 80KB/event) Block Diagram of BTeV Trigger & DAQ L1 rate reduction: ~50x L2/3 rate reduction: ~20x Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

7. BTeV Trigger R&D The design of the BTeV trigger and data acquisition system was the result of a decade of research and development. Three phases of R&D occurred during this time: • Simulations to establish trigger requirements and develop a baseline design. • Prototyping of algorithms and hardware components to establish performance metrics and determine cost estimates. • Optimizing the design to reduce cost of construction and maintenance, and to address commissioning and operations. Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

8. History of Trigger R&D (beginning in 1998) • 1998 – Algorithms, Simulations • Baseline: UPenn trigger for 3-plane pixel detector (Selove) • Alternative trigger algorithms (Gottschalk, Husby, Procario) • 1999 – Algorithms, Simulations • Digital signal processor (DSP) timing & optimization studies (Gao) • Systolic associative-array track finder (Husby) • UPenn trigger for mixed 2-plane & 3-plane pixel detector (Selove) • Carnegie Mellon trigger for 2-plane pixel detector (Procario) • Megapixel trigger for 2-plane pixel detector (Gottschalk) • New baseline: BB33 trigger for 2-plane pixel detector (Gottschalk) • 2000 – Trigger timing & optimization • BTeV Trigger Movie (Gottschalk) • DSP timing & optimization studies for BB33 (Gao) • Reduction in size of non-bend view pixel planes (Pixel Group) Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

9. History of Trigger R&D (cont.) • 2001 – Trigger timing & optimization, RTES • DSP timing on Texas Instruments TMS320C6711 (Wang) • BB33 algorithm modified for staggered pixel half-planes • Introduction of 8-fold trigger/data-acquisition system “highways” • Real Time Embedded Systems (RTES) Project for fault tolerance and fault-adaptive computing ($5M NSF Grant) • 2002 – FPGA (Field Programmable Gate Array) and DSP hardware • BTeV descoped from a double-arm to a single-arm spectrometer • DSP optimization on Texas Instruments TMS320C6711 (Wang) • L1 timing for commercial-off-the-shelf (COTS) processors (Wang) • L1 FPGA functions implemented and simulated (Zmuda) • L1 prototype system with 4 DSPs (Trigger Group) Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

10. History of Trigger R&D (cont.) • 2003 – FPGA and DSP hardware, L1 & L2 studies • Implementation of BB33 segment tracker in an FPGA (Zmuda) • Tests of the L1 4-DSP prototype (Trigger Group) • FPGA track triplet hash sorter to reduce L1 processing time (Wu) • L1 trigger studies for 396 ns between bunch crossings (Penny K.) • L2 trigger timing improved by a factor of x120 (Penny Kasper) • 2004 – Baseline changes • Baseline change to replace L1 DSPs with COTS processors • Baseline change to replace custom switch with Infiniband switch • Evaulation of Blade Servers for L1 trigger (Wang) • “Tiny Triplet Finder” proposed to simplify FPGA algorithm (Wu) • 2005 – Baseline change • Work on baseline change request to modify L1 trigger architecture Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

11. Trigger R&D Highlights • 1999 • New baseline: BB33 trigger for 2-plane pixel detector • 2000 • BTeV Trigger Movie • 2001 • Introduction of 8-fold trigger/data-acquisition system “highways” • 2002 • L1 prototype system with 4 DSPs • 2004 • Baseline change to replace L1 DSPs with COTS processors • Baseline change to replace custom switch with Infiniband switch Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

12. Eight-Fold Trigger/DAQ Highway Architecture Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

13. Slide taken from trigger status talk (April 21, 2001) Introducing the Eight-Fold Highway… • The DAQ group liked our 8-fold split in the L1 pixel trigger, so they decidedto look into this for the entire data acquisition system and came up with someintriguing possibilities. • Questions: • What are the trade-offs? • What is the impact on front-end electronics? • What if the L1 trigger group wants to use a four-fold split (instead of 8)? • Note: Sheldon wins the naming contest by suggesting the word “highway.” Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

14. Hitachi programming Xilinx programming cable ArcNet serial console PCI test adapter TI DSP JTAG emulator L1 Trigger 4-DSP Prototype System (2002) Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

15. 33 “8GHz” Apple Xserve G5’s with dual IBM970’s (two 4GHz dual core 970’s) Front ends L1 muon Trk/Vtx node #1 Trk/Vtx node #2 Trk/Vtx node #N Ethernet switch Infiniband switch Xserve identical to track/vertex nodes Change to COTS L1 Track/Vertex Hardware (2004) Other detectors 56 inputs at ~45 MB/s each L1 buffers Level 1 switch 33 outputs at ~76 MB/s each GL1 ITCH PTSM network Track/Vertex Farm L2/3 Switch L2/3 Farm 1 Highway Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

16. LHCb Trigger System Frederic Teubert CERN, PH Department on behalf of the LHCb collaboration Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

17. 10 MHz L0:hight pT + not too busy • Fully synchr. (40 MHz), 4ms latency • On custom boards 1 MHz L1:IP + high pT • Ave. latency: 1 ms (max 50 ms) • Buffer: 58254 events 40 KHz HLT + reconstruction • Full detector: ~ 40 kb / evt ≤ 2 KHz Single PC farm ~1600 CPUs LHCb Trigger Overview • 40 MHz crossing rate • 30 MHz with bunches from both directions • Luminosity: 2·1032 cm-2 s-1 • 10 to 50 times lower than @ ATLAS, CMS • LHC rates: (for visible events  at least 2 tracks in acceptance) • Total rate (minimum bias): 10 MHz • bb:~100KHz • Whole decay of one B in acceptance: 15KHz • cc: ~600KHz Pileup system VELO + Trigger tracker Calorimeters + Muon system Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

18. LHCb Level-1 Performance Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

19. Happy Birthday Sheldon! Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

20. Additional Slides Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

21. Trigger/DAQ Glossary DCB Data Combiner Board DDR Double Data Rate FCC Feynman Computing Center FPGA Field Programmable Gate Array GBE Gigabit Ethernet GL1 Global Level 1 Infiniband Third generation high-speed networking standard ITCH Information Transfer Control Hardware L1B Level 1 Buffer L2/3 Package 1 Components needed to complete software for L2 trigger L2/3 Package 2 Components needed to complete software for L3 trigger PCI Peripheral Component Interface PCI-Express High-speed serial version of PCI PCR Project Change Request PP&ST Pixel Preprocessor and Segment Tracker PTSM Pixel Trigger Supervisor and Monitor RCS Run Control System RTES Real-Time Embedded Systems Xserve G5 Apple’s PowerPC based 1U server with dual 64-bit processors Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

22. New integrated upstream event builder architecture Current baseline architecture Change to upstream event builder architecture described in: BTeV-doc-3342 Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

23. Infiniband switch 16-node Apple G5 farm Technical Progress Since CD-1 Review • Modified baseline architecture for the L1 trigger by replacing two of three custom-designed trigger subsystems with commodity hardware. The revised WBS has “8 GHz” PowerPC processors (consistent with IBM roadmap) & Inifiniband switches. • Performed L1 network simulations • Reviewed results in the trigger group • Presented results to BTeV Tech. Board • PCR approved: August 2004 • Purchased and installed 16 Apple G5(dual 2 GHz) nodes and an Infinibandswitch at Feynman Computing Center • Started evaluation of real-timeoperating systems for the L1 trigger • Acquired and installed a 100-nodepre-pilot farm for the L2/3 trigger (located next to the L1 hardware). Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006

24. L1 Trigger 4-DSP Prototype System (2002) Beauty and Charm Decays: A Window on New Physics – May 20-21, 2006