1 / 35

Calorimeter Trigger R&D

Calorimeter Trigger R&D. Calorimeter Trigger Upgrade Hardware R&D Program T. Gorski, W. H. Smith, S. Dasu, A. Farmahini-Farahani, P. Klabbers, R. Fobes, D. Seemuth, I. Ross, M. Bachtis, M. Grothe, M. Schulte, K. Compton, T. Gregerson University of Wisconsin October 4, 2010.

richardpaul
Download Presentation

Calorimeter Trigger R&D

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Calorimeter Trigger R&D • Calorimeter Trigger Upgrade Hardware R&D Program • T. Gorski, W. H. Smith, S. Dasu, A. Farmahini-Farahani, P. Klabbers, R. Fobes, D. Seemuth, I. Ross, M. Bachtis, M. Grothe, M. Schulte, K. Compton, T. Gregerson • University of Wisconsin • October 4, 2010

  2. Hardware R&D Timeline 2008-2010 Data Alignment & Latency Studies on 2x2 Test Fabric Standard RCT FPGA Env Concept. Design S-Link & Serial Link Checkout 100 Base-T, 1000 Base-X Ethernet, TCP/IP Microblaze Processor Checkout Aux Card PCB & Test Fixture PCB Design IPMI MMC Prototype Software TTC & QPLL Checkout TCP/IP JTAG Server/ Client Aux Card Sch. Design 2008 2009 2010 CMC RevA FPGA ML-506 Eval Bd Aux Cards 3&4 First 2 Aux Cards First MicroTCA Crate

  3. Part 1: Hardware R&D Building Blocks • Aux Card: New Hardware Technology/Tools • Aux Card PCB on Mentor Graphics DxDesigner/PADS • FPGA with 3Gbps serial links, S-Link and TTC interfaces • Xilinx firmware tools, embedded (firmware) processing core—Microblaze processor • The MicroTCA Operating Environment • 1000 Base-X Ethernet direct to the FPGA • Running a TCP/IP stack on the FPGA • IPMI MMC support on each MicroTCA Card (AMC Module) • System Alignment with Serialized Links • Synchronous operation of many links with common distributed (LHC-derived) clock • Alignment of cross-connected links at individual FPGAs – “Channel Bonding”

  4. Aux Card Block Diagram JTAG/ PROM Interface PWR Supplies (12Vin) LPC2364-based µController (LANL Matrix card) RS-232 Port TTS LVDS Driver Xilinx XC5VLX110T MGT Link I/O S-LINK 64 LSC µTCA Backplane parallel Full-Duplex MGT Connections to X/R backplane pairs 0-15 Fabric & Link Clocks Clock Distribution CLK1 & CLK3 from BP TTCrx/ QPLL Circuit 40/80/160 MHz 40 MHz to BP CLK2

  5. Aux Card Assembled Board (03/2009) RS-232 GTP Links TTS Pushbutton Reset (for Microblaze) S-Link Connectors TTCrx

  6. Aux Card/S-Link Stackup • 21mm of separation between S-Link and Aux Card board surfaces • TTS, TTC and S-Link connectors just fit • Slightly over the 28.95mm spec (unavoidable)

  7. Aux Card Inter-Board Link Testing (10/2009) Two Aux Cards in Test Fixture Running a 3.125 Gbps GTP Link Test (6 connections—4 passive, 2 through switches)

  8. Rocket I/O Test Program Output (Xmt Ch 4 to Rcv Ch 3) Rx Port Snapshot LFSR Pattern Bd/Ch ID (E4) Infinite Length Starting Seed Register Dump for Block 4 (uTCA port 11) Transmit Block Register Dump: Config: 0x00000013 ParamA: 0x45ECBB34 ParamB: 0x00000000 Length: 0x01000000 Status: 0x00000002 XCntH: 0x00004852 XCntL: 0x0CC2A276 Register Dump for Block 3 (uTCA port 4) Receive Block Register Dump: RCfg: 0x69780000 Config: 0x0000E413 ParamA: 0x45ECBB34 ParamB: 0x00000000 Length: 0x01000000 Status: 0x00000402 RCntH: 0x00004852 RCntL: 0x0998CF36 ErrCtr: 0x00000000 CapExData: 0x00000000 CapAcData: 0x00000000 CapRCtrH: 0x00000000 CapRCtrL: 0x00000000 CapErrCtr: 0x00000000 4-byte words Xmitted 4-byte words Rcvd/Verified Zero Errors

  9. Aux Card R&D Summary • 4 boards built • Subsection tests: • Rocket I/O—all links tested/verified 100% functional at 3.125 Gbps, using local/off-board 125 MHz oscillators • S-Link—tested & verified at burst rates up to 80 MHz • TTS output—tested & verified • TTC input—QPLLs lock, TTCrx broadcast frame decoding tested & verified • Successful demonstration of hybrid Microblaze/HDL design approach • Cards being used as platform for system alignment R&D

  10. MicroTCA Prototype Crate (02/2010) ELMA Enclosure and Backplane NAT MCH uBlade Puma 600W Power Supply

  11. Ethernet R&D: BASE-X Ethernet Development Platform (MicroTCA std) SATA cable running 1000BASE-X Ethernet Performance: Running TCP/IP Stack under Xilkernel— XMT: 4-40 Mbps (traffic) RCV: 15-20 Mbps Virtex-5 FPGAs running Echo server and client under Xilkernel and TCP/IP stack on two modified ML506 evaluation boards

  12. Ethernet R&D: Initial MicroTCA BASE-X Ethernet Test (May/June 2010) PC (Running Iperf Client) ELMA MicroTCA Backplane NAT MCH (BASE-X Switch) (1000BASE-X Ethernet on UTCA Fabric A) (SATA to Fabric A Test Board) (CAT-5 Cable) (SATA Cable) (Network Uplink) BaseT Switch ML506 Board (Iperf server runs on FPGA)

  13. MicroTCA Base-X Ethernet and IPMI MMC Development Board Atmel AVR32 Processor on Mezzanine Card for IPMI MMC Software Development (CMC Rev A) Connector for Linking ML506 to MicroTCA Hub Controller via backplane Fabric A CMC Rev B bare boards (45mm x 40mm), 64-pin PMC Connector

  14. Ethernet R&D: Initial MicroTCA BASE-X Ethernet Test (May/June 2010) • Running lwIP TCP/IP stack under Xilkernel • Connected to departmental network • Test #1: iPerf Xmt/Rcv between ML506 and PC: • Rcv: 14 Mbps • Xmt: 12-19 Mbps • Test #2: Echo server between two ML506 bds • Both bds running server and client app

  15. MicroTCA R&D Summary • Successfully demonstrated GbE (Fabric A) connection between MCH and FPGA • TCP/IP Connection between Linux-based client and FPGA-based embedded server • Bandwidth sufficient for run control operations • Demonstrated IPMI MMC function on a portable mezzanine card (CMC) • MMC arbitrates with the system for basic resources such as power (operates before FPGAs are loaded) • Our board based on Atmel AVR32 Microcontroller • Not trivial to implement! MMC function is “defined” within 4 major specifications and several smaller ones w/ no clear roadmap • Can remotely manage crates from Linux via LAN (ipmitool) • Rev B CMC is reduced size (40 x 45mm), can be made available for wider use in CMS with a generic IPMI MMC programmed into it

  16. The Data Alignment Problem • Mixture of different-length physical links for data sharing with different intrinsic delays • Want all time-correlated data to arrive at correct moment at algorithm interface in each FPGA, regardless of physical connection length • SERDES circuits have some uncertainty on prop. delay FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA Intra-card Sharing Intra-crate sharing Inter-crate sharing

  17. System Alignment R&D • 4 Aux Cards in a 2x2 test fabric • TTC-based timing and link synchronization test bed • Simulates 2 separate crates of 2 cards each • Demonstrate alignment of 48 separate channels all operating on the same timebase

  18. Data Alignment R&D: 2x2 Firmware Test Bed Fabric Ch15: Config Channel (Ring) 0 (master) 1 (slave) 4X (Passive) Ch14: Align Cmd Brdcast 40 MHz 4X “Crate 1” TTCvi “Crate 2” 4X 40 MHz 4X 2 (slave) 3 (slave) 4X

  19. Data Alignment R&D(A. Farahani-Farmahini) • First Step: Use built-in Xilinx “Channel Bonding” function • Works, but forces tradeoff between minimum latency & max allowable pre-align skew between channels • Second Step: Implement custom alignment logic in HDL • Key system goal: longest physical link has lowest possible silicon latency • Custom alignment demonstrated successfully in single board with loopback and simulated physical delays • Ongoing through 2010: Firmware for 4-board, 48 channel alignment • Currently in firmware coding

  20. System R&D: FPGA Firmware Standard Environment Block Diagram (SLHC RCT AMC Board) MODULE POWER FPGA Flash SDRAM (Xilinx FPGA) Pwr Ctl & Monitoring BASE-X Ethernet to Fabric A MAC Core Microblaze Processor (Firmware) or Hard Core SDRAM Ctrllr (Parallel I/O To other On- Board FPGAs) Controller MezzanineCard (CMC) Config (SPI) Processor Local Bus (PLB) Interface IPMI Trigger I/O Interface (HDL) Simple Asynch. Interface TTC & Trigger Pipeline(HDL) High Speed Links carrying Trigger Data, Clocks (Backplane and Front Panel)

  21. System R&D: SLHC RCT Functional Hierarchy Implementation Function Bandwidth Trigger Pipeline HDL HDL in FPGA Fabric Level 1 Trigger Processing Very High (>> 1 Gbps) Timing and Synchronization Custom circuit Board infrastructure, FPGA Fabric HDL (not std uTCA products) TTC Clock distribution, Serial link alignment, L1 Readout control High (10-100 Mbps) Supervisory Processor TCP/IP running on Xilinx FPGA cores (Microblaze, MAC, SDRAM) Function-specific config/control (LUTs, self-test, etc.) Medium (< 10 Mbps) IPMI Controller Mezzanine Card on uTCA Module (AVR32) Power and base configuration control (IP addr, boardd geographical info) Low (<100 kbps)

  22. System R&D: Design Constraint Space Well-Defined Constraints Simplify Constructon and Validation of New Trigger Algorithms Algorithm changes within the constraint space can be efficiently validated prior to deployment Supervisory Interface Physical Link (Data) Infrastructure Trigger Algorithm HDL Design Space Timing Constraints

  23. TCP/IP JTAG Server/Client R&D (D. Seemuth) • Trigger Mode • FPGAs load normal trigger processing algorithms • Ethernet connection used to configure system and monitor operation • Maintenance Mode • Master FPGA loads a special maintenance image from the Maintenance Flash containing JTAG Server • Image contains an Ethernet-JTAG bridge application to drive JTAG chain from FPGA GPIO pins • New Trigger Flash images loaded via JTAG TCP/IP Server/Client connection • Boot Mode of FPGA controlled via IPMI through the CMC • Software project

  24. Trigger Mode JTAG Configuration Example (Trigger Mode--Default) BASE-X Ethernet to Backplane Fabric A Master FPGA (Xilkernel+ HDL) Slave FPGA (HDL-only) JTAG chain viewed from connector FPGA Auto-Load Port CPLD Maint. Flash Trigger Flash JTAG Connector (Flash ld ctl) (Cascaded Flash Config) Boot Mode • Trigger Mode: • FPGAs run Trigger FW • JTAG via board cnctr • (Xilinx cable) • Full JTAG chain Controller MezzanineCard (CMC) Trigger Flash IPMI

  25. Trigger Mode JTAG Configuration Example (Maint. Mode) BASE-X Ethernet to Backplane Fabric A Master FPGA (Xilkernel+ HDL) Slave FPGA (HDL-only) GPIO-based JTAG interface (FPGA-based server) FPGA Auto-Load Port CPLD Maint. Flash Trigger Flash JTAG Connector (Flash ld ctl) (Cascaded Flash Config) Boot Mode • Maintenance Mode: • FPGA runs Maint FW • JTAG via LAN to FPGA- • based server • Limited JTAG chain Controller MezzanineCard (CMC) Trigger Flash IPMI

  26. Hardware R&D Part 1 Summary • Activities Focus on 3 General Areas: • Designing & building boards with large FPGAs and high speed serial links • Aligning a system built of cross-connected serial links to support Calorimeter Trigger Algorithms • Understanding how to effectively operate, maintain and reprogram this system in the MicroTCA crate context • Our goals in these areas have been met, allowing us to take the next step….

  27. Hardware R&D Part 2: Calorimeter Trigger Prototypes • 2012 Shutdown Target: Calorimeter Trigger Slice Crate Project • MicroTCA Crate • Covers the 8φ×28η region of 1 current RCT crate • Accepts optical inputs with HCAL and ECAL trigger primitives • Simultaneous upgrade of RC Mezzanine Cards on 1 RCT crate to optical inputs • Test in 904, deploy as a pilot project during shutdown • Simultaneous operation of existing RCT with new crate • Two main board designs: Optical Receiver Mezzanine (ORM) and Calorimeter Trigger Prototype (CTP), plus other support boards

  28. Calorimeter Trigger Evolution Step 2: ↓ OR ↓ Step 1 (2009) Step 3 Step 4 ETCC:TPGs HTR:TPGs uTCA-HTR:TPG ETCC:TPGs HTR:TPGs ETCC:TPGs uTCA-HTR:TPGs ETCC:TPGs uTCA-HTR:TPGs oSLB oSLB SLB SLB SLB oSLB oSLB oSLB oSLB RMC RMC RCT RCT RMC oSLB oSLB oSLB RCT oSLB RCT/ uTCA RCT/ uTCA Matrix& AuxCards  Cu GCT/uTCA GCT:Sources GCT:Sources GCT/uTCA GCT:Sources GCT/uTCA GCT/uTCA FO GCT:Main GT/GMT GT/GMT GT/GMT GT/GMT

  29. RCT-side Optical Receiver Mezzanine (in collaboration with LIP) Latency for Current 7216-based Link: 3.4 bunch crossings for cable (85ns) 0.8 bunch crossings for 7216 Tx (19ns) 2.5 bunch crossings for 7216 Rx (62ns) TOTAL: 6.6 bunch crossings (166ns) Measurements with Rocket I/O suggest that an optical-based link would require about 9 crossings in a best-case scenario Optional Repeater Xmitter 120.24 MHz Clock SFP Xcvr Spartan6 FPGA SFP Xcvr 120 MHz Synchronous Parallel Data to Phase ASIC Incoming TP (2.4 Gbps raw) (from J. C. De Silva)

  30. Calorimeter Trigger Prototype (CTP) Card Backplane Side Front Panel Side Secondary Power Supplies 12-Channel Optical Receiver SDRAM GTX Links (4.8Gbps) on Ports 4-11 for η-sharing 2.4Gbps x 32 Trig Prim. Links 12-Channel Optical Receiver Front End FPGA XC6VHX250T 8x8 Region Processing FPGA XC6VHX250T 12-Channel Optical Receiver TTC/DAQ to Aux SFP Optical Transceiver Fabric A GbE CMC Summary I/O Links Link Clock Conditioning Circuitry SFP Optical Transceiver IPMI

  31. Slice Crate (8φ×28η) Output to GT Clock/Control from TTC Region Output Links for Summary S-Link to DAQ Ethernet Uplink(s) MicroTCA MCH+Fabric CTP Input Card CTP Input Card CTP Input Card CTP Input Card CTP Summary Card UWHEP Aux Card Fabric Suppt in MCH Slot HCAL/ECAL TPGs From TP Crates or RCT ORM cards

  32. SLHC Cal Trig Crate—1 of 7 (Barrel+Endcap+HF) Output Links from 6 Region Crates to Summary Crate (Crate 7) Clock/Control from TTC 2nd-Level Cnr/ ΦSharing to other crates Crate Output to DAQ Ethernet Uplink(s) MicroTCA MCH Input Card Input Card Input Card Input Card Processing Card Processing Card Input Card Input Card Input Card Input Card TTC/DAQ Card HF Front and Back TPGs Corner-sharing Links to Input Cards in other Crates Φ-sharing Links to Input Cards In other Crates HCAL/ECAL TPGs

  33. Slice Crate R&D Summary • ORM Mezzanine replaces current mezzanine on RCT Receiver Cards (one RCT crate) • Raw data rate of 4.8Gbps on two 2.4Gbps links • Dual 2.4Gbps links supports Calorimeter geometry and keeps FPGA cost low • Link rate excellent fit for synchronous operation to keep latency impact to a minimum • 56 Cards to retrofit one RCT crate • CTP Card • Covers 8x8 tower region • Receives HCAL/ECAL Trig Primitives on 32 2.4Gbps links from ORM cards or Trig Primitive crates • Two firmware images: one for 8x8 region processing, and one for final processing of the 3 subregions

  34. Slice Crate R&D Summary, cont’d • Slice Crate: • One crate covers 8φ×28η • ~½ SLHC Cal Trigger Crate, narrowed in φ to match boundaries of current RCT crate • Other new support cards: • ORM test carrier board (AMC module) • MCH tongue 3&4 fabric boards (passive or simple switches) • Goal is to validate in 904 and deploy one crate to Pt. 5 during 2012 shutdown

  35. Personnel Responsibilities • T. Gorski • Lead Engineer, Schematic Design, PCB Design, AVR32 software (IPMI), HDL design • R. Fobes • Electronics Technician, PCB Design • Farmahini-Farahani • HDL Design, Embedded software (FPGA) • D. Seemuth • JTAG Server/Client Software, Embedded software (FPGA)

More Related