Tracker Strip and Pixel FEDs

1. Tracker Strip and Pixel FEDs John Coughlan Tracker Readout Upgrade Meeting September 12th 2007

2. Strip FED 9U VME64x 96 optical fibres Analogue ORx 96 ADC channels Limited by I/O Component density 1 nsec timing Board mounted Digital FPGAs Xilinx Cluster Algorithms Raw input data rate >3 GB/s. Processed Output rate < 200 MB/s Cost Driven VME FPGA Front-End data processing FPGA Output to DAQ Opto Receivers Event Builder FPGA 96 ADC channels Modular FE Unit TTC rx Synch APV Power DC-DCs on board Double Sided Board 450 x boards + Slinks

3. Strip FED Board Manufacture High density components. Close up of analogue section on primary side Almost all components on board are Surface Mount. VALUE is in the COMPONENTS Must have HIGH YIELD Design for TEST • Board parameters: • - 9U x 440 mm VME64x form factor • - Optical/Analogue/Digital logic ; 96 ADC channels • Double-sided (secondary side with half of analogue channels) • 6,000 components (majority of passives 0402) (finest pitch < 20 thou) • 25,000 tracks • 37 BGAs (typical FPGA 676 pins on 1mm pitch). All BGAs located on primary side. • 14 layer PCB (incl. 6 power & gnd) • controlled impedance

4. Pixel FED 9U VME64x 36 optical fibres Analogue ORx 36 ADC channels Limited by SLINK output rate 1 nsec timing Digital FPGAs Altera Inputs are ZS Unsynchronised Events on fibres ROC 16 events Build DAQ Events Timestamp TTC Encoded header

5. Pixel FED 9U VME64x boards + Slink Mezzanine cards 9 Analogue, 5 FPGA Altera cards (not ORx) Power Reduces Motherboard complexity 10 layer (Micro vias for Alteras on mezz) Less risk. Testing, repairing easier. Spares Takes more space Connector reliability Signal integrity. Terminations Common Electrical SLINK P3 64 bits @ 80 MHz + TTS throttle FMMs 40 x 9U boards + Slinks

6. Strips and Pixel FEDs • Same Form Factor 9U VME . 450 vs 32 boards • Both Analogue & Digital designs • Same Interfaces ORx, ADCs • Different Constraints on Channel Count ; I/O vs Output rate • Same TTC, SLINK • Same fine clock skew 1 nsec • One board type in each system. • No trigger functions. • Minor differences, power supply, slink connectors, fpga config, vme monitor bus • Different PCB implementations • - Motherboard vs Motherboard + Mezzanines • Main Difference Digital Processing. (Altera vs Xilinx FPGAs) • - Data Inputs. Raw & Synch vs ZS & Unsynch

7. FED Lessons • “Good Design and Robust Manufacturing”. • Design evolves in response to technology (ASICs in TDR) • Invested a lot to get design right first time (2 board versions.) • Protoype to production was a long process. • Started at cutting edge finished with v. mature technology FPGAs (support) • Cost estimates evolve ADCs, FPGAs, Manufacture. Got FPGAs right. • Support of detector development v. underestimated , PMCs cf 9U FEDs • Prototype support underestimated ~ 50 proto + pre-prodn 9U FEDs • Tender process takes a long time. (not needed for pxFED) • Both ended up with “local” manufacturers. Good relationship essential. • Quality Control critical. Early BGA failures. V. High production yield. (also pxFED) No problems ORx assembly • Doing board testing at Assembly Plant took effort but paid off • - Need for Custom Test equipment Optical Testers

8. FED Lessons • System interfaces came late. Electrical SLINK cards, FMMs. More elegant solutions. • Choice of industry standard VME for mechanics, power was good one • VME bus as monitoring is poor , better Ethernet (late design change?) • VME64x features not all used. (plug & play cost pxFED one unnecessary board iteration). Keep it simple. • Think about practical issues cf prototypes with final system , FPGA configuration. • Design issues with cooling. Tests in final configurations necessary. • (Un)expected effects (temperature variation of optical inputs for pxFED encoding) • -Boards are built, debugged and delivered. But Firmware is never finished. • Custom protocols , SLINK, TTC, -> custom Firmware. Duplication of Firmware blocks (across CMS), SLINK, TTC, I2C, VME. • Firmware Libraries? Commercial/Vendor • Duplication of test software? Software effort ? (costings)

9. Future FED Possible Implementation Industry Standards ATCA crates Industry Protocols Data Transmission Serial PCIe … More use of Commercial Firmware cores. Data protocols, memory COTS Carrier Motherboards + CMS Mezzanines, Transition cards ATCA Crate 8U Off Detector sFED Rear Transition Module ORx STTC GBT ORx ORx Cntrl/Mon ORx 12 x 3 Gbps ORx FE ORx ORx Power ORx SNAP 12 SDRAM Buffer FPGA Switch 10G Serial Backplane DAQ Crate Event Builder FPGAs MGBTs GBT PHY MAC Sparsification 1 per ORx ORx and FPGA on Mezzanine? Mezzanine Prototype

10. Future FED-FEC Integrate FEC functions on FED ATCA Crate 8U Off Detector sFED Rear Transition Module sTTC OTx STTC Trigger Throttle ORx Digital Inputs Zero Suppressed FE Data Inputs? Constraint Data Volumes on output ORx 12 x 3 Gbps ORx ORx sAPV Cntrl/Mon Ethernet ORx ORx Power SDRAM Buffer FPGA Switch 10G Serial MPT 12 Backplane DAQ Crate Event Builder FPGAs MGBTs GBT Recv MAC ZS 1 per ORx ORx and FPGA on Mezzanine?

11. Final System Ideas • New sDAQ (sFEDs connected direct to Filter via Super Event Builder Network) • New sTTC (Broadcasting Filter Addresses to FEDs) • Crates • Just Mechanics, Power, Cooling. -> Control/Monitoring via Ethernet. • Serial Backplane based crates (Telecom ATCA , VME46?). • Less Slots (but wider) • Better Power & Cooling ? • Better control & monitoring ? • FED Event Builder Crate

12. Future FEDs • Common Tracker FED h/w probably technically feasible. Practical? • Common CMS FED ? Common SLINK, Common FEC • Go to Digital Input Data • Zero Suppressed at FE tbd • Large systems channel counts. Large form factors (cost driven). • Constraints on Channel count? • Not considered Tracker Trigger • Use of Emerging Industry Crate Standards e.g. Telecomms ATCA (VME) • Exploit Industry Data Protocols e.g. Serial PCIe (SLINK) • More use COTS Vendor Firmware cores, Industry standards • Use common COTS ATCA Carrier boards • with custom CMS Mezzanines (cost effective)

13. Spare Slides