DOM MB Hardware

1. DOM MBHardware CDR and PRR, May 5, 2004 Gerald Przybylski Lawrence Berkeley National Laboratory Lawrence Berkeley National Laboratory

2. Once upon a time, in 1996… • Dave Nygren, Buford Price, Francis Halzen & JPL- Use Stewart Kleinfelders ATWR (SCA) • Two DOMs, deployed Jan ’97 (Author recruited to take ‘em to the pole that year for deployment) • Built around a PIC, a One-Time, EPROM Programmable, Microcontroller, with ADC • Data: Push Mode, FSK modulation • Marginal stability, but it produced! Lawrence Berkeley National Laboratory

3. SwAmp vs. JPL DOMTypical AMANDA Analog OM, JPL Digital optical module 132ns 7.5us Thanks to Tim Miller and Doug Lowder Lawrence Berkeley National Laboratory

4. Lessons Learned • PMT Waveforms ARE Interesting • Software Underestimated • One Time Programmable CPU Worrisome:- No Post Deployment Evolution- Front Loaded Software Development- Not Enough Memory or Speed • Event Rate Limit by Slow ADC in PIC (dead-time) • JPL Version Not Designed to be Scalable • Get to Network ASAP Lawrence Berkeley National Laboratory

5. Scaling up:for AMANDA String 18:40 Optical Modules, January 15, 2000 Lawrence Berkeley National Laboratory

6. Hybrid:Fiber & Digital Compromises/problems… +EIM and Noise Storm Immunity +100 PE linear dynamic range -Fiber ~7 PE dynamic range (without tricks) -Fiber Attrition -PMT HV Power Supply problems Lawrence Berkeley National Laboratory

7. Design Drivers for IceCube • Pulse Fidelity/Waveform Quality • Wide Dynamic Range • Event Timing Accuracy • Low Dead-time • Low Power Consumption • Adequate CPU and Memory • Remote Reprogramability, Software, and Firmware • Off-Board Interfaces: PMT Power, and Flasher Board • Cable Sharing: Bidirectional, Multi-Drop, Power, NO Fiber • Long Lifetime/High Reliability/Cost Tradeoff • Self Identification – No mapping files • Built-in Calibration and Monitoring Features • IceTop & In Ice DOMs Identical Lawrence Berkeley National Laboratory

8. DPRam ADC DAC fADC Digital Optical Module Block Diagram Trigger (2) 10b FPGAandSOPC 1 megabaud Pulser DOR x16 8b Delay ATWD 10b LPF x2 CPU +/-5V, 3.3V, 2.5V, 1.8V DC-DC x0.25 ATWD 10b x 9 x 2.6 10b Configuration Device 8Mbit MUX 40 MHz 32b OB-LED SDRAM 16Mb 16Mb (n+1) LC SDRAM (n–1) 20 MHz Flash Flash CPLD 16b Monitor & Control Oscillator 4Mb 4Mb Corning Frequency Ctl (was Toyocom) Flasher Board 8b DACs & ADCs Lawrence Berkeley National Laboratory 64 Bytes PMT Power 8b, 10b, 12b GTP, LBNL, 26 Mar, 2004

9. Rev4.2 Lawrence Berkeley National Laboratory

10. Design Methodology • Schematic Entry (Mentor) • Layout Tools (Mentor) • Simulations of Hardware subsystems • Simulations of Firmware • Testing in full scale prototype; 4 Major Revisions • Prototyping in isolation • Trade Studies • Interface and Envelope Coordination with UW • Circuitry Reviews; Reliability Reviews; HALT; HASS • Framework Testing Lawrence Berkeley National Laboratory

11. For more information: Hardware: See http://rust.lbl.gov/~gtp/DOMContact Gerald Przybylski FirmwareContact Thorsten Stezelberger TestingContact Azriel Goldschmidt SoftwareContact Chuck McParlandContact Arthur Jones All of whom are at LBNL Lawrence Berkeley National Laboratory

12. Part II: Subsystems • Power conversion and conditioning • Memory: Boot Flash, Flash, and SDRAM • Configurable Logic: FPGA and CPLD • Communications: DOM to Surface, LC • PMT Signal Acquisition • Local Clock • Control, Monitoring and Forensics Lawrence Berkeley National Laboratory

13. Waveform Quality Digitize at the PMT Anode to Overcome Dispersion in Transmission Line - PMT Waveform risetime of a few ns- Cable risetime from 1us to 2us- Improve noise immunity- High ATWD sampling rate, 128 Samples Lawrence Berkeley National Laboratory

14. Wide Dynamic Range • x0.25, x2, x16 gain inputs to ATWD- Achieve around 14 bits from 10 bit part • 40 count SPE = 5mV into ATWD channel 0 • 800 PE = 1V into ATWD channel 2 • PMT saturates at 5 to 6V into 50 ohms - Divided by 4 at input of ATWD channel 2- System limited by PMT saturation! Lawrence Berkeley National Laboratory

15. Event Timing Accuracy • Synchronous Triggering- Requires Delay Line, but- Simplifies Noise Removal- 25ns Course Clock- ~ 3ns ATWD sample- < 1ns resolution from curve fitting • Clock Phase Tracking- < 2ns jitter in 2us rise-time pulse Lawrence Berkeley National Laboratory

16. Low Dead Time • 128 ATWD samples per hit = ~ 420ns- Up to 4 Channels- ~50us to digitize and store data • Two ATWDs per OM- Acquire into B while Digitizing A- or Deployed spare… • Acquire fADC data for > 4000ns @ 40 msps • Digitizing when Local Coincidence “Tag” is present Lawrence Berkeley National Laboratory

17. Low Power Consumption • High Performance, Low Power Amplifiers, Comparators, and Multiplexers • High Performance, Low Power Data Converters- ATWDs, fADCs, and Communications DAC • Coolrunner II CPLD • ARM Hard Core CPU/FPGA • Laptop SDRAM Main Memory • High Efficiency Switcher Power Supplies Lawrence Berkeley National Laboratory

18. Adequate CPU and Memory • SOPC with ARM CPU and 400K Gate FPGAReplaces 256 Pin FPGA + 256 Pin CPU- Wider, Faster Bus… no bottle-necks- Improved speed – power- Bus to FPGA Bridge - for general I/O- FPGA to Bus Bridge - for Event DMA- Dual Port Ram - for Communications Lawrence Berkeley National Laboratory

19. Remote Reprogrammibility • FPGA as Configurable Peripheral Controller- Communications Rx Tx- PMT Event Acquisition- Calibration Controller • CPLD as Glue Logic- Mediates Rebooting- Off-Board Peripheral Interface and Firewall- Serial DAC and ADC interfaces Lawrence Berkeley National Laboratory

20. Off-Board Interfaces • HV Control Pin-out Doubled-up - Enhance the reliability of Ribbon Cable - Digital Control and Read-out Interface • 48 Pin Flasher Interface- Power-Down mode- Just Another Memory Mapped Peripheral- Firewalled through the CPLD- Event initiation on command from FPGA- Flasher Pulse Current Monitor Lawrence Berkeley National Laboratory

21. Cable Sharing • AMI Modulation, demonstrated in String 18 • RAPCal, demonstrated in String 18- Round Trip- Drift Tracking • Two DOMs per DOM Hub Channel + Half the Cables, saving $$$, and Flights+ Half the DOM Hubs, saving Power and Space- Double the Data Rate- Higher Firmware Complexity- Higher Cable Complexity- Higher Deployment Complexity Lawrence Berkeley National Laboratory

22. Cost and Reliability • Industrial Grade, or Better-40°C to +85°C …- Mil-Spec too costly, and narrow selection • Vetted Component Manufacturers • Qualified PCB maker, and Board Loader • HALT and HASS Testing • Manufacturers Reliability Information Studied • STF Testing • Component Cost ~ 10% to 15% more- e.g… 10 x Cost for Corning Oscillator Lawrence Berkeley National Laboratory

23. Self Identification • 64 bit Unique Digital ID in Each Flash Chip • Configuration Building Software- Executed when DOMs are powered up • Plug and Play- A or B identified by Jumper during manufacture.- Any DOM may be assigned A or B- Any A DOM and any B DOM may be installed on any pair. - Cable pairs may occupy any DOM-Hub Input • Ident in PMT HV System, and in Flasher Board Lawrence Berkeley National Laboratory

24. Built In Calibration Pulsers • Front End Pulse Generator- Generates PMT like signals • On-Board Single LED Pulser- Optimized for narrow pulses- PMT Transit Time Calibration Tool • Daughter Board: Bright, Long Range LED Pulser- Geometry Calibration Tool- Shower Simulator • Crude arbitrary waveform pulser circuit Lawrence Berkeley National Laboratory

25. Built-in Monitoring/Forensics • 24 slow ADC channels monitoring- Voltages- Currents- Temperature- Pressure • 8 fast signal monitor inputs- Clock x1, Clock x2, - Communications input, - Upper and lower Local Coincidence, - LED Pulse Monitors • Post deployment failure investigation Lawrence Berkeley National Laboratory

26. IceTop “vs.” IceCube One OM per Pair doubles data rate. Insulation moderates extreme temperature (-80°C) Dual Range operation spans air shower energy range. Active, critical involvement in meeting needs. Lawrence Berkeley National Laboratory

27. The End Lawrence Berkeley National Laboratory

28. XA Block Diagram CPU Side (Stripe) __________ “PLD” Side Lawrence Berkeley National Laboratory

29. Glossary ADC, Analog to Digital Converter AMI, Alternate Mark Inversion ATWD, Analog Transient Waveform Digitizer ATWR, Analog Transient Waveform Recorder ARM, Advanced Risk Machine CPLD, Complex Programmable Logic Device CPU, Central Processing Unit DMA, Direct Memory Access DOM, Digital Optical Module FPGA, Field Programmable Gate Array FSK, Frequency Shift Keying HALT, Highly Accelerated Life Testing HASS, Highly Accelerated Stress Screening JPL, Jet Propulsion Laboratory LED ,Light Emitting Diode PCB, Printed Circuit Board PIC, Programmable Intelligent Computer. PMT, Photomultiplier Tube PSK, Phase Shift Keying RAPCal, Reciprocal Active Pulsing Calibration SCA, Switched Capacitor Array SOPC, System on a Programmable Chip STF, Simple Test Framework SwAmp, Swedish Amplifier for Analog Optical Modules over Twisted Pair Lawrence Berkeley National Laboratory