The CERN LHC central timing A vertical slice

1. The CERN LHC central timingA vertical slice Pablo Alvarez Jean-Claude Bau Stephane Deghaye Ioan Kozsar Julian Lewis Javier Serrano

2. Talk layout • LHC Injector Chain Timing • Timing Distribution Overview • Central Timing Overview • Reflective Memory • UTC Time and GPS • The Multitask Timing Generator • The Controls Timing Receiver Module • Safe Machine Parameters Hardware Checking • LHC Software Architecture and Event Tables

3. TI8 Dump SPS CPS TCLP PSB Linac TI2 Dump SPS Dump D3 Dump R2 LHC The LHC Proton Injector ChainStrongly time coupled R1 LHC CNGS

4. ….…. ….…. CERN accelerator networksequenced by central timing generator LHC Experimental area SPS Experimental area CPS Experimental Area PSB

5. CBCM Sequence Manager

6. The LHC Beam start-ramp event The LHC timing is only coupled by extraction LHC Injection plateaux Injection Injection • LSA Beam request: • RF bucket • Ring • CPS batches Extraction Extraction Extraction Forewarning Extraction Forewarning SPS injection plateaux SPS Cycle for the LHC CPS Batch 1 CPS Batch 2 CPS Batch 3 CPS Batch 4 PSB1 PSB1 PSB2 PSB2 PSB3 PSB3 PSB4 PSB4

7. Timing Distribution Overview • RS485 drop nets distribute timing around the CERN accelerator complex • Long distance transmission over optical fibers • One timing network for each accelerator • Hundreds/Thousands of timing receiver modules distributed around the complex • One timing generator drives one timing network

8. A T CODE PAYLOAD Timing Frames Millisecond Millisecond 7 Other frames • Timing frames are Manchester encoded at 500kbit using a 1MHz clock. • Each frame carries 32 data bits, parity, start and stop bits. • One frame transmitted each 125us, 8 per millisecond. • Frames are broken into bit fields • 4 Bits Accelerator [A] • 4 Bits frame Type [T] • 8 Bits Code [CODE] • 16 Bits Payload [PAYLOAD] • Some frame are recognized by the hardware and cause special treatment • Two UTC frames carry the time of day in their payload • Millisecond frames are always sent in phase with the PPS • Telegram frames are stored in double buffers • Event frames cause counters to be loaded and triggered and may produce bus interrupts

9. CBCM Sequence Manager LSA Sequencer “LSA” TI8/TI2 Dump Central TimingOverview LHC User Master ON/OFF LHC Fill Requests: Bucket Ring Batches Request TI8/TI2 Dump CS-CCR-CTMGW FESA API LHC Fill Requests: Bucket Ring Batches Reflective Memory Link Request LHC User Normal Spare LIC Sequence LHC Central Timing CBCM Controlling Injector Chain SPS destination request R1,R2 LHC Timing Inhibits Requests Interlocks SIS TI8/TI2 & SPS Dump Inhibits LSA Master CPS Batch Request 1,2,3,4 SEX.FW1K SPS.COMLN.LSA_ALW CTR HIX.FW1K LSA changes Allowed LIC Timing

10. Reflective Memory LHC Timing generator A LHC Timing generator B Identical except ID event 2Gbit/S Token ring VMIACC-5595 Single Mode Hub Reflective memory: A and B must always be in the same state. If no restrictions for switch over Protects token ring 64Mbyte VMIPMC-5565 Reflective memory LHC Gateway Implements FESA API

11. PPS 10 MHz CTR 1 KHz Delay 40MHz Symmetricom CS4000 portable Atomic Clock UTC Time and GPS GPS Basic Period 1200/900/600 ms Event tables Synchronization module in each timing generator crate GPS One pulse per Second Advanced (100us) One pulse per Second PLL One pulse per Second MTT Multitask Timing Generator MTT Synchronized 1KHz (slow timing clock) Symmetricom XLI Phase locked 10MHz RS485 Timing RS485 Timing 40MHz PLL Phase locked 10MHz CERN UTC Time CERN UTC Time Timing receiver Control System Phase looked 40 MHz Event encoding clock UTC time (NTP or GPS) Set once on startup & on Leap Seconds 25ns steps External events

12. Multitask Timing Generator MTT • Hardware multitasking for 16 tasks • 32 local registers per task • 218 Global registers • 6 Memory mapped IO registers • Timing frame out register • VME P2 in register … • Host processor access to all registers • Implements general purpose CPU • Op-codes are triadic: AND SrcREG,SrcREG,DstREG – AND 0x7,VMEP2,TMP • Arithmetic and logical • Move indexed, literal, register • Wait value, relative • Conditional branch • Interrupt host • Tasks defined from host via Task Control Block • PC • PC Offset • Processor Status Word • Command and status registers allow host access to running tasks

13. MTT hardware module See: The LHC central timing hardware implementation P. Alvarez, J. Lewis, J. Serrano CERN, Geneva, Switzerland This conference

14. MTT External Events Task

15. Controls Timing Receiver CTR Timing 22Bit 50MHz Counters Content Addressed Memory - Triggering • Start • Previous • External • Event • Clock • Previous • 40MHz harmonic • External • Modes • Once • Multi-pulse • Burst • Trigger • Event frame • Wildcard • Telegram • Action • Output • Bus interrupt • Output • TTL/TTL bar • Pulse width Bus Interrupt Hybrid PLL 40MHz Delay Lookup Load Trigger Output Frame Decoder Counter Configurations Loader UTC TGM TGM CTRV – CTRI – CTRP formats

16. BIS Kickers BLM BLM Kickers BLM BIS BIS Timing Network Safe Machine Parameters Distribution If length > 5m SMP @ 10Hz 16Bit 1010 (Flags, E & Int.) Events, UTC, & Telegrams (including SMP) Flags TTL Hw Output Energy A EXP Line driver BEM CTRx @1kHz-24Bit 108 CTRx CTRx CTRx LHC Timing Generator Safe Machine Parameters Controller for LHC BEM EXP Energy B CTRV CTRV BCT “A” CTRV I_beam1 & 2 (CTRV) BCT “B” BIS CTRV CTRV Reads thresholds Reads status LSA Management Critical Settings CTRV CTRV CTRV

17. LSA and FESA • The FESA API is implemented on the LHC timing gateway • Accesses timing generators across reflective memory • Implements • Load or Unload event table • Get running tables list • Set event table run count and synchronization event • Stop or Abort event table • Set telegram parameters • Send event • Read status of tasks and MTT module

18. Event tables • Event tables are ASCII strings sent from LSA to the LHC timing gateway • Each entry in the table contains • The name of the event to send “HX.START-RAMP” • The relative time to wait in seconds before sending the event “1.01” • The payload the event will carry “0x0005” • A carriage-return <CRTN> terminates the entry • The table run count and synchronization event will be filled in later when the table is to be executed

19. Event Table Processing Event Table LSA HX.Start-Ramp 1.01 0x5 HX.Start-Freq 0 0x05 FESA API Event Table template Table Compiler waitr MSFR,1001 movv 0x14020005 EVOUT movv 0x14030005 EVOUT Translation Translate and Merge MTT Program Memory Table Task Garbage Collect Load object Initialize TCB Run task Assemble task Assembler Task Loader Data Reflective Memory Object Code Process Hot Standby

20. Conclusion • The LHC machine processes such as squeezing, physics… may occur at any time and run independently of each other, they share no common basic time period. This has led to a new approach in which machine processes are modeled as independent timing event tasks that can be started, stopped at any time. • Reuse of existing timing boards was facilitated by using FPGAs and writing new VHDL • The LHC timing distributes machine safety critical data around the control system. This has led to the development of hardware to monitor its performance and dump the beam when necessary. • Reflective memory has greatly simplified the implementation of the hot-standby and the timing generators themselves which are no longer required communicate directly with the rest of the control system. This has increased their reliability considerably.