Liaison Training Seminar

1. Liaison Training Seminar Brought to you by the BaBar/PEP-II Liaison Team (First Version, Sep 2000 TIMeyer)

2. Outline • The job • Accepting backgrounds • Mice, EPICS, and other clicks • Hands-on tutorial

3. Being a Good Liaison • READ the online shift manual • http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/ • SIGN UP for Liaison hypernews • http://babar-hn.slac.stanford.edu:5090/HyperNews/get/babar_pepii.html • VISIT two (2) shifts before your first for hands-on experience • ATTEND this training workshop • BE smart while on shift; DON’T pass the buck • GIVE feedback to other Liaisons and Liaison Coordinator(s)

4. Job Description

5. Job Details • Shifts are taken at MCC, Bldg 005 • Answer x5121, call x5255 • Call/page experts when appropriate • Use your toolbox to stay up-to-date • Make notes in e-log • SWING should attend 3:30pm IR2 meeting • Bring next shifter up-to-date

6. PEP-II Hierarchy • Typically 3-4 PEP-II operators on a shift • 1 or 2 for PEP-II • 1 for Linac/FFTB • 1 “EOIC” – engineer operator in charge • This is the person with whom you should speak most often • They determine short term schedule • They know a LOT • 1 Program Deputy (PD) who manages weekly goals • An accelerator physicists, and very friendly • Ask them technical question, but be prepared to answer some in return!

7. Responsibilities • Protect BaBar • from user errors • from background conditions • Communicate • Maximize integrated luminosity • Help understand and mitigate backgrounds

8. Why Protect BaBar? • Machine backgrounds at PEP-II are serious • High b/c of unprecedented currents and luminosities • Consist primarily of • lost particles scattered by beam-gas interactions which then shower into detector • Low angle radiatiove bhabhas which scatter off machine elements into detector • High backgrounds can cause • Detector deadtime / reco ineffeciencies • Radiation damage

9. Communication • Liaison is primary source of information • For BaBar about PEP-II • For PEP-II about BaBar • Responsible for • Coordinating downtime and repair work • BBR or PEP won’t always TELL you; you need to ask • Answering questions • Condensing, distilling, and CONTROLLING information • Writing it down in the e-log • The Liaison makes BaBar and PEP-II a team

10. What PEP-II Will Ask • PEP-II really appreciates Liaison presence • Backgrounds • Are they okay? • How are they measured? • What is their impact? • Beam aborts • Who did it? • Why? • Luminosity • What is it? How is it measured? • How much is it • BaBar detector and physics • Why can’t we inject? • Anything else…

11. What IR-2 Will Ask • IR2 wants to know, too • Are backgrounds okay? • What “just happened?” • What’s going on over there? • What happened to beams? • Who aborted the beam and why? • What is taking so long? You’ll soon discover that true DIPLOMACY is required…answer the important questions without bothering too many people, and keep both sides synchronized and content.

12. Who did the Beam Abort? • This is an important question whose LACK of an answer is sure to irritate many, many people. LEARN HOW TO TELL. • Only the SVT diodes will abort the beam from the BaBar side • Check the SVT Diode EPICS panel for a REDTrip light (more on this later) • PEP-II keeps a running logfile of Beam Abort Triggers on a monitor on their side of the room • Ask another Liaison if you don’t know it • Ask operators per “MPS CUD” and/or SLCX9 or MCCX38 • YES, you’ll have to move from your chair to read that display • Key to messages you’ll see • SVT_H(L)ABT : SVT diode beam abort • EMC_H(L)INH : EMC_TOTL H(L)ER injection inhibit • BBRHVH(L)INH : BaBar !INJECTABLE injection inhibit • MCC_ABORT : PEP-II op big red button abort • KLYS_xxxxx : RF trip • BOAT : Bad Orbit Abort Trigger • Don’t BLAME the operators; share, and work on it together This is the only abort that BaBar can cause, and it should show up in BaBarSVTBackground EPICS

13. Backgrounds

14. Backgrounds at BaBar • High rate of backgrounds can threaten • SVT, DCH, DRC, EMC, IFR, TRG • Several subsystems have dedicated background monitoring and protection systems • It is the Liaison’s job to care and feed them or contact an expert when appropriate • Different systems sensitive to different things • Sensors are not generally correlated!! • When are backgrounds typically the highest? • injection

15. Backgrounds Differ Daily • Plot shows bkgs during StableBeams for 1 SVT diode as compared to best bkgs in July,2000 • Note that “ratio” decreases, but a LOT of noise, and lately, even central value getting worse! Scrubbing of the vacuum August ROD Vac leak

16. What To Do • What does high background mean? • “Pain thresholds” are defined for each detector such that running within 5% of them is OKAY • “Top of the stripchart” = pain threshold • When backgrounds are high • Verify that it makes sense • Be sure that not just injection backgrounds • Observe what PEP-II operators are doing • If situation persist for more than 90 seconds, ask operators nicely about it (be armed w/info!)

17. SVT Diodes : Intro • System of 12 PIN diodes & dedicated hardware • Monitor and control radiation exposure • 4 MID-plane diodes • Receive highest doses (5-10x top/btm diodes) • Are in a special ABORT circuit which aborts beam • 8 TOP/BTM diodes (more precise) • Signal in mR/s (~5-20 mR/s) • Requires exact knowledge of pedestal • Temperature sensitive • Changes with radiation damage, too Biggest source of error is pedestal subtraction!

18. SVT Diodes : Summary Panel Latched trip status Can be reset under DetailsTrip History See this menu for: Calibrations FastHistory Captains Panel

19. SVT Diodes : Trips • Frontend hardware offers “fast” protection • Typically visible on stripcharts, but not always • After “fast” trip, post-mortem buffer read out to disk which will flatline stripchart signal for 60 sec • EPICS use stripchart signals to implement 5-minute “Soft Abort” timer (typically 50 mR/s) • Whistles at MCC to warn operators Easily affected by BAD calibrations! BW:MID disabled right now

20. SVT Diodes : Calibration Dose Rate • Dose rate signals calibrated AUTOMATICALLY after most beam aborts • Frontend ABORT hardware needs calibration 2 times per day MANUALLY by Liaison w/ NO BEAMS • More dose received, more often we need to calibrate • Radiation damage HUGE issue • BW:MID pedestal has gone from 1 nA  1100 nA !! Abort Threshold During DAILY_CALIB is a guaranteed time to do this!!

21. SVT Diodes : Tips • Use the SVT stripcharts to watch bkg levels • Keep the SVT Diode EPICS panel open • TOP/BTM diodes MUCH more accurate, although less sensitive • They are a good way to verify degree of bad bkgs • All 4 MID diodes ocassionally exhibit large oscillations/noise of +/- 100 mR/s for 3-8 min. • Cause unknown, but typically harmless (doesn’t trip) • BW:MID diode has unpredictable pedestal • expect poor quality dose rates • When in doubt, calibrate the diodes

22. DCH Currents and DCH Trips • DCH total current is the most precise signal for measuring some types of bkg • Available on BKG stripchart • Trips occur when HV segment exceeds its power supply current threshold for more than 5 seconds • DCH trips cause inefficiency and stress the chamber • Can tolerate 3-4 per shift • Often trips at top of fill; ops can fix this if occurs repeatedly

23. EMC Pin Diode • EMC worries about long-term integrated radiation dose leading to light losses in crystals • EMC has placed several diode/CsI-crystal sensors on the endcap • The sum of these sensors’ is used to inhibit injection and cause EPICS alarms • Referred to as EMC_TOTL • Inhibits injection at ~0.7V • Most sensitive to poor HER injection • Rare, but can suffer problems • Pedestal drift in which signal is >0.1V during NO BEAMS • Sometimes can get saturated beyond 1.0 V

24. EMC Readout Diodes • EMC crystals are read out using biased photosensitive diodes • Poor injection or high backgrounds can actually trip the readout diodes’ bias supply • Mild history of power supply spike-trips, too • Watch stripcharts on BKG panel • COLORS carry no alarm value (just match color to plot trace)

25. DRC Scalers • DRC phototube rates sensitive to LER tunes, LUMI, and global orbit • DRC has connected several phototubes to scalers • Some read out in EPICS • Some read out into MCC • High DRC ‘tube rates == DAQ deadtime • Pain threshold is 200 kHz • EPICS alarm and stripchart signals Often excessive during injection and at top of fill

26. DRC CsI Sensors • DRC has added multiple diode/CsI packages on beamline to understand source and distribution of backgrounds • In EPICS • DRCSOB/CsI-MonCsI-Monitors • Available on BKG stripchart • Not mission critical, but a vast wealth of information

27. TRG Rates and Deadtime • Fundamentally important • High TRG rate incurs deadtime • Available on FCT stripchart • EPICS alarms • Well correlated with other bkg signals • If isolated, probably BaBar problem • FCT LUMI not always well calibrated wrt BaBar LUMI

28. Common Background Issues (Thanks to B. Murphy for contributions)

29. Common Beam Loss Reasons *Look for cause on white text screen near ceiling above LINAC desk in MCC If you can’t find it, ask operators to point out “MPS CUD” display (SLCX9 or MCCX38)

30. What You MUST Watch • Liaison Alarm Hander • Launched from BaBarMainBKG • 3 Liaison-specific stripcharts • Launched from BaBarMainBKG • BaBarMain • BaBar State • Injectable status • Alarms

31. BaBar State Machine • BSM mediates activity between BaBar and PEP-II • BaBar accepts requests from PEP-II for injection • PEP-II declares StableBeams, after which BaBar can go Runnable

32. Your Toolbox

33. Your Toolbox • Liaison workspace/environment • Starting EPICS • Using EPICS • Maintaining EPICS • Using StripTool • Responding to the Liaison Alarm Handler

34. Liaison Computing • Two computers available • Sun/Solaris <bbr-con13> and NCD term • Since Netscape and EPICS often color-conflict, we strongly recommend • Use NCDterm as an xterm from <bbr-con13> • Username babarpep, password ****** • Use <bbr-con13> to run EPICS • If you have time to e-mail and do other work, use additional windows on NCDterm

35. Your Two Screens

36. EPICS • You MUST learn EPICS to survive • EPICS is the universal BaBar detector control and monitoring environment • Hardware connected to VME single-board computers (called IOCs) • Each IOC shares its knowledge across IR-2 local ethernet in virtual EPICS environment • Users can connect with client displays, alarm handlers, strip charts, UNIX tasks, etc. • Similar to web and browser with “links” • MCC has been using EPICS longer than BaBar and they’re better at it

37. Starting EPICS • To start EPICS • Open a window on <bbr-con13> • You should be logged into the console as babarpep • Close all copies of Netscape • Type “startepics” • Ignore warning messages and type “BaBarMain” • You’ll get the windows shown here

38. EPICS & Mice • EPICS uses THREE mouse buttons • LEFT click () • Activates buttons, switches, allows typing • Calls FIRST menu option by default • CENTER click • Calls dmChan on the EPICS channel, returning a window with the name of the channel and options for details • Useful for learning name of a channel to add to a strip chart • RIGHT click () • Reveals pull-down menu of sub-panels • Critical to realize this (you might never find ½ the panels in EPICS because you never RIGHT clicked and dragged!)

39. EPICS & Colors & Buttons • As might be obvious, a lot of EPICS channels are color-coded to indicate alarm severity • WHITE = INVALID • RED = MAJOR • YELLOW = MINOR • GREEN = NO_ALARM • On indicators, color-coding usually represents ON/OFF (digital bit) status • Often HELP button in upper right corner • LEFT click on this will open proprietary browser on usually helpful and specific html file Use this!

40. Vital EPICS Panels • Always start from BaBarMain • SVTBackground • Expert ControlsCalibrate ABORT Thresholds • Expert ControlsAUTO Fast History • Expert Controls1-minute Diode Dose Rates • DetailsTrip History • History PlotsMonitoring Diode Dose Rates • BKG • SVT Diodes Abort Status • FCT • PEPInj Cntrl • LUM • Details

41. Starting & Using Stripcharts • StripTool is an application bundled w/ EPICS • Allows scrolling time series plot of up to 10 EPICS channels simultaneously • History starts accumulating when you START the tool • Key features • LEFT clicking on side legend changes displayed y-axis scale to that channel • RIGHT clicking and dragging in plotspace gives menu. Controls Dialog is very useful. • Controls Dialog allows changing of scales, color of plotted curve, and duration of time displayed • Amount of scroll-back history stored is mostly random (depends on EPICS release)

42. StripTool Windows

43. Liaison Alarm Handler • Automated method of alerting user when EPICS channels exceed predefined thresholds • Beeps and flashes • Launched from BKG panel in EPICS • Click on Liaison ALH button launches a new session • Liaison ALH is specific to Liaison • Channels in it are relevant • LEFT click on main window gives detailed tree/hierarchy

44. Alarm Handler Tips • LEFT click on flashing square acknowledges and silences • Can always FileClose to dismiss tree window • This will NOT close ALH top window • Good thing to have Occupy-All-Workspaces Detailed breakdown Tree hierarchy Click G for help Click P for EPICS panel Click Arrow for sub-tree

45. Clipping a Picture into the E-log • From babarpep on <bbr-con13> • Open an xterm window • Type “xv &” • LEFT click in the new xv window that opens • Move the windows around so that they don’t block your view of what you want to grab • LEFT click “Grab” in the xv controls window • Move the cross cursor to the corner of the area you want to grab; LEFT click and drag open a selection rectangle. When it is the right size, RELEASE. • Hit “Save” in the xv controls window. • Enter a filename and save it. • Using the e-log running in Netscape on the NCDterm, start a new entry, and click “More Lines” and then “Browse.” • Use the menus to select the file you just saved. You may need to edit the filter or type the filename in by hand, as the directory contents are not refreshed immediately. • You’re DONE!

46. xv Windows

47. Our Tutorial • We’ll use IR-2 workstations to invoke EPICS just as in MCC • We’ll practice • Finding panels • Interpreting signals • Answering questions (a quiz?!)

48. Tutorial Tasks • SSH into <bbr-con13> as babarpep • Start EPICS • Are we taking data? • What is the BaBar livetime percentage for this shift? • How much logged lumi to date (by BaBar)? • What is EMC_TOTL’s current value? • Put BaBarMain in all workspaces (make the BaBarMain window appear in all desktop workspaces) • Is PEP-II “respecting” BaBar? (find the BaBar State Machine page for help) • What is the LER beam lifetime right now? • Start the Liaison Alarm Handler • Is there an alarm currently? If so, why? • How many conditions contribute to a PEP-II alarm? • Start the 3 Liaison Stripcharts • What is the DCH total current pain threshold? • What is the current L1 trigger rate? • SVT Diodes • When were the abort thresholds last calibrated? • What are the current SVT diode thresholds? • When was the last dose calibration? • By how much did the last dose rate calibration adjust the FE:TOP diode’s (stripchart) pedestal? • Find the 5-minute timer / SoftAbort panel • For the advanced • When will the HER beam current cause a YELLOW alarm? • Add a TOP/BTM SVT diode to a stripchart • What is the FE:TOP diode’s Total Integrated Dose? • What is the BW:MID diodes’s pedestal current?

49. More Tutorial Questions • The DRC scalers turn RED and the Alarm Handler beeps. What do you do? • You look up and see that the beam currents are zero. What do you do? • PEP-II trips a magnet supply and tells you that it will 20 minutes to bring it back up. What do you do?

50. References • http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/ • http://hep.stanford.edu/babar/monitoring/svtrad/docs.html • http://hep.stanford.edu/babar/monitoring/svtrad/briefIntro.html • http://www.slac.stanford.edu/~meyertim/svtrad/bkgs/ • http://www.slac.stanford.edu/~schieti/background/