Liaison Training Seminar

1. Liaison Training Seminar Brought to you by the BaBar/PEP-II Liaison Team (First Version, Sep 2000 TIMeyer) (2nd Version, Nov 2001 SSHertzbach)

2. Outline • Job Description • Backgrounds • Your Toolbox • Mice, EPICS, and other clicks • Some PEP-II information • Tutorial • Hands-on time (?) at MCC or IR-2 • More details in the online shift manual • http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/

3. Being a Good Liaison • READ the online shift manual • http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/ • SIGN UP for Liaison hypernews • http://babar-hn.slac.stanford.edu:5090/HyperNews/get/babar_pepii.html • SIGN UP for Detector Ops hypernews (or read) • http://babar-hn.slac.stanford.edu:5090/HyperNews/get/DetOper.html • VISIT two Liaison (2) shifts before your first for hands-on experience (visit IR-2 if needed) • ATTEND this training workshop • BE smart while on shift; DON’T pass the buck • GIVE feedback to other Liaisons & Coordinator

4. Job Description - Intro • Shifts are taken at MCC, Bldg 005 • Attend PEP shift-change meeting 15 min before shift starts, as an observer. • Answer x5121, call x5255 (IR-2) • Call/page experts when appropriate • Use your toolbox to stay up-to-date • Make notes in main e-logbook, e.g., beam conditions, changes, aborts; and diode calibrations • See Liaison Shift Manual for details • Bring next shifter up-to-date

5. PEP-II Hierarchy • Typically 3-4 PEP-II operators on a shift • 1 “EOIC” – engineer operator in charge • This is the person with whom you should speak most often • They determine short term schedule • They know a LOT • 1 or 2 for PEP-II • 1 for Linac, FFTB, other programs • 1 Program Deputy (PD) who manages weekly goals • An accelerator physicists, and very friendly • Ask them technical question, but be prepared to answer some in return!

6. Responsibilities • Protect BaBar • from user errors • from background conditions • by monitoring protection systems & backgrounds • Maximize integrated luminosity • 15 min. is 3% of shift  95% vs98% eff. • Communicate (BaBar  PEP-II) • Help understand and mitigate backgrounds • affects data quality

7. Why Protect BaBar? • Machine backgrounds at PEP-II are serious • High because of unprecedented currents and luminosities (~90% of L1 rate is background) • Consist primarily of • lost particles scattered by beam-gas interactions which then shower into detector • Low angle radiative bhabhas which scatter off machine elements into detector • High backgrounds can cause • Detector deadtime / reco ineffeciencies • Radiation damage

8. Trapped Event:Note elevated L1 and DT AVLWT Maximizing Luminosity • Communication • always know BaBar needs and desires • make best use of down time; coordinate BaBar and PEP-II • Monitor topoffs (use AVLWT in FCT stripchart) • topoff starts at appropriate time? • PEP hands beams to BaBar promptly? • BaBar DAQ running? • Anything else you can think of doing!

9. Communication • Liaison is primary source of information • For BaBar about PEP-II • For PEP-II about BaBar • Responsible for • Coordinating downtime and repair work • Unscheduled and short-term periods with no beam • BBR or PEP won’t always TELL you; you need to ask • Answering questions • Condensing, distilling, and CONTROLLING information • Writing it down in the e-logbook • The Liaison makes BaBar and PEP-II a team

10. What PEP-II Will Ask • PEP-II really appreciates Liaison presence • Backgrounds • Are they okay? • How are they measured? • What is their impact? • Beam aborts • Who did it? • Why? • Luminosity • What is it? How is it measured? • How much is it • BaBar detector and physics • Why can’t we inject? • Anything else…

11. What IR-2 Will Ask • IR2 wants to know, too • Are backgrounds okay? • Backgrounds don’t look OK, WHY? • What “just happened?” • What’s going on over there? • What happened to the beams? • Who aborted the beam and why? • What is taking so long? You’ll soon discover that true DIPLOMACY is required…answer the important questions without bothering too many people, and keep both sides synchronized and content.

12. Who caused the Beam Abort? • This is an important question whose LACK of an answer is sure to irritate many, many people. LEARN HOW TO TELL. • Only the SVT diodes will abort the beam from the BaBar side • Check the SVT Diode EPICS panel for a REDTrip light (more on this later) • PEP-II keeps a running logfile of Beam Abort Triggers on a monitor on their side of the room • Ask another Liaison if you don’t know it • Ask operators per “MPS CUD” and/or SLCX3 • YES, you’ll have to move from your chair to read that display • Key to messages you’ll see • SVT_H(L)ABT : SVT diode beam abort • EMC_H(L)INH : EMC_TOTL H(L)ER injection inhibit • BBRHVH(L)INH : BaBar !INJECTABLE injection inhibit • MCC_ABORT : PEP-II op big red button abort • KLYS_xxxxx : RF trip • BOAT : Bad Orbit Abort Trigger • Don’t BLAME the operators; share, and work on it together This is the only abort that BaBar can cause, and it should show up in BaBarSVT  Background EPICS

13. Backgrounds at BaBar • High rate of backgrounds can threaten • SVT, EMC, DCH, DRC, IFR, TRG • Several subsystems have dedicated background monitoring and protection systems • It is the Liaison’s job to care for and feed them or contact an expert when appropriate • Different systems sensitive to different things • Sensors are often correlated, but not always!! • When are backgrounds typically the highest? • Injection (SVT, DCH, IFR ramp down voltages) • Especially at low currents, early in fill from scratch

14. Backgrounds Differ Daily • Plot shows bkgs during StableBeams for 1 SVT diode as compared to best bkgs in July,2000 • Note that “ratio” decreased, but a LOT of noise, and even central value got worse in Aug. and Sep.! Scrubbing of the vacuum August ROD Vac leak

15. What To Do • What does high background mean? • “Pain thresholds” are defined for each detector such that running within 5% of them is OKAY • “Top of the stripchart” = pain threshold • When backgrounds are high • Verify that it makes sense • Be sure that not just injection backgrounds • Observe what PEP-II operators are doing • If situation persist for more than 90 seconds, ask operators nicely about it (be armed w/info!)

16. SVT Diodes: Intro • System of 12 PIN diodes & dedicated hardware • Monitor and control radiation exposure • 4 MID-plane diodes • Receive highest doses (5-10x top/btm diodes) • Are in a special ABORT circuit which aborts beam • 8 TOP/BTM diodes (more precise) • Signal in mR/s (~5-20 mR/s) • Requires exact knowledge of pedestal • Temperature sensitive! • Changes with radiation damage, too! Biggest source of error is pedestal subtraction! BW-MID Pedestal 1200 nA, and 1 nA = 5 mRad/s

17. SVT Diodes: Summary Panel See this menu for: Calibrations FastHistory Captains Panel Latched trip status Can be reset under DetailsTrip History NOTE

18. SVT Diodes: Trips • Front end hardware offers “fast” protection • Typically visible on stripcharts, but not always • After “fast” trip, post-mortem buffer read out to disk which will flatline stripchart signal for 60 sec • EPICS uses stripchart signals to implement 5-minute “Soft Abort” timer (typically 50 mR/s) • Whistles at MCC to warn operators (unless disabled) Easily affected by BAD calibration! BW & FE have higher thresholds.

19. SVT Diodes: Calibration • Two independent systems (HW & SW) • Both SVT diode calibrations are beam-off leakage current pedestal measurements. • Software: Dose rate (monitoring) calibration for • stripcharts, EPICS, and soft abort 5-minute timer • corrected in software (between calibrations) for temperature & radiation damage, using elaborate model • Hardware: Abort threshold calibration for • fast hardware aborts • corrected in hardware (between calibrations)only for temperature, using linear model

20. Dose Rate Monitoring SVT Diodes: Calibration No Liaison action required • Monitoring(dose rate) signals calibrated automatically, and continuously, if no beam. • Front end ABORT hardware needs calibration 2 times per day MANUALLY by Liaison w/ NO BEAMS • Takes 3 - 5 minutes (~1% of shift) • More dose received, more often we need to calibrate • Radiation damage HUGE issue • BW:MID pedestal has gone from 1 nA  1100 nA !! Abort Threshold Calibration During daily BaBar Global Calibration is a guaranteed time to do this!!

21. SVT Diodes: Tips • Use the SVT stripcharts to watch bkg levels • Keep the SVT Diode EPICS panel open • TOP/BTM diodes MUCH more accurate, although less sensitive • They are a good way to verify degree of bad bkgs • All 4 MID diodes occasionally exhibit large oscillations/noise of +/- 100 mR/s for 3-8 min. • Cause known, fix unknown, but typically harmless (doesn’t trip) • BW:MID and FE:MID have the highest radiation dose • BW:MID diode has unpredictable pedestal • expect poor quality dose rates • When in doubt, calibrate the diodes

22. 2000 DCH Currents and DCH Trips • DCH total current is the most precise signal for measuring some types of bkg • Available on BKG stripchart • Trips occur when HV segment exceeds its power supply current threshold for more than 5 seconds • DCH trips cause inefficiency (and stress the chamber) • Can tolerate 3-4 per shift (can, but should not have to) • Often trips at top of fill; ops can fix this if occurs repeatedly • Now (Oct. 2001) trips are infrequent!

23. EMC Pin Diode • EMC worries about long-term integrated radiation dose leading to light losses in crystals • EMC has placed several diode/CsI-crystal sensors on the endcap • The sum of these sensors’ is used to inhibit injection and cause EPICS alarms • Referred to as EMC_TOTL • Inhibits injection at ~0.7V • Most sensitive to poor HER injection • Rare, but can suffer problems • Pedestal drift in which signal is >0.1V during NO BEAMS • Saturation beyond 1.0 V (rare?) is a sign of a system problem • Can bypass only in hardware at IR-2 (see Liaison instructions)

24. EMC Readout Diodes • EMC crystals are read out using biased photosensitive diodes • Poor injection or high backgrounds can actually trip the readout diodes’ bias supply • Mild history of power supply spike-trips, too • Watch stripcharts on BKG panel • Colors carry no alarm value (just match color to plot trace)

25. DRC Scalers • DRC phototube rates sensitive to LER tunes, LUMI, and global orbit • DRC has connected several phototubes to scalers • Some read out in EPICS • Some read out into MCC • High DRC ‘tube rates  DAQ deadtime • Pain threshold is 200 kHz • EPICS alarm and stripchart signals Often excessive during injection and at top of fill

26. DRC CsI Sensors • DRC has added multiple diode/CsI packages on beamline to understand source and distribution of backgrounds • In EPICS • DRC  SOB/CsI-Mon  CsI-Monitors • Available on BKG stripchart • Not mission critical, but a vast wealth of informationbut need to learn how to make best use them.

27. TRG Rates and Deadtime • Fundamentally important • ~90% of L1 rate is background • High L1 trigger rate incurs deadtime • With L3 on 32 nodes, DT is OK up to L1 ~2 kHz • Even if DT is OK, want L1 1.5 kHz at lumi ~41033 • Available on FCT stripchart • EPICS alarms on DT • Well correlated with other bkg signals • If isolated, probably BaBar problem • FCT LUMI not always well calibrated wrt BaBar LUMI

28. Problem Cause Solution SVT spike or step change Dust (photoionized micron scale material drawn electrostatically into HER beam, typically vaporizes and disperses quickly, but can lead to sustained “trapped event”) Wait High DRC rates LER LER tunes, skews, collimators SVT diodes oscillate Noise None (has never tripped the beam) High L1 rates (sudden) Background spike or step change Wait , or “shake beams”, or top-off, or dump beams High L1 rates (chronic) LER if also high DRC Trapped event if also high SVT SVT diode drifts upward during course of fill Radiation damage Wait and re-calibrate when beams lost Common Background Issues (Thanks to B. Murphy for contributions)

29. Common Beam Loss Reasons Problem Cause* Solution Both beams aborted • BaBar abort (SVT diode, 90%) • Vacuum pipe temp • Magnet failure Liaison may be asked to confirm BaBar abort. Ops reset PR02 latch in SCP. Liaison may be asked for “post mortem” history HER beam aborted • RF trip (90%) • dI/dt Dump LER beam (usually) and re-fill both from scratch LER beam aborted • RF trip (90%) • dI/dt Refill LER beam, usually keeping existing HER fill *Look for cause on white text screen near ceiling above LINAC desk in MCC If you can’t find it, ask operators to point out “MPS CUD” display (SLCX3)

30. What You MUST Watch • Liaison Alarm Hander • Launched from BaBarMainBKG • 3 Liaison-specific stripcharts • Launched from BaBarMainBKG • BaBarMain • BaBar State • Injectable status • Alarms

31. BaBar State Machine • State Machine mediates automated activity between BaBar and PEP-II • BaBar accepts requests from PEP-II for injection • PEP-II declares StableBeams, after which BaBar can go Runnable

32. Your Toolbox • Liaison workspace/environment • Starting EPICS • Using EPICS • Maintaining EPICS • Using StripTool • Responding to the Liaison Alarm Handler

33. Liaison Computing • Two computers available, many workspaces • Sun/Solaris and NCD term, boot from IR-2 • Because Netscape and EPICS often color-conflict, we strongly recommend • Use NCDterm for EPICS only • Username babarpep, on bbr-con13, password ****** • Use Sun/Solaris to run EPICS and Netscape • Username babarpep, on bbrmcc, password ****** • If you have time to e-mail, and do other work, use additional windows on Sun

34. EPICS • You MUST learn EPICS to survive • EPICS is the universal BaBar detector control and monitoring environment • Hardware connected to VME single-board computers (called IOCs) • Each IOC shares its knowledge across IR-2 local ethernet in virtual EPICS environment • Users can connect with client displays, alarm handlers, strip charts, UNIX tasks, etc. • Similar to web and browser with “links” • MCC has been using EPICS longer than BaBar and they’re better at it

35. Starting EPICS • On the NCD (runs only EPICS) • Open a window on <bbr-con13> as babarpep • Type “startepics” • On the Sun (runs EPICS and … ) • Left click the yellow EPICS icon on the bar • Color map problems can cause EPICS to crash • Close all stripcharts and other applications, restart EPICS • If crashes continue, restart the display manager • You’ll get these windows:

36. EPICS & Mice • EPICS uses THREE mouse buttons • LEFT click () (or  in Liaison Shift Manual) • Activates buttons, switches, allows typing • Calls FIRST menu option by default • RIGHT click () (or  in Liaison Shift Manual) • Reveals pull-down menu of sub-panels • Critical to realize this (you might never find ½ the panels in EPICS because you never RIGHT clicked and dragged!) • CENTER click • Calls dmChan on the EPICS channel, returning a window with the name of the channel and options for details • Use to learn name of a channel to add to a strip chart • Use to find the alarm limits for the channel • Close the dmChan window when finished with it

37. EPICS & Colors & Buttons • As might be obvious, a lot of EPICS channels are color-coded to indicate alarm severity • WHITE = INVALID • RED = MAJOR • YELLOW = MINOR • GREEN = NO_ALARM • On indicators, color-coding usually represents ON/OFF (digital bit) status • Often HELP button in upper right corner • LEFT click on this will open proprietary browser on usually helpful and specific html file Use this!

38. Vital EPICS Panels • Always start from BaBarMain (ODC) • SVT  Background • Details  Trip History • Expert Controls  Calibrate ABORT Thresholds • Expert Controls  AUTO Fast History • Expert Controls  1-minute Diode Dose Rates * • History Plots  Monitoring Diode Dose Rates • BKG (main background panel) • Abort Status  SVT Diode Soft Abort 5-minute Timer * • FCT  FCTS Trigger Detail • PEP  State Machine • LUM

39. Starting & Using Stripcharts • StripTool is an application bundled w/ EPICS • Allows scrolling time series plot of up to 10 EPICS channels simultaneously • History starts accumulating when you START the tool • Key features • LEFT clicking on side legend changes displayed y-axis scale to that channel • RIGHT clicking and dragging in plotspace gives menu. Use Controls Dialog to customize current chart: • change: scales, colors, line width, duration of time displayed; add variables; etc. • Coordinates of cursor in plotspace are displayed • Length of scroll-back history defined with the stripchart

40. StripTool Windows

41. Liaison Alarm Handler • Automated method of alerting user when EPICS channels exceed predefined thresholds • Beeps and flashes • Launched from BKG panel in EPICS • Click on Liaison ALH button launches a new session • Liaison ALH is specific to Liaison • Channels in it are relevant • LEFT click on main window gives detailed tree/hierarchy

42. Alarm Handler Tips • LEFT click on flashing square acknowledges and silences • Can always FileClose to dismiss tree window • This will NOT close ALH top window • Have Occupy-All-Workspaces (iconify & tuck away) • Open only ONE ALH (on one workstation) Detailed breakdown Tree hierarchy Click G for help Click P for EPICS panel Click Arrow for sub-tree

43. Clipping a Picture into the E-log • From bbrmcc on Sun, LEFT Click xv icon(From babarpep on NCD <bbr-con13>, open an xterm window, type “xv &”) • RIGHT click in the new xv window that opens • Move the windows around so that they don’t block your view of what you want to grab • LEFT click “Grab” in the xv controls window • Either • LEFT click on the window you want to grab, or • Move the cross cursor to the corner of the area you want to grab; MIDDLE click and drag open a selection rectangle. RELEASE when it is the right size. • Hit “Save” in the xv controls window. • Enter a filename and save it. • Using the e-logbook running in Netscape on the Sun, start a new entry, and click “More Lines” and then “Browse.” • Use the menus to select the file you just saved. You may need to edit the filter or type the filename in by hand, as the directory contents are not refreshed immediately. Be sure you are in the correct directory. • You’re DONE!

44. xv Windows

45. Some PEP-II Info • See Liaison Shift Manual: further useful links: • ICHEP 2000 (Osaka) paper for PEP-II, and 1st year experience with backgrounds • BaBar NIM paper for detector overview, more details on PEP-II and backgrounds, plus trigger • PEP-II regions map linked to drawings • and a few others

46. PEP-II Site Overview

47. HER e e LER PEP-II Regions

48. Web Page to display PEP-II Regions http://www.slac.stanford.edu/accel/pepii/ring/regions_test.html (Steve Meyer) Web page allows you to click on region to display details of beamline components in region.

49. (South) (North) PEP-II IR and BaBar in IR-2

50. 40 mrad EAST LER3.1 GeV HER9 GeV HERFWD  ~South HER9 GeV LER3.1 GeV LER BKWD ~North WEST PEP-II IR Details