Status of the Linac Controls Upgrade

1. Status of the Linac Controls Upgrade Tom Himel June 8, 2009

2. Outline • Review of reasons for upgrade and plan • Progress • Problems • Summary

3. Reasons for the Upgrade • Through retirements and layoffs, have lost most people who know how to maintain the SLC control system. • Have decided to upgrade the SLC control system to EPICS with a series of AIP projects. • This will reduce chances of long downtimes due to some problem where expertise is lacking causing a long MTTR. • Hardware is old. Upgrade is overdue. • SLC control system has many years of applications being added and fine tuned. Main difficulty of the upgrade is to not lose too much this functionality.

4. Phases • Phase 1 makes LCLS independent of the Alpha and multibus micros • Replaces about 15 micros with VME crates, CPUs, and CAMAC interface. • 1 micro in each of last 10 sectors, few in BSY, few system micros (e.g. MPG) • Main work is software • Phase II replaces the CAMAC with more modern hardware • Main expense is hardware • Phase III-N propagates above upgrades to other regions of the accelerator complex • Little planning here yet • There are different types of functionality and hardware modules in other regions so not just simple replication

5. “Signed off” List of Tasks • The June 08 upgrade review recommended we make a detailed list of functionality and get it “signed off” by operations, accelerator physicists, and maintenance groups. • We have made such a list and gone over it with above groups at a series of meetings. • With modifications made at those meeting, it was approved, but no formal signatures.

6. “Signed off” List of Tasks

7. AIPs for Upgrade • Linac dipole correctors to EPICS – 300k - ~approved • Linac new quadrupole PS + EPICS control – 800k - ~approved • Linac BPMs to EPICS – 700k - approved • Linac micro -> IOC hardware conversion - ~495k – approved long ago • Linac/BSY unique devices to EPICS – 200k – awaiting better cost estimate • Linac software conversion – 2.03M - ~approved • Linac gate generator and master trigger generator – 139k - ~approved • RF controller upgrade R&D - 1008 k - ~approved

8. We DO have documents and Reviews • 6/4/09 LI17/18 test station design review • 5/5/09 Camcom design review • 1/23/09 PIOP initialize and status check design review • 11/25/08 SAM and IDIM device support design review

9. Hardware Progress • Have 2 crate test setup in lab • Expect to have IOCs to control LI17 and 18 by July 1, 2009 for testing purposes • Have chosen to delay installation of IOCs for LCLS sectors until more software development and testing is done so can be sure we are getting the right stuff.

10. Progress • VME CAMAC board has had several bugs exterminated. In state of no known bugs. • Still need to test output to multiple branches and heavy loading • SAM, IDIM, and DAC device support are all done. • PIOP image can be downloaded and booted and fast status check works. • Indian problems nearly killed Bob. • Analog and digital status DBs will be mass produced from data in old control system.

11. Progress • Got more hardware AIP money than originally planned • Don’t have to develop device support for CAMAC power supply controllers nor handle bulk+boost supplies. • Have started R&D needed for phase II replacement of RF hardware • Functional requirements partly written (some details still need to be filled in) • Have a microTCA crate, ADC, DAC to test as a possible implementation. Starting detailed ADC testing. Has been delayed by lack of software person. New hire will go here.

12. BSY • The BSY area has several micros each which control some LCLS devices and some devices for ESA, PEP, SLC. • As the last two are obsolete but ESA is considering future runs, did not want to break ESA by converting micros to IOCs and then supporting only the LCLS devices • Few enough channels for LCLS that have decided to just change all of them to EPICS style hardware, eliminating the use of CAMAC for LCLS in the BSY.

13. Slow getting people on board • As predicted by the June 08 review team, it has been difficult to get people on board the upgrade project • At that time, plan was to finish the project by Nov 08. • In fact, all work stopped after that review • People were promised first for Nov 08, then Jan 09. • For software, now have 2 ~80% people and 4 ~40% people. More are on the way. • Not complaining. LCLS had to be finished and there are still many important improvements needed. • That review team was really sharp. Especially the chair!! • With the approval of FACET and plans for LCLS upgrades, controls is now hiring many people.

14. Schedule • There still isn’t a detailed integrated schedule for this project • Task list (with durations) is being used for planning. Not many dependencies • Need to get unique device hardware installed during the Jan 10 downtime. • No other external deadlines. • (Un)Availability of manpower has been the main schedule determinant Good to leave obvious reason for the FAC to beat me up

15. FACET • FACET is a newly approved $13M project using S0-19 and the damping rings for accelerator physics experiments • Planned to be done Oct 2010. • Bad timing for controls (busy and in middle of upgrade) • New stuff will be done in EPICS, old, left in legacy control system • Will compete for manpower with advantage of a tight deadline • Fortunately we are now hiring people.

16. Old Worries with status • As implementation of phase I will take longer than originally planned and as expertise in the SLC control system is needed until the rest of the linac, damping rings, NLCTA, the gun lab, and the test lab are converted, people experienced with the SLC control system must be kept or new ones trained. - This is being done • If FACET proceeds on its schedule of completion in Oct 2009, it clearly will not use the new control system and that much more expertise in the SLC control system will need to be maintained. If FACET proceeds on a slower schedule, management will need to decide soon whether a controls upgrade is done as part of the project, or with operating or AIP funds before the project, or if it will use the SLC control system. - chosen

17. Old Worries with status • This is a large, important project. Even so, many LCLS needs discovered during the commissioning process will be higher priority. It is better to do this project right and, if necessary, slowly, than to rush it to meet a schedule deadline and lose too much functionality when it is turned on. Delays have happened, project is being done slowly (and hopefully right). • There are 2 steering feedback in the region being converted. If they need to run at 120 Hz, then the EPICS version of 120 Hz feedback must be done before the switchover. (Planned for Jan 2010.) - Still planned for Jan 2010, so should be OK.

18. Should we go straight to phase II? • As phase I is going slowly and we are converting considerable hardware away from CAMAC (BPMs and magnet control), I (and others) periodically ask, should we skip phase I and just replace all the CAMAC as soon as we can. Pros and Cons • Would save the work of developing CAMAC device drivers and support • Much of DB and most displays will remain unchanged when we move from Phase I to II. Some will change and this is extra work. • Phase II will cost several $M and needs RF R&D to be completed before it can start. Hence would be stuck with VMS and micros for 1-4 years longer • With phase I software done, other areas could be converted fairly cheaply (BPMs and PSC-II and “unique” devices would need to get new CAMAC drivers or new hardware). • Phase II will require significant downtime and commissioning to implement. This will also lead to delays.

19. Summary • Upgrade is started, but slowly • Phase I is mostly planned, but a real schedule needs to be developed • The AIP projects which are tentatively approved should allow completion of Phase I.