Creation of an instrument maintenance program at W.M. Keck Observatory

1. Creation of an instrument maintenance program at W.M. Keck Observatory

2. This is the story of the IPM project which ran for FY09 and FY10: • The following slides were presented at SPIE 2014 • Article reprints available (just ask) • Intended as advice for those attempting something similar • Emphasis on recounting lessons learned despite how obvious they might be in hindsight • The motivation, challenges, history, etc. are presented frankly The project team included: • OID: Grant Hill, Dwight Chan, Mike Wagner, Eric Appleby, Rich Matsuda • Software: Shui Kwok, Jeff Mader, Kevin McCann • Observing Support: Greg Wirth, Scott Dahm, Bob Goodrich, Marc Kassis, Jim Lyke

3. Background Instrument maintenance has historically posed a challenge at Keck • Resource constrained (lean) operations model  catch-22 • Just enough resources for support & corrective maintenance • Inability to perform PM’s leads to……. corrective maintenance Instrument PM system creation was difficult at first • Early efforts suffered from……. • Attempting “as time permits” within operations (pre-project) • Resource competition from higher priority projects • Incomplete combination of skills allocations • Well resourced project was begun in 2009 • Coincident with creation of cross-disciplinary an OID • Right mix and quantity of skills with sufficient priority to retain them • Project ran for 2 years & used about ¾ FTE/year.

4. Project Vision & Deliverables Instrument PM tasks……. • No heritage of systematic, documented instrument maintenance • Even simple lists of PMs did not exist • Key deliverable  develop, document & perform PMs for 1st time Facilitated by extensive, flexible & intuitive software • Capable of interfacing with virtually all other WMKO scheduling tools • Highly automated and intelligent (critical in a lean operations model) • Schedules, generates, and delivers work orders • Able to make decisions based on schedules and resources • Considerable interaction via email and web pages • Extensive archive to store task descriptions & performance records • Highly searchable via complex queries • Capable of storing drawings, pictures, FITS files, documents, etc.

5. Key planning decisions Build or buy? • Purchase commercial (turnkey) package vs. develop in-house • Allows greater control and integration Development and Deployment strategy • Broad but shallow  vs. Narrow but deep • Proceed in order of cost/benefit • More immediate return  frees up more time  do more PMs  • Charge ahead  vs. wait for software • Start simple  find problems  pays back any missteps • Results aid software development Who will develop and perform the tasks? • Small summit instrument team stretched thin • Luxury of choice did not exist (engineer and technician too busy) •  Instrument Scientist + ………

6. The system in use (tasks & work orders) Tasks • Procedural and scheduling information entered & viewed via web • Pictures, drawings, documents etc. can be attached • System now expanded to include facilities, safety, laser, telescopes, coatings, HQ facilities, AO, & ……….. Work orders • Notifications delivered via email • Contains links to task page and work order page • Basic interaction can be via replies to email • Work results and links to other pages (e.g. day log) can be included on work order pages • System will bug you if a work order is not responded to.

10. The delivered system……Success? • Competition from other projects led to some de-scope • But works very well for preventive and predictive maintenance • Rest of observatory voting with their feet (system still growing) • Through 2nd year of project and into ops…….. • Lag between task entry and activation at first • Gap between work orders issued and completed is growing • Jump in problems found after jump in task activation • Gap in problems found and fixed persists • Already seeing inability of small operations staff to keep up • What about the only metric that really matters? • Fault rate for non-AO instruments dropped • Equivalent to 3 full nights of science per year • Roughly equal to 3 papers per year • System paid for itself roughly twice over by the time of its handover to operations

13. Lessons learned (project phase) • Dedicated effort may be needed to create an instrument PM system • Very challenging to use operations effort • Project must have priority, resources, and right mix of skills • Competition with other projects can send you below critical mass • Leveraging time savings capabilities of software may be critical • Especially if operations model is resource constrained • For Keck it made sense to build the system software in-house • Shallow but broad approach will pay off sooner • Reduction in corrective maintenance may kick in sooner than you think • Don’t wait for software… Just start doing PMs no matter how simple • Be pragmatic. Right combination of people is better than no-one • It’s a jungle out there. Be prepared to be out-prioritized • Have viable de-scope options prepared • Emphasize user driven design and flexibility • People will vote with their feet

14. Lessons learned (post-project) Be prepared for turnover to operations • Invest effort (user testing and design) into making software simple & intuitive • Don’t de-scope documentation How will you run the system? • Be prepared to rationalize staffing changes to leverage the system • Make sure a system “champion” exists. Someone who…… • Monitors system usage, work order completion, looks for problems • Optimizes task parameters such as scheduling frequency • Ensures corrective maintenance leads to new PM’s entered • “Opportunistic” PMs are done • PMs are created for new instruments and upgrades

15. Lessons Learned (+ 2.5 years Epilog) • Make sure the system facilitates analysis, reporting and planning • Need stats and plots • Need to be able to manage PM’s in good times and especially when times are tough • Be prepared to revisit (and change) “great” ideas • Work orders auto-close and re-issue according to telescope schedule • Conversely, make sure good features are used. • Tasks are owned by aliases… not by people. • System complements (not replaces!) the day log system

16. Lessons Learned (+2.5 years Epilog) • Make sure there is oversight • Users may seem to know what they are doing but there can be pitfalls. • More features means more ways to trip the unwary or inexperienced. • Don’t rely on automation to warn you that things are amiss. • Be flexible in how you use the system • Leverage the natural skills and inclinations of people. Don’t square peg and round hole things. • Different approaches/methods may work best for different functional areas.