1 / 36

AvailFIT2020

AvailFIT2020. Subtitle of Presentation. Nick Walker - MPY DESY M Betriebsseminar , Travemünde , 21.02.2019. Accelerator Availability Why so important?. “Mathematically, this [Availability] is expressed as 100% minus  unavailability .”. - Wikipedia.

jon
Download Presentation

AvailFIT2020

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AvailFIT2020 Subtitle of Presentation Nick Walker - MPY DESY M Betriebsseminar, Travemünde, 21.02.2019

  2. Accelerator AvailabilityWhy so important? “Mathematically, this [Availability] is expressed as 100% minus unavailability.” - Wikipedia

  3. Warum ist die Verfügbarkeit („Availability“) so wichtig? Einige Definitionen Gemittelt Zeitrahmen ist wichtig Über eine Stunde? Tag? Monat? Jahr? Normalerweise reden wir über eine „Run“ Periode | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  4. Warum ist die Verfügbarkeit („Availability“) so wichtig? Einige Definitionen Gemittelt Zeitrahmen ist wichtig Über eine Stunde? Tag? Monat? Jahr? Normalerweise reden wir über eine „Run“ Periode Fachbücher Mean Time Between Failure Gemittelte Zeit zwischen „trips“ Mean Time To Recover Gemittelte Zeit zum Erholen | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  5. Definitions – extended for accelerators Mean Time Between Failure (MTBF) = inverse of average trip rate Mean Time To Repair (MTTRep) Mean Time To Recover (MTTRec) Repair – time to repair failed subsystem (e.g. power supply) Recover – time to establish acceptable beam to users (accelerator recovery) MTBF A = MTBF + MTTRep + MTTRec Both important. Both need attention. AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  6. Where are we? PETRA-III (Michael) AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  7. Where are we? (Juliana) PETRA-III (Michael) AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  8. Where are we? (Juliana) PETRA-III (Michael) (XFEL, Winni) AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  9. What’s the problem?

  10. No real BIG problem but ... • PETRA-III • MTBF – “need to do better” (Michael) • Competition from many 3rd gen light sources • Several quoted >98% (routinely) • XFEL • Like to be in the 90’s (Winni) • Competition is coming! • XFEL & PETRA-III • Average down due to specific ‘big’ events • FLASH & XFEL • Importance of “tuning time” But we can always learn something from it (and try to avoid it) AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  11. Warum ist die Verfügbarkeit („Availability“) so wichtig? Ein kleines Beispiel Claudia Fotonuser hat ein 5-Tage Experiment, und braucht unbedingt 95% Availability. Unser Maschine liefert im Moment (Durchschnitt) MTBF = 40 Stunden MTTR = 1,5 Stunden → A =96.4% 👍😀 | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  12. Warum ist die Verfügbarkeit („Availability“) so wichtig? Ein kleines Beispiel Claudia Fotonuser hat ein 5-Tage Experiment, und braucht unbedingt 95% Availability. Unser Maschine liefert im Moment (Durchschnitt) MTBF = 40 Stunden MTTR = 1,5 Stunden → A =96.4% 👍😀 Aber es gibt eine ~ 1:5 Chance, dass wir 95% nichtin diesen 5 Tagen schaffen🤔 | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  13. Warum ist die Verfügbarkeit („Availability“) so wichtig? Ein kleines Beispiel Claudia Fotonuser hat ein 5-Tage Experiment, und braucht unbedingt 95% Availability. Unser Maschine liefert im Moment (Durchschnitt) MTBF = 40 Stunden MTTR = 1,5 Stunden → A =96.4% 👍😀 Aber es gibt eine ~ 1:5 Chance, dass wir 95% nicht in diesen 5 Tagen schaffen🤔 Claudia sagt mit 1:10 ist es das Risiko wert MTTR: 40 → 50 Stunden A: 96.4% → 97.1% | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  14. Definitions – extended for accelerators Hard fault AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  15. Definitions – extended for accelerators Hard fault Soft fault ← contracted Intensity Pointing stability Energy Energy spread ... AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  16. AvailFIT2020What, Why, and How Maintain 98%...

  17. AvailFIT2020 - Process • Conducted informal meetings with Machine Coordination and Technical Group Leaders • November 2017 – April 2018 • In some cases follow up meetings where scheduled • Developed list of themes to cover in the discussions • Not enforced • Allowed meetings to develop naturally • Every meeting was unique • Ranged from ~1 hour – 3½ hours AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  18. Was genau ist AvailFIT2020? Themen (Beispiel) Informelle themenbezogene Interviews Eins-zu-Eins oder in Kleiner Gruppe Beschleuniger/Technische Koordinatoren Technische Gruppenleiter In der Regel 1-3 Stunden Ansatz Informationsbeschaffung Brainstorming Meinungen Ideen 10.000m Aussicht | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  19. Focus on organisation and workflow Not technical issues and engineering solutions • Focus was on how we worktogether • Operations (Availability) perspective • Organisational and management issues are important. Add Motivation AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  20. Why the focus on organisation and workflow? The last few % are hard to get! • Focus was on how we worktogether • Operations (Availability) perspective • Organisational and management issues are important. • Achieving VHA requires attention to detail AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  21. Why the focus on organisation and workflow? • Focus was on how we worktogether • Operations (Availability) perspective • Organisational and management issues are important. • Achieving VHA requires attention to detail • Maintaining VHA requires diligence Whenyouare at the top, theeasiestwayis down! AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  22. Making Good Better by Formalising Workflow and Documentation Achieving VHA requires hard work, attention to detail and diligence Maintaining VHA even more so

  23. Formalising (and documenting!) the way we do work Developing workflows with Structured Documentation Change Management AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  24. Fault life-cyclemanagement Keepingtrackofwhat happend. Understanding whatto do. | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  25. XFEL ”Major Event” Process (8D) Jost Müller AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  26. Eight Disciplines (8D) for XFEL • 16h down time on 8.02.19 • 3.5h system recovery (linac RF) -- MTTRep • 12.5h beam (SASE) recovery -- MTTRec • 2 hour focus meeting of team D1: Form team D2: Fault description (chronology) D3: Interim containment D4: Root cause D5: Developing permanent fixes D6: Implementing permanent fixes D7: Preventive measures D8: Closeout XFEL ELOG: docs/Event reports AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  27. Root cause analysis • Generally a team effort • Stakeholder acceptance / consensus • Amount of effort leveraged against criticality / benefit • Thresholds need to be set • Formal response levels defined • Avoid “Today hot topic, tomorrow forgotten” syndrome. • True root cause is not always obvious: AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  28. Root cause analysis A Fun Example (XFEL RF station A4) A4 voltage suddenly increased, causing cavities to quench Root cause → operator error? Further investigation showed that A4 is under control of an energy feedback, which increased the voltage as observed above Root cause → feedback system? Keep looking. ”Spectrometer” BPM suddenly showed 100mm offset to which Energy FB reacted Root cause → BPM? At that time, magnet server “failed”. BPM software had “offset” calculated using magnetic field that faulted server reported as zero, causing 100mm BPM result Root cause → magnet server? Finally, max allowed voltage was too high, and should have been lower in the first place. Similarly, Energy Feedback should be made more robust against erroneous signals Should the BPM software not have caught this and reacted more robustly? Yes server failed and is arguably “root cause” Would an FMEA have caught this? AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  29. Ideen, die es wert sind verbreitet zu werden | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  30. Ideen, die es Wert sind verbreitet zu werden PETRA III VerfügbarkeitsSeminarserie Initiative von H. Ehrlichmann, D. Haupt „Confluence“ System benutzt als Ticket Tracker | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  31. Ideen, die es Wert sind verbreitet zu werden MHF-p Verfügbarkeit Verbesserung Maßnahmen (F. Eints) https://indico.desy.de/indico/event/19040/session/5/contribution/12/material/slides/ | Präsentationstitel | Vorname Name, Datum (Eingabe über "Einfügen > Kopf- und Fußzeile")

  32. Ideen, die es Wert sind verbreitet zu werden MSK: Verwendung des REDMINE-Tools • Flexible Projektmanagement-Anwendung • Open Source, plattformübergreifend • Bug-Tracking • Feature Anfrage • Dokumentation • Wartungsübersicht • Ausfallzeit Statistiken • Besprechungsprotokoll • Sendungsverfolgung Wird als getaggtes Ticketsystem verwendet sub-project project tagged meta data filtering time and effort tracking link to individual task description courtesy: J. Branlard

  33. Procedures / checklists • Automates the steps • Promotes conformity in approach • Fields for key data entry and observations • Generated documentation ‘on the go’ • I know of cases where having such a checklist would have • Prevented mistakes • Saved time • Reduced wasted effort Pour tea Designing / developing useful templates is THE challenge Note: British tea! | AvailFIT2020| Nick Walker | MSK Seminar 25.01.2019

  34. Change Management Preventing mistakes during maintenance and development • BIG theme in industry (especially software) • Standards available (e.g. CM-II) • Benefits (the good) • Adds some resistance to change • Is this really necessary and worth the risks? • Informs/includes all stakeholders • A ‘heads up’ at all the interfaces • Risks have been carefully considered by a larger group • Provides formal documentation • The bad • Bureaucratic overhead • Process requires a lot of fine tuning • Gets in the way if not implemented correctly | AvailFIT2020| Nick Walker | MSK Seminar 25.01.2019

  35. More fun stuff to come MKK High Availability Technical / engineering How to get the most out of you power supplies and other systems Hans-Jörg Eckoldt • Das PETRA-III Availability-Meeting • Organised by H. Ehrlichmann, D. Haupt • ~one day “review” of all PETRA trips from recent user run • Including follow up on action items • All relevant tech. groups represented • Typically 15-20 people. • JIRA ticket / Confluence systems used for tracking and documentation (WIP) • Independent overview by Michael Ebert. AvailFit2020 - Nick Walker (MPY) - M Betriebssemiinar, Travamünde, 22.02.2019

  36. Vielen Dank

More Related