1 / 20

Self-Healing on Network Failures With Vitrage, Mistral & Heat

Explore self-healing capabilities using Vitrage, Mistral & Heat in OpenStack. Learn to create self-healing scenarios, detect failures, and automate recovery for improved system reliability and performance.

braga
Download Presentation

Self-Healing on Network Failures With Vitrage, Mistral & Heat

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ifat Afek, Nokia Rico Lin, EasyStack Danny Lahav, Nokia Self-Healing on Network Failures With Vitrage, Mistral & Heat

  2. Self-Healing Motivation

  3. Self-Healing Challenges • Automatic • Stability • Scalability • Consistency • Find root cause

  4. Vitrage The OpenStack Root Cause Analysis Service which: • Detects the root cause of failures • Updates the state of affected resources • Provides an holistic and complete view on the system

  5. Heat, Vitrage & Mistral Heat Configure Self Healing Vitrage System Insights Mistral Workflow Execution Heat, Vitrage & Mistral: Automate Self-Healing Scenarios !

  6. Demo

  7. How Did it Work? Heat Heat Template Heat Stack Creation Heat Stack Nova Instance Vitrage Template Mistral Workflow Failure At Runtime Instance down External Monitor Vitrage Mistral Execute Healing Workflow NIC down

  8. Vitrage Templates • The business logic behind alarm analysis • Human-readable yaml files • Condition->action scenarios • Sub-graph matching algorithm • Stein: templates with parameters Write once and reuse scenarios: - condition: host_alarm [on] host AND host [contains] instance actions: - set_state: state: SUBOPTIMAL target: host - set_state: state: SUBOPTIMAL target: instance - raise_alarm: target: instance alarm_name: CPU performance degradation severity: WARNING causing_alarm: host_alarm

  9. Heat: VitrageTemplate Resource • Create a Vitrage template for the new Heat Stack • Reuse the same Vitrage template • Provide parameters for the current Heat stack • Allow calling the Mistral Workflow that was defined in the Heat template • Support self-healing scenarios with minimal effort • To be included in Train release

  10. Creating the Vitrage Template Heat Vitrage Create Vitrage Template template_name workflow_name heat_stack_id instance_id Parameters:

  11. Two Steps to Create Your Own! • Open the self-healing example from heat-templates repo • Copy the Mistral::Workflow resource • Copy the Vitrage::Template resource • Adjust the Vitrage template parameters • Instance_alarm_name • instance_id And... you’re done :-)

  12. Self-Healing SIG (Special Interest Group) Scope: > Coordinate the use and development of several OpenStack projects which can be combined in various ways to manage OpenStack infrastructure in a policy-driven fashion, reacting to failures and other events by automatically healing and optimising services.

  13. Next Steps Documentation How to build your own Code Target current under-development features in Train Tests • Enhance cross-project gating • Tempest with Heat, Vitrage & Mistral New Use Cases • Auto-Scaling • What else?

  14. Auto-Scaling Heat Heat External Monitor Vitrage Mistral Scale out Monitor Queue service (Vitrage/Zaqar/etc...) Heat Signal to scale CPU overload

  15. Auto-Scaling SIG scope: > Auto-scaling SIG scope is to improve experience on develop, operate, and use autoscaling and its related features (like metering, cluster schedule, life cycle management) and to coordinate across projects and communities (like k8s cluster autoscaling on OpenStack). Also provide a central place to put tests, documentations, and even common libraries for Auto-scaling features.

  16. Summary Easy to support your own self-healing scenarios Use Vitrage to quickly identify affected resources Auto-healing for a specific instance upon a specific alarm Solution can be extended to other use cases

  17. Contact Us • Vitrage • https://wiki.openstack.org/wiki/Vitrage • Email openstack-discuss@lists.openstack.org with [vitrage] tag • IRC: #openstack-vitrage • Meeting: UTC 0800, Wednesday, weekly at #openstack-meeting-4 • Heat • https://wiki.openstack.org/wiki/Heat • Email openstack-discuss@lists.openstack.org with [heat] tag • IRC: #heat • Meeting: UTC 1300, Wednesday, weekly at #heat

  18. Contact SIGs • Self-Healing SIG • https://wiki.openstack.org/wiki/Self-healing_SIG • https://storyboard.openstack.org/#!/project/openstack/self-healing-sig • Email openstack-discuss@lists.openstack.org with [self-healing-sig] tag • IRC: #openstack-self-healing • Meeting: UTC 0900 and 1700, Wednesday, biweekly-odd #openstack-self-healing • Auto-scaling SIG • https://wiki.openstack.org/wiki/Auto-scaling_SIG • https://storyboard.openstack.org/#!/project/openstack/auto-scaling-sig • Email openstack-discuss@lists.openstack.org with [auto-scaling-sig] tag • IRC: #openstack-auto-scaling • Meeting: TBD (there will be:) )

  19. Questions ?

More Related