1 / 9

Enhancing Anomaly Detection in Elasticsearch Service: Lightning Talk

Dive into anomaly detection and degradation prediction using deep autoencoder classifier in Elasticsearch Service by Jennifer Andersson on 15/08/2019, exploring challenges, data preprocessing, LSTM networks, model evaluation, and future work.

lesparza
Download Presentation

Enhancing Anomaly Detection in Elasticsearch Service: Lightning Talk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Anomaly Detection in the Elasticsearch Service Lightning Talk 15/08/2019 Jennifer Andersson

  2. Introduction Elasticsearch Service: Project goal: • Search- & analytics engine • 30 clusters & 160 use cases Detect service issues before they cause problems

  3. Anomaly Detection & Degradation Prediction Deep autoencoder Classifier

  4. Project Challenges • Better preprocessing of the input data required • Evolution of cluster characteristics over time requires frequent retraining • Convergence issues makes retraining hard (vanishing gradient problem) Apache 300 return codes Time [two years]

  5. Data Preprocessing Raw data Preprocessed data

  6. Long Short-Term Memory Neural Networks ht+1 ht-1 ht Xt+1 Xt Xt-1 x x x + ᶴ ᶴ ᶴ ᶴ ᶴ Colah’s blog post: Understanding LSTM Networks

  7. Long Short-Term Memory Neural Networks ht+1 ht-1 ht Xt Xt+1 Xt-1 x x x + ᶴ ᶴ ᶴ ᶴ ᶴ Colah’s blog post: Understanding LSTM Networks

  8. Model Evaluation • Ongoing work: • Compare to • non-ML methods, e.g. moving average • other ML-methods , e.g. (Extended) isolation forests • Index anomaly scores to Elasticsearch and create a dashboard • Future work: • Classification of anomalous events • Apply the method to other services • Achievement: • Much better convergence than the previous model

  9. Thank you for listening Contact me: Jennifer Andersson jennifer.r.andersson@gmail.com

More Related