Detecting Large-Scale System Problems by Mining Console Logs

1. Detecting Large-Scale System Problems by Mining Console Logs Author : Wei Xu* , Ling Huang†, Armando Fox* David Patterson* ,Michael Jordan* Conference: ICML 2010,ACM SOSP2009 Advisor: Yuh-Jye Lee Reporter: Yi-Hsiang Yang Email: M9915016@mail.ntust.edu.tw

2. Outline • Introduction • Methodology • Evaluation and Visualization • Conclusion

3. Introduction • Information of console logs? • Console logs rarely help in large-scale datacenter services • Operational problems are dependent on the deployment and runtime environment • Typical console log is much more structured • Anomalydetection • Unusual log messages often indicate the source of the problem

4. Workflow • Log Parsing • Convert a log message from unstructured text to a data structure • Feature creation • Constructing the state ratio vector and the message count vector features • Anomaly detection • Principal Component Analysis(PCA)-based anomaly detection method • Visualization • Decision tree

5. Workflow

6. Log Parsing with Source Code • Difficulty: Templatize automatically • C language • fprintf(LOG, "starting: xact %d is %s") • Java • CLog.info("starting: " + txn) • Not easy to distinguish variables、states

7. Parsing Approach-Source Code • Generate the source code’s abstract syntax tree (AST) • Use AST to identify all method calls on objects of the classes (or their subclasses) • Deduce the types of variables in message templates

8. Parsing Approach-Source Code

9. Parsing Approach-Log • Apache Lucene reverse index • Implement as a Hadoop map-reduce job • Replicating the index to every node and partitioning • The map stage performs the reverse-index search • The reduce stage processing depends on the features to be constructed

10. Parsing Approach

11. Feature Creation • The state ratio vector • Each state ratio vector : a group of state variables in a time window • The message count vector • Each vector dimension : different message type • Value of the dimension : messages appear in the message group

13. Feature Creation-The message count vector

14. Anomaly Detection-Principal Component Analysis (PCA)

15. Anomaly Detection-Principal Component Analysis (PCA) • Applied Term Frequency / Inverse Document Frequency (TF-IDF) • Replace each entry yi,jwith a weighted entry wi,j ≡ yi,jlog(n/dfj), where dfjis total number of message groups that contain the j-th message type

16. Evalution and Visualization • From Elastic Compute Cloud (EC2) • 203 nodes of HDFS and 1 nodes of Darkstar

17. Evalution and Visualization • Parse fails when cannot find a message template that matches the message and extract message variables.

18. Evalution and Visualization • 50 nodes, takes less than 3 minutes , less than 10 minutes with 10 node

19. Evalution and Visualization-Darkstar • DarkMud • Provided by the Darkstar team • Emulated 60 user clients in the DarkMud virtual world performing random operations • Ran the experiment for 4800 seconds • Injected a performance disturbance by capping the CPU during time 1400 to 1800 sec

20. Disturbance by capping the CPU

21. Evalution and Visualization-Darkstar • Ratio between number of ABORTINGto COMMITTING increases from about 1:2000 to about 1:2 • Darkstar does not adjust transaction timeout accordingly

22. Evalution and Visualization-Darkstar • Augmented each feature vector using the timestamp of the last message in that group

23. Evalution and Visualization -Hadoop

24. Evalution and Visualization -Hadoop

25. Evalution and Visualization-Hadoop

26. Conclusion • Using source code as a reference to understand the structure of console logs are able to parse logs accurately • New opportunities for turning built-in console logs into a powerful monitoring system for problem detection

27. Thanks for your attention Q&A