1 / 25

Work Together Effectively

Discover how to effectively monitor logs and system performance in the cloud using Logstash, Grafana, Elasticsearch, and more to ensure smooth operations and identify performance bottlenecks.

cnevers
Download Presentation

Work Together Effectively

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Work Together Effectively

  2. Content in the Cloud - Operations

  3. Open Source Monitoring in the cloud Ishara Fernando Senior Linux Systems Administrator – Infrastructure & Security Senior Linux Systems Administrator – Infrastructure & Security Senior Linux Systems Administrator – Infrastructure & Security Senior Linux Systems Administrator – Infrastructure & Security

  4. What’s our problem ? • Small Team • Lots of Users (All untrusted) • Lots of Systems • Lots of Logs • Lots of workload on Systems • Lots of performance bottlenecks • Lots of performance metrics to monitor

  5. How to overcome ? Monitoring Logs Monitoring Performance Collectd Graphite Database (eg: PSQL) Grafana Dashboard • Logstash • Redis • Elasticsearch • Kibana Dashboard

  6. Software – Log Monitoring

  7. Log Monitoring software : What it does • Logstash –Collects ,Stores and Parses logs • Redis - Message broker • Eleasticsearch - Search Server • Kibana - Provides nice visualization for Elasticsearch , could be served on Nginx/Apache

  8. Data Sources - Logstash Logstash forwarders • Linux/Unix Syslog • Windows Event Log • Packet captures (Eg: tcpdump pcap files) • SNMP traps

  9. Software – Log Monitoring

  10. Logstash configuration • input {    file {#Alfresco Audit Syslog File        path => [ "/var/log/audit.log" ]    } • filter {    grok {#Regex expression to filter logs which needs to be sent to elasticsearch         type => "alfresco"        match => [ "message", "%{SYSLOGTIMESTAMP} (?:%{SYSLOGHOST} )?%{DATA}: %{GREEDYDATA:data}" ]    } • output {#Output to elasticsearch elasticsearch_http {       host => localhost       port => 9200    } 

  11. Logstash / Elasticsearch in the Cloud Infrastructure

  12. Log Analysis – Kibana Dashboard

  13. AWS – Elasticsearch & Kibana • We can use the elasticsearch managed service provided by AWS which will be ease : • Administration • Scalable • To integrate with logstash & Kibana • Highly secure

  14. Performance monitoring Software – What it does • Collectd – Collects system performance statistics inorder to send it to carbon-cache • Graphite – 3 components • Carbon-cache – Network service which Listens for incoming metrics in a buffer/cache • Whisper - Stores time series data (RRD format) • Graphite webapp – Renders graphs • Database (Eg: Postgresql) – Stores user info , permissions,graph and dashboard configurations • Grafana – Graphite data displayed with customized cool graphs (Could be served on Nginx/Apache)

  15. Graphite in the Cloud Infrastructure

  16. Data Sources - Graphite • By default Collectd sends data to graphite such as : • CPU • Network • Memory • Disk • Processes • ‘Collectd’ can be used to monitor performance/workload of : • Databases such as MySQL , Postgresql • Java based applications (Eg: Tomcat) by sending JMX related data (Eg : Heapmemory / Non heap memory usage , busythreads , etc)

  17. Collectd Configuration for JMX cat collectd.conf LoadPlugin java <Plugin "GenericJMX"> <MBean "classes"> <Value> Type "jmx_memory” Table true Attribute "HeapMemoryUsage" InstancePrefix "heap-" </Value> </MBean>

  18. Performance Analysis – Grafana Dashboard(Output of JMX data)

  19. Solr Memory/Load Usage displayed in Grafana

  20. Grafana with AWS Cloudwatch Grafana ships with built in support for AWS cloudwatch Authentication & Authorization from Grafana to AWS cloudwatch is handled by AWS access & secret keys The keys are stored under $GRAFANA_HOME/.aws/credentials

  21. Grafana in AWS Cloudwatch

  22. Summary • Real-time centralized logs from many sources • Real-time suspicious & intrusion logs • Logs can be stored for any amount of time • Logstash/Kibana can be easily deployed in cloud services such as AWS • Real-time performance monitoring metrics from many sources • Easy to identify bottlenecks • Easy to identify and co-relate any bottlenecks caused for further system performance tuning • Grafana is available for IAAS services such as in AWS which can use AWS Cloudwatch as the source of metrics

  23. Thank you

  24. Founded 2007 London& Colombo Experts in digitisation 66 People Certified Staff

More Related