1 / 25

Managing Your Security Logs with Elasticsearch

Managing Your Security Logs with Elasticsearch. Vic Hargrave | vichargrave@gmail.com | @ vichargrave. $ whoami. Software Architect for Trend Micro Data Analytics Group Blogger for Trend Micro Security Intelligence and Simply Security Email: vichargrave@gmail.com Twitter: @ vichargrave

Download Presentation

Managing Your Security Logs with Elasticsearch

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Managing Your Security Logs with Elasticsearch Vic Hargrave | vichargrave@gmail.com | @vichargrave

  2. $ whoami • Software Architect for Trend Micro Data Analytics Group • Blogger for Trend Micro Security Intelligence and Simply Security • Email: vichargrave@gmail.com • Twitter: @vichargrave • LinkedIn: www.linkedin.com/in/vichargrave

  3. What is ? • Open Source SECurity • Open Source Host-based Intrusion Detection System • Founded by Daniel Cid • Log analysis and file integrity monitoring for Windows, Linux, Mac OS, Solaris and many *nix systems • Agent – Server architecture • http://www.ossec.net

  4. OSSEC does SIEMs Syslog commercial or open source SIEM syslog Syslog Syslog

  5. Commercial SIEMs are great, but… commercial SIEM = +

  6. Now there’s a whole new (open-source) ballgame Logstash Kibana

  7. OSSEC Log Management with Elasticsearch

  8. Elasticsearch • Open source, distributed, full text search engine • Based on Apache Lucene • Stores data as structured JSON documents • Supports single system or multi-node clusters • Easy to set up and scale – just add more nodes • Provides a RESTful API • Installs with RPM or DEB packages and is controlled with a service script.

  9. Elasticseach Elements • Index – contains documents, ≅ table • Document – contains fields, ≅ row • Field – contains string, integer, JSON object, etc. • Shard– smaller divisions of data that can be stored across nodes • Replica– copy of the primary shard

  10. ElasticsearchMulti-node Configuration # default configuration file - /etc/elasticsearch/elasticsearch.yml ######################### Cluster ######################### # Cluster name identifies your cluster for auto-discovery # cluster.name:ossec-mgmt-cluster ########################## Node ########################### # Node names are generated dynamically on startup, so you're relieved # from configuring them manually. You can tie this node to a specific name: # node.name:"es-node-1"# e.g. Elasticsearch nodes numbered 1 – N ########################## Paths ########################## # Path to directory where to store index data allocated for this node. # path.data:/data/0, /data/1

  11. Logstash • Log aggregator and parser • Supports transferring parsed data directly to Elasticsearch • Controlled by a configuration file that specifies input, filtering (parsing) and output • Key to adapting Elasticsearch to other log formats • Run logstashin logstash home directory as follows: bin/logstash––conf<logstashconfig file>

  12. OSSEC – logstash.conf input { # stdin{} udp{ port => 9000 type => "syslog" } } filter { if [type] == "syslog" { grok { # SEE NEXT SLIDE } mutate { remove_field => [ "syslog_hostname", "syslog_message", "syslog_pid", "message", "@version", "type", "host" ] } } } output { #  stdout{ #    codec => rubydebug #  } elasticsearch_http{ host => "10.0.0.1" } }

  13. OSSEC Alert Parsing • OSSEC syslog alert • grok { } Jan 7 11:44:30 ossecossec: Alert Level: 3; Rule: 5402 - Successful sudo to ROOT executed; Location: localhost->/var/log/secure; user:user; Jan 7 11:44:29 localhostsudo: user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/bin/su match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_host}%{DATA:syslog_program}: Alert Level: %{NONNEGINT:Alert_Level}; Rule: %{NONNEGINT:Rule} - %{DATA:Description}; Location: %{DATA:Location}; (srcip: %{IP:Src_IP};%{SPACE})? (dstip: %{IP:Dst_IP};%{SPACE})? (src_port: %{NONNEGINT:Src_Port};%{SPACE})? (dst_port: %{NONNEGINT:Dst_Port};%{SPACE})? (user: %{USER:User};%{SPACE})?%{GREEDYDATA:Details}" } add_field=> [ "ossec_server", "%{host}" ]

  14. Kibana • General purpose query UI • Javascript implementation • Query Elasticsearch without coding • Includes many widgets • Run Kibana in browser as follows:http://<web server ip>:<port>/<kibana path>

  15. Kibana– config.js /** @scratch /configuration/config.js/5 * ==== elasticsearch * * The URL to your elasticsearch server. You almost certainly don't * want +http://localhost:9200+ here. Even if Kibana and Elasticsearch * are on the same host. By default this will attempt to reach ES at the * same host youhave kibanainstalled on. You probably want to set it to * the FQDN of your elasticsearchhost */ elasticsearch: http://+"<elasticsearch node IP>"+":9200",

  16. Elasticsearch Cluster Management • ElasticHQ • Elasticsearch plug-in • Install from Elasticsearch home directory: bin/plugin -install royrusso/elasticsearch-HQ • Provides cluster and node management metrics and controls

  17. And now for something completely different.The OSSEC virtual appliance

  18. Back to Reality Free

  19. Elasticsearch Security Caveats • Designed to work in a trusted environment • No built in security • Easy to erase all the data • Use with a proxy that provides authentication and request filtering such as Nginx • http://wiki.nginx.org/Main curl –XDELETE http://<server>:9200/_all

  20. Further Information • Elasticsearch • http://www.elasticsearch.org • Logstash • http://logstash.net • Kibana • http://www.elasticsearch.org/overview/kibana/ • ElasticHQ • http://elastichq.org • Elasticsearch for Logging • http://vichargrave.com/ossec-log-management-with-elasticsearch/ • http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html

  21. Thanks for attending! Any questions?

More Related