LOG MONITORING GUIDE

1. LOG MONITORING GUIDE: USING LOKI, PROMTAIL, AND GRAFANA FOR EASY LOG MONITORING https://cloudtechservice.com/promtail-log-monitoring/ Log monitoring using promtail and loki can help us to know about various activities that is happening in a system or application. It can help us to find the cause of errors and hence save a lot of our time while debugging. Logs should be made accessible to developers for debugging and if we want developers to have access to logs without giving them access to servers then this guide is a really effective way to setup log monitoring. If you read my previous blog, we learnt how to setup mysqld_exporter in ubuntu server for MySQL database monitoring. Now, we will setup Loki and promtail in a container which will basically collect all our logs and store them. Also we need to setup promtail in our server for Promtail is an agent which scrapes logs from server and sends the logs to loki. It is usually deployed to every machine that has applications needed to be monitored. Then the logs collected by loki is displayed in grafana dasboard. We dont need to configure anything in prometheus. Loki to work. PART 1: SETTING UP LOKI AND PROMTAIL CONTAINER In my previous blogs i had setup node_exporter, cAdvisor, mysqld_exporter in the same server. In this case too we will learn how to setup loki and promtail in the same server. So lets add extra configs in our docker-compose.yml file that we have created in my previous blogs. Add this extra volume: volumes: loki-data: driver: local You conifg should look like this:

2. Also add settings for loki and promtail: loki: image: grafana/loki:2.5.0 container_name: loki ports: - "3100:3100" volumes: - loki-data:/loki - "./loki/config:/etc/loki" command: -config.file=/etc/loki/local-config.yaml restart: unless-stopped networks: - prometheus-network promtail: image: grafana/promtail:2.5.0 container_name: promtail volumes: - "./promtail/config:/etc/promtail" - /var/log:/var/log - /home/ubuntu:/home/ubuntu command: -config.file=/etc/promtail/config.yml restart: unless-stopped networks: - prometheus-network Your config should look like this:

3. Now just build the containers by using this command: sudo docker-compose up --build -d

4. Now you should be able to see 2 extra folders loki and promtail. This is where we will add extra config files for promtail. First lets add config file for loki. sudo nano loki/config/local-config.yaml Now add this config in the local-config.yaml file: auth_enabled: false server: http_listen_port: 3100 common: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem

5. schema: v11 index: prefix: index_ period: 24h It should look like this: Now we are going to add config file for promtail. Here we will configure address of loki to send our logs, jobname, labels for each job and folder to scan for logs. sudo nano promtail/config/config.yml server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: prometheus-system static_configs: - targets: - localhost labels: job: prometheus-varlogs

6. __path__: /var/log/*log Our config file should look like this: In our above config file, all we need to know is the client section and scrape_config section for now. We have set the client url: http://loki:3100/loki/api/v1/push It means that promtail will push the log data to our loki api. Here the above url is only working because the container name of loki is loki and promtail is in the same network as of loki. The scrape_config section is for adding location of log files and job names. Job_name is simply a name which can be set to whatever u want. And in labels section we have “job” whose name is displayed in grafana. Set it properly as this is important for classifying out log files. I will show this in grafana later. __path__ is set to /var/log/ directory and it will include all files whose names end with log. Now lets restart our docker-containers and check containers sudo sudo docker-compose ps docker-compose restart

7. Looks like everything is running fine. PART 2: CONFIGURING LOKI IN GRAFANA Now its time to add Loki as a datasource in grafana. Login to your grafana. If u are confused check my previous blog. Goto settings and click on data sources: Now Enter url and click on save and test:

8. Now lets browse for our log files and add them on a dasboard. Go Also make sure u have selected on code as shown in screenshot below: to explore and select loki.

9. Now click on log browser and click on the jobname that we had specified in our promtail config file. It will display all the files that falls under the jobname.

10. Here we can see three files; syslog, auth.log and kern.log All of these file names end with log. Similarly we can add other job names and location of other folders in our promtail config file under scrape_config section. This will help to monitor other log files as per our need. Now, lets check the log and Simply click on a filename and then show logs: add it to our dashboard.

11. Now we can see the content of syslog:

12. We can change the time of log as per our need and also check logs in love mode. Lets add it to a dashboard: Click on add to dasboard, select new dasbhoard and then open in new tab:

13. Then a panel will appear. You can press E to edit the panel and give a name. Also lets add a new row and give it a name. My final dashboard looks like this now:

14. I have changed the dashboard name and saved in general folder. You can explore and tweak around with many options.